So if you remove the "accident waiting to happen" category of pilots, the fatal accident rate will likely be much lower. I've heard it said that GA has a similar accident rate to riding motorbikes, which is another activity where the risk mostly depends on the rider.
The type of pilot who is most dangerous is the type-A business owner who feels they have to get to their meeting no matter what the weather. Unfortunately these are the people who will have $250k in their pocket and want to use a plane for transportation. Perhaps the only solution is to have self-flying planes, similar to google's cars. (And of course, they would need to be hacker-proof!)