There's a lot of things working against the concept. Weather constraints and sport pilot restrictions are logistical issues, but by far the #1 issue is cost. The Transition, Terrefugia, and A5 aren't doing much to solve this by selling for $190-300k, so they definitely won't be changing how the general public travels.
Regardless, I couldn't be happier with my Sport License. For now I can only afford to rent, and not much at that. Were it not for the LSA category though, flying would still just be a wish.
I'd rather do that and then live nearly anywhere in the 400 mile range that it provides, and probably be happier.
So if you're a hobbyist, and you fly a light sport aircraft six hours a week for thirty years, you've got a 20% chance of it killing you. (Right? I haven't had coffee yet.)
Several issues though:
The risk is far from linear. Infrequent pilots have much higher risk per hour.
The number of hours for part 91 (most private flying) flights cannot be reliably determined.
About 75% to 95% of risk is under the control or strong influence of the pilot. Weather, fuel, crew readiness, pre-flight planning and related issues are much more under pilot control (if you're willing to cancel or delay flights) than in comparable car trips. It's extremely rare for another pilot to kill you.
I came up with 9% if you assume two fatals per accident.
Light sport is restricted to day VFR, which is lower risk, other things being equal. Unfortunately, that also saps some utility and pushes you towards taking a late day flight rather than waiting for better night weather. I don't think LSA is viable for transportation, which makes it all the more unusual for someone to fly 300 hours per year.
(Context: I fly about 150-200 hours per year, which is 80+% of our vacation travel. I also read every NTSB fatal report.)
So if you remove the "accident waiting to happen" category of pilots, the fatal accident rate will likely be much lower. I've heard it said that GA has a similar accident rate to riding motorbikes, which is another activity where the risk mostly depends on the rider.
The type of pilot who is most dangerous is the type-A business owner who feels they have to get to their meeting no matter what the weather. Unfortunately these are the people who will have $250k in their pocket and want to use a plane for transportation. Perhaps the only solution is to have self-flying planes, similar to google's cars. (And of course, they would need to be hacker-proof!)
The way I usually explain it is to say that in a car you are much more at the mercy of other drivers. A huge percentage of people killed driving are killed by drivers of other cars. In an airplane nearly all accidents involve only one aircraft. With that, and the fact that so many accidents are pilot indiced the stats for a well trained conservative pilot are far far better than "average." Good pilots constantly re-train and do things like read NTSB reports to minimize this risk as far as possible.
There's still a "crap happens" factor (eg catastrophic mechanical failure) that can take down even the best of pilots but nothing in life is completely devoid of "crap happens" risk.