Since development follows highways, you already know where transit needs to be. Caltrain already closely matches 101 on the peninsula.
Then tax cars on the road to reduce congestion as the busses use the road more efficiently.
Really, you'd need to do these both at once and getting support for such a thing would be extraordinarily difficult without it being proven first.
Also, I doubt it could work as well or better than trams/streetcars in dense areas. I mean the model used by trams/light rail in European cities (the one that comes to my mind is Zurich) rather than that used by Muni. In Zurich there's a stop every 2-3 blocks and it'll usually have a tram coming within a few minutes. I've seen cities were a wait longer than 3 minutes during the day is rare.
A bus like you describe would certainly be more convenient (it can turn and go down streets without rails, as well as come to pick me up) but I'm not sure it would be better overall (the pickup costs time that's lost by the current passengers of the bus for example).
Well we could solve the second two problems, and not needing rails seems like an advantage.
This will likely take 90 minutes or so to get from SJ to SF, which is a <60-minute drive (when traffic is clear). The current time from Fremont to Embarcadero is 58 minutes, so it will definitely take longer than driving. Going to anywhere south of SF, e.g. The current southern-most stop Millbrae (BART stop close to SFO, actually requires taking a separate train from there into SFO) from Fremont currently takes 1 hour 35 minutes, so from Berryessa that'll be closer to 2 hours.
Had they ran BART up the peninsula, which takes Caltrain 1 hour 10 mins to get from Diridon in downtown SJ to the Millbrae station, which again doesn't get you into but only close to the airport, the BART could have likely made it in less than 1 hour, making it somewhat useful if you wanted to get from SJ to SFO without taking a $150 cab ride or taking 2 hours and 3 separate fares door-to-door going Uber -> Diridon - Caltrain -> Millbrae -> BART SFO connector.
By running BART down the east bay, they're also limiting the options for going useful places on the peninsula. Only Caltrain could get you from e.g. SJ to Palo Alto. If you want to go to Stanford and you start at the SJ Caltrain stop (so, best-case) you're looking at 60 minutes of travel to cover 15 miles and a 30 minute walk from Caltrain to Stanford once you get off.
For as many people as the Bay Area has, and for as much money as the Bay Area has, our public transit really doesn't work for many use-cases, and we pay for that with our time spent in traffic, along with all the accidents, pollution, and stress that brings. Drunk driving deaths are also caused because people have no good way to get home after BART and Caltrain stop running at midnight.
Elect me and I'll declare eminent domain on the NIMBYs in the peninsula and build 4 tracks of BART so it can run 24x7 all around the bay, from Santa Cruz up to Tiburon, from Santa Rosa down to Gilroy, with a Sacramento connector. And I'll make the billionaires and local corporations who currently off-shore corporate and capital gains taxes pay for it.
Even with the new transbay transit center they are under building it with only 6 platforms, they should build 8. And the circuitous route they are digging for access from the existing Caltrain station makes no sense: there should be a station at 7th at instead which will provide a better transfer point for future metro lines.
Fixed systems never get you from where you are to where you want to be. Have you ever just happened to be at a BART station and decided I want to go to this other BART station? I doubt it. (OK, in high-density urban situations this can happen.) Changing transportation modes is a real joy killer.
The problem to solve with autonomous vehicles is a packing problem. Better packing while in motion (driving) and storage (parking). I submit this is a much easier set of problems to deal with given the technology we have and that which is on the horizon than getting fixed systems to transport you between arbitrary points A and B.
I'm a life long Bay Arean, and I'm familiar with all the major corridors and transportation modes. Here is my boiled down list of the some of the bigger issues:
1) BART is maxed-out. The bandwidth on the stretch from Millbrae to West Oakland effectively constrains the capacity of the entire system. The cars are too small and uncomfortable when overcrowded. The station size limits the number of cars to 10. Tunnel overhead prevents double-decker cars.
2) Bandwidth utilization on the Caltrain line is laughably small. There are still many grade crossings which snarl surface traffic at commute time when the occasional train does pass through.
3) The tie-ins of the east-west corridors to the north-south corridors are inefficient and cause some of the biggest back-ups. (237, 238, Dumbarton Bridge, San Mateo Bridge, Bay Bridge. The tie-in of the San Rafael Bridge is the happy exception, backups on 101 and 80 having more to do with maxing-out the local bandwidth. The Bay Bridge bandwidth itself is pretty much maxed-out.
So how do we solve the autonomous vehicle packing problems?
1) Self-driving (truly autonomous) cars will eventually allow platooning and so better packing while in motion. This should also lead to fewer fender-benders at commute time.
2) Self-driving should also lead to better packing in storage (parking).
3) Better monitoring of vehicle systems to prevent breakdowns from stalling traffic.
4) More capacity at critical network connection points (intersections). There is still room for auxillary lanes in many cases around the east-west connections I mentioned above and elsewhere. Caltrans has in some cases intentionally created choke points to meter traffic.
5) Can we hurry up with the current construction? I think it is a cost saving move to stretch out improvement projects, but is the savings worth the delay?
6) With self-driving and better monitoring of vehicle systems it is possible to make the rail corridors dual-purpose. In other words pack autonomous vehicles into the space between trains. That will require a sophisticated traffic control system and access ramps at stations (expensive to build at underground stations). Think of packing vehicles with the form factor of a Smart Car on a road surface build around the tracks. How much would that increase bandwidth? The Caltrain corridor alone could contain the equivalent of 2 lanes of traffic in each direction, possibly 3 if you really packed it. There are also underutilized Union Pacific / Amtrak lines in the East Bay, not to mention increasing the bandwidth of BART.
7) There is also room for better separation of vehicles and pedestrians and bicycles. This problem is still expensive, but solvable.
