It's not a 40 minute thing, this isn't Caltrain, but it's still annoying.
(I'll let you think on that puzzle.)
The second reason is closer, but not exact. Caltrain has a variety of different types of trains, some stop at all the stops, some stop at only a few. Here's the scenario: there is a fast train then a slow train. Your normal schedule doesn't allow you to get to the station on time to take the fast train, so you normally take the slow train. If both trains are behind schedule, you might be able to make it to the fast train, and since it goes faster you might get to your destination earlier than normal.
Why don't they just build out BART between Millbrae and Santa Clara? Once the Silicon Valley Extension is finished, BART will be almost a loop around the Bay except for that gap.
Plus the fact that the citizens (through their representatives) voted against it decades ago, so it was removed from the original plan. Given the current fuss among the citizenry over similar problems with running the nascent high-speed rail up the Caltrain corridor, I’d guess a vote today on BART would have the same outcome.
Couldn't they build a tunnel? With a tunnel, the requirements for eminent domain would be relatively limited (portals, station entrances, maybe some ventilation/utility buildings)
Sydney, Australia's new metro system which is being built relies mostly on tunnels (plus some conversion of existing surface lines, and elevated rail in outer suburban areas where land is relatively cheap). Sydney's metro area (5.2 million) isn't hugely larger than the Bay Area's (4.7 million).