This is basically true, actually. It'll either work or it won't, and you're just out of the bandwidth if it fails verification. At the absolute worst, you won't be able to accept any blocks from the network as a whole if it's fake, because the hashes won't line up. IIRC the official client (and most others) also include a hard-coded hash part way through the chain to help ensure you're on the correct one up to that point.
That's of course assuming there isn't some exploit in your client, due to e.g. unsafe reading of the file that could allow it to execute arbitrary code hidden in the blockchain file.