* Some number of people are sensitive to latency
* Some people are unable to read my website
You're gambling that there are more people who would walk away due to latency, than there are who would walk away due to being unable to interact with your site at all.
Personally I suspect that to be a poor tradeoff.