For the vast majority of the population it was impoverished even before Castro.

It was that way, and yes suffered continuous population exodus to Florida on homemade rafts, since... EVER. It's simply a "bad country" overall and it has very little to do with Castro or Communism. It never had a permanent population of any considerable size that lived there voluntarily - pre-Columbian Taino people were extremely few in numbers, and lived in a completely prehistoric manner compared not just to Latin American native American pre-Columbian civilizations of same era, but even to North American first nations. Because only people who would live there were losers who were pushed out from everywhere else.

After that, vast majority were slaves, brought continuously by force, because they kept dying and dying and even with insane birth rate of the era there was no chance to ever sustain their population without continuous imports. And there was no way to make agriculture profitable without this horrible exploitation.

After slavery, population was... whoever didn't have balls to step on a raft and sail to Florida. This country is simply not made for living. Centuries of continuous negative selection because of massive one-way emigration and no immigration at all, took a genetic toll, too.

It doesn't exonerate Castro or Commies in general, of course they built a social model that beats even North Korea in dysfunction (North Korea at least made nukes), but just for a fact, Cuba is a useless place to live and if it was free and democratic, it would be very, very sparsely populated with people living there only for specific purposes of exploiting natural resources/shipping/fishing, but not permanent argicultural let alone industrial settlement.