I don't know that anecdotal evidence about driving conditions is really a great way to talk about driving on gravel. I'm also an Iowan who grew up on gravel roads and frequently drove faster than 50 mph, but if I'm being totally honest I recognize that it wasn't a great idea. On more than one occasion a loose patch of gravel or surprise deer has nearly sent me into a tree at 60 mph.
Sure, the people who frequently use those roads are the ones who will know them best and be able to predict their condition, but your stopping/maneuvering power is so much worse on gravel that assuming an average speed of anything greater than, maybe even near 50 mph seems pretty reckless.