We have four kids, but only three are special needs. Most of them were able to follow "adult" works from a fairly young age, but one of them was not until age 8; he struggled following simpler works (e.g. Gannett's
My Father's Dragon series) in the first grade. Once it "clicked" though, he was great.
When I looked into it, I found similar stories from educators of verbal skills not clicking until the 3rd or 4th grade, followed by a rapid catch-up and then zero lag. According to most special-ed folks I talked to, if there are more specific issues at a younger age (e.g. problems with reading comprehension, but not auditory, or vice-versa) it can be indicative of a learning disability.
You are 100% true about reading to them from a young age (and not worrying too much about "hard" words; kids don't know which words are supposed to be hard unless you tell them) makes a big difference. Seeing kindergarteners from homes without a single book in them vs ones read to from an early age is rather eye-opening.