The problem is two fold. The first is that most authors and publishers are not very technical. The second is tools like InDesign created EPUBs that would fail the idpf epubcheck tool with errors [1]. Adobe has fixed some of these issues in later versions, but it's expensive for a publisher to re-export all their books.
What this leads to is basically fixing random, one off issues depending on publisher and book. I would definitely suggest not writing your own reader and instead looking at something like Readium (and contribute if you have time!).
[1] http://mademers.com/two-more-indesign-cs5-export-to-epub-bug...
