Fair use (or i guess fair dealing in australia) isn't totally about not negatively impacting other people's businesses. It helps your fair use argument if you don't impact, but there are plenty of clear cases where its fair use despite negative business impact. For example writing a scathing negative review of something that has quotes for context.
You are right. By "business impact" I was rather thinking that the aggregator simply diverts ad revenue, without adding any value. If I made a business which summaries other books without sharing revenue, I would likely get busted.
