> This led me to my next and (currently) final stop, Kullish, which searches through a number of link aggregation and discussion websites (including Reddit) for a URL before providing a single feed of comments from everywhere.
But reddit for instance disallows everything in its robots.txt
And again, there is no crawling involved.
Crawling is a very specific, well defined behavior. I'll assume with good intent that you aren't familiar with the definition of crawling, in which case you probably should become familiar with it before making inaccurate comments like these in the future.
If you had taken the time to do some relevant reading before writing your inaccurate comments, you would have seen:
> Our [Reddit] robots.txt is for search engines, not Data API users.
