0: http://www.wired.com/entertainment/theweb/magazine/17-09/ff_...
Both terms are terribly over-used and make discussion challenging, since both sides end up assuming what they words probably mean (without every discussing the meaning).
These are the first noticeable innovations on CL in many years. Might they be the result of the on-going parade of sites improving upon CL w/ their own data, not to mention the criticisms on HN and elsewhere?
Whether they 'hate' competition is debatable. Clearly they view their data as THEIR property. They occasionally let things like PadMapper go while putting the ban hammer down on other things, very arbitrarily.
The Wired story on them about 5-6 years back suggested they are just very skeptical of third parties and human nature in general. It left me with the impression that Craig is just deeply selfish and authoritarian.
With no disrespect to the OP, that's not a ton of users, even for a single day...so I'm kind of surprised Craigslist noticed enough to block them. There must be tons of unapproved scraper scripts checking CL on a daily basis.
The past few years I have been learning over and over: ship it, get feedback, improve, repeat.
So, getting back to how they blocked me. If I understand correctly, craigslist blocked any HTTP requests from the Google App Engine User Agent. HTTP Requests from Google App Engine always append 'AppEngine-Google' to the end of the User Agent (http://code.google.com/p/googleappengine/issues/detail?id=34...).
You can recreate this block by doing this in python (you need the amazing python requests library, so install that): import requests
r = requests.get('http://craigslist.com) print r.status_code
This should print 200 and you're good to go.
Then, try it with the 'AppEngine-Google' in the user agent: r = requests.get('http://craigslist.com, headers={'User-Agent': 'AppEngine-Google'})
and r.status_code will return 404
bingo bango, you've been blocked. I could have just moved off of Google App Engine, etc. but I would probably be faced with a similar fate as Padmapper if I ever got any kind of sizable user base.
I created this gist: https://gist.github.com/wlindner/4752122 since the python code formatting is wonky
If you don't like the way "he" plays with his toys, then tough cookies - "he" doesn't care and neither does anyone else except those who want to also complain about him not sharing in the way they want him to. You'll see this damn near every time someone posts a "Help! My API provider just changed the rules!" type of post. There are two types of responses: "I hate that - they should be more open", or "It's ridiculous to even think that you can build a business based off of leaching from someone else and that that other entity would be okay with that forever."
[1]Unlike children (who typically are given their toys), these toys are the result of "his" work.
Er, no. They are the work of everyone who posts on CL.
This doesn't really solve the problem of Craigslist hoarding all their data to themselves (as pointed out numerous times already), but there are enough other sources to make it usable.
