Share code that uses new URL Search tool and win AWS credit (opens in new tab)

The main intent of the search is to retrieve a list of urls that the site crawled for a given subdomain, domain or tld. So for now you can do that using reversed url notation. Which I admit is not very intuitive.

We're toying with the idea of implementing some sort of wild card that way we can present the urls in natural order. Something like *.google.com to retrieve all urls under google. But we wanted to judge the level of interest first. After all "done" is better than "perfect".

Just an oversight. Most of our work is done by people who graciously volunteer their time. We'll get that fixed.

Yes, that's a little odd. If you shorten the search term, the results come up just fine: http://urlsearch.commoncrawl.org/?q=nl.utwente.cs.wwwhome/~h...

Works on me!

http://urlsearch.commoncrawl.org/?q=com.pbm.www%2F~lindahl

So there's a bug there, but not all the time for ~.

Is it possible to get a list of webhosts, like all the domains and subdomains, stripped from the rest of the url?

good suggestion

I think it's just that the site converts unicode urls for display purposes. If you click on one of the links with "@" in it, you'll see the real url in url encoded format

http://%2E%2E%2E@harunyahya.com/Ajax/get.comments/oid/4612 176.34.181.212 20120516214328 text/html 912

Thanks for the catch Djoerd! We will fix it now