Grub Crawler – Python Client
Since the last December I've been hacking on a python client for the (in)famous crawling project, Grub. The Grub project's goal is to crawl the whole web, in a decentralized way, and provides the data, for search engines to be build. Wikia Search is actually based on Grub's data. Only a C# client existed -which is actively developed now- so because I'm not a big fan of the whole .Net (or Mono, if you like it this way) idea I decided that should write a Python client.
You can grab the sources and start crawling yourself. Remember that you need a Wikia account first. The client is not in complete state but it works good enough for me to produce more than 3000 units of crawled pages, with 255 pages per unit, every day. Not bad, right?
Please give it a shot and let me know what you think and fill out some bug reports.