Broback's Blog

Are you a Web data geek? Are you into scraping sites? If so, you may be one of the many people who rely on the highly-regarded Needlebase to help you in your efforts.

After reading about Needlebase on RWW, (very, very cool example of the potential here.) I spent many hours playing with it — and while I was able to do some interesting/fun things, I finally gave up on Needlebase because of several issues:

Byzantine interface. Think this. Built by engineers, for engineers. While I’m sure Professor Frink had no issues, mortals were no doubt lost much of the time.

Latency. Since it was a Web-based service running in the browser, and required clicking to page after page after page to set set up a sequence, it always felt like I was running in molasses. (Might have been more of a Safari issue than a Needlebase issue…?)

Constant denials. Since it was a Web service, and sites hate getting scraped, sites learned to deny access to Needlebase. This is not Needlebase’s fault, BUT — you generally didn’t know until well into the process that your efforts were all for naught. (See interface issues above).

The bad news is that Google (who acquired the service) has now killed it.

The good news is that most people can happily get what they need using the excellent FireFox plug-in Outwit Hub.

I discovered Outwit Hub months ago and never looked back. Discovered I wasn’t a dope after all. Half an hour after installing it, I was happily extracting data from a myriad of sites.

Outwit Hub Works like normal humans would expect. Define fields, establish pre and post tag/html sets, enter a URL, and scrape away. No fool questions, denials by sites, or convoluted questions popping up. Since much of what’s happening is running locally on your CPU, the latency issue largely vanishes. Cheap too. Pay $35 one time, scrape forever.

R.I.P, Needlebase, I know there were a lot of people who relied on it and liked it a lot. I guess I’m glad now I just couldn’t warm up to it.