Are you a Web data geek? Are you into scraping sites? If so, you may be one of the many people who rely on the highly-regarded Needlebase to help you in your efforts.
After reading about Needlebase on RWW, (very, very cool example of the potential here.) I spent many hours playing with it — and while I was able to do some interesting/fun things, I finally gave up on Needlebase because of several issues:
Byzantine interface. Think this. Built by engineers, for engineers. While I’m sure Professor Frink had no issues, mortals were no doubt lost much of the time.
Latency. Since it was a Web-based service running in the browser, and required clicking to page after page after page to set set up a sequence, it always felt like I was running in molasses. (Might have been more of a Safari issue than a Needlebase issue…?)
Constant denials. Since it was a Web service, and sites hate getting scraped, sites learned to deny access to Needlebase. This is not Needlebase’s fault, BUT — you generally didn’t know until well into the process that your efforts were all for naught. (See interface issues above).
The bad news is that Google (who acquired the service) has now killed it.
The good news is that most people can happily get what they need using the excellent FireFox plug-in Outwit Hub.
I discovered Outwit Hub months ago and never looked back. Discovered I wasn’t a dope after all. Half an hour after installing it, I was happily extracting data from a myriad of sites.
Outwit Hub Works like normal humans would expect. Define fields, establish pre and post tag/html sets, enter a URL, and scrape away. No fool questions, denials by sites, or convoluted questions popping up. Since much of what’s happening is running locally on your CPU, the latency issue largely vanishes. Cheap too. Pay $35 one time, scrape forever.
R.I.P, Needlebase, I know there were a lot of people who relied on it and liked it a lot. I guess I’m glad now I just couldn’t warm up to it.
It’s not just housing bubbles that are caused by good intentions and market intervention. Mungowitz reports on yet another prime example of Soviet-style meddling in the U.S. pharmaceutical market.
While the Food and Drug Administration monitors the safety and supply of the drugs, which are sold both as generics and under brand names like Ritalin and Adderall, the Drug Enforcement Administration sets manufacturing quotas that are designed to control supplies and thwart abuse. Every year, the D.E.A. accepts applications from manufacturers to make the drugs, analyzes how much was sold the previous year and then allots portions of the expected demand to various companies.
How each manufacturer divides its quota among its own A.D.H.D. medicines — preparing some as high-priced brands and others as cheaper generics — is left up to the company.
Gee, setting production quotas based on brainy insights of a handful of technocrats. Much better than letting supply and demand determine a market clearing price. After all, those unwashed masses have no idea what they really want, right?
Naturally, the response to the limited supplies and resulting high prices will be… wait for it…. demand for more intervention!
I’m adding a new category to this blog. It’s Iatrogenesis. This is the formal term for what’s better known as the situation where the cure becomes worse than the disease.
Given the recent SEC indictments, I couldn’t resist this passage from Krugman in July of 2008 (emphasis mine:)
“But here’s the thing: Fannie and Freddie had nothing to do with the explosion of high-risk lending a few years ago, an explosion that dwarfed the S.& L. fiasco. In fact, Fannie and Freddie, after growing rapidly in the 1990s, largely faded from the scene during the height of the housing bubble.
Partly that’s because regulators, responding to accounting scandals at the companies, placed temporary restraints on both Fannie and Freddie that curtailed their lending just as housing prices were really taking off. Also, they didn’t do any subprime lending, because they can’t: the definition of a subprime loan is precisely a loan that doesn’t meet the requirement, imposed by law, that Fannie and Freddie buy only mortgages issued to borrowers who made substantial down payments and carefully documented their income.”
“So whatever bad incentives the implicit federal guarantee creates have been offset by the fact that Fannie and Freddie were and are tightly regulated with regard to the risks they can take. You could say that the Fannie-Freddie experience shows that regulation works.“
Two things going on here. Of lesser importance is Krugman being so galactically wrong about the role of Fannie and Freddie in the subprime mess. More important is the blind faith in institutions and the smart, well-intentioned people who create and run them.
Trusting regulations and rules to this degree is a classic case of Sowellian Unconstrained thinking and belief in engineered solutions with goals/intentions at their core. The Constrained view prefers systemic, evolved solutions based on incentives. Consider the thinking behind Fannie/Freddie (and a dozen other like-minded agencies):
“We need to help more people get houses, let’s get those rates down.”
Compare to:
“The interest rates on loans automatically adjust to match the risk of the loan.”
The former is based on good intentions and is a proven failure. The latter is based on the self-interest of individuals, and the more you rely on the latter, the better you do.
I believe history will show that monopolistic government-backed entities (like ratings agencies) are prone to behave irresponsibly — despite the feel-good motives and regulations that frame them. Incentives matter.
Here may be an entrepreneurial opportunity. Register at thousands of sites using a variety of non-intuitive unique email address that are used nowhere else. When you start getting emails about viagra sent to one of these addresses, flag the related site in a database as one to avoid. Build a browser plugin that accesses that database in order to help others from becoming victims to sloppy or malevolent list management.
FYI: Here are a few sites that I foolishly registered at and now get spam sent (viagra, penis enlargement, etc.) to the unique addresses I used. Seven sites, seven different addresses.
A fitting legacy. This troll of global proportions who threatened to turn Seoul’s presidential palace office into a “sea of fire,” will be be thought of in this hilarious context for all time. Many thanks to Parker, Stone and team for this lasting impression.
Had a chance to sit in on Steve Blank’sLean Launchpad class last week at the Haas School of Business. Came away very impressed with the real-world/pragmatic aspects to his curriculum, and the enthusiasm of everyone in the room.
Prior to attending the class I decided to read through Steve’s blog, and came across a post he wrote stressing the importance in all dealings of having a reality-based grasp of the price/value relationship. In this case Steve presents a scenario where a set of investors discounted the value of a new advisory board member and offered a price in stock that was too low, which caused them to lose a great opportunity.
It reminded me of how Bill Gates was able to capture overwhelming market share for Internet Explorer and almost entirely displace all competitors, significantly because he was able to see that the “correct” price for a browser was zero.
Here’s a textbook example of the scenario in reverse. My dad knew a test pilot who was offered a job testing a new exotic plane. He reviewed the situation, and determined at the risk level being offered, $100,000 was an appropriate market price, and put that price forth.
Regrettably for him, another pilot understood the market better and determined zero was the proper price. That other pilot’s name was Chuck Yeager, and the plane was the X-15.
I can’t remember the name of the pilot who asked for $100,000, and I guess that helps me make my point…
Harvard Professor Greg Mankiw wrote about the students who walked out of his introductory econ class because “the biased nature of Economics 10 contributes to and symbolizes the increasing economic inequality in America.”
He hit upon a few points that I’ve been meaning to post about. I’ve wanted to go into detail how those with a fundamentally unconstrained worldview will find most of the foundations of modern economic theory disconcerting.
The one to start off with is scarcity. They hit you with this one in your first Econ class.
From Wikipedia: Scarcity is the fundamental economic problem of having humans who have unlimited wants and needs in a world of limited resources. It states that society has insufficient productive resources to fulfill all human wants and needs. Alternatively, scarcity implies that not all of society’s goals can be pursued at the same time; trade-offs are made of one good against others. In an influential 1932 essay, Lionel Robbins defined economics as “the science which studies human behavior as a relationship between ends and scarce means which have alternative uses.”
While we have been steadily moving (largely thanks to embracing market-based economies) toward making more and more resources available to more and more people, not everything is available to everyone yet. While the obese now outnumber the hungry worldwide, and hunger is no longer an issue in America, (the Department of Agriculture in 2006 stopped using the word “hunger” in its reports), there’s a lot left to provide. Example? Up to 20 percent of American households defined as “poor” still do not have air conditioning.
Now of course the simple answer is to just have the government buy them air conditioners, but what about the people missing microwaves, cars, personal computers, internet access, or a house to call their own? Dialysis? Can we do it all? Shouldn’t people be able to vacation in Cabo? Don’t people want to go to Cabo? Yes they do. Want? Humans have unlimited wants. If you run the numbers you quickly hit the wall. Wants are greater than resources. Even rich countries who specifically try to satisfy the wants of their people bump up against scarcity.
Can’t the rich just pay for it? Let’s pencil it out. Take all the money Bill Gates has (56 Billion) and divide it up among the population (7 Billion). Everyone will get 8 bucks. A very cheap air conditioner is $400.00. You’ll need to confiscate all the wealth of 13 Bill Gates just to buy everyone one little box for a house of 4. Power it? Do you even have AC power? How many Bill Gates do we need to bring power to your village? Crap, an air conditioner costs a couple hundred dollars a year to power! We’re going to need a lot more rich guys to make all the good stuff happen. We aren’t even close to Cabo yet.
You get the idea.
Even the most unconstrained thinker I know had to finally/reluctantly agree that scarcity was real after he gave it a little thought. If we have scarcity, then we know resources have to be allocated somehow. If we have allocations, and some “get” and some don’t, we’re faced with a challenge we can’t just wish away.
Economics is the study of the allocation of scarce resources. Get over it kids.
Next episode: The inconvenience of gains from trade…
Ouch. This would appear to handily refute the non-intuitive argument Krugman made (and bloggers parrot) about the imagined efficiency of Medicare and Medicaid.
The official in charge of Medicare and Medicaid for the last 17 months says that 20 percent to 30 percent of health spending is “waste” that yields no benefit to patients, and that some of the needless spending is a result of onerous, archaic regulations enforced by his agency.
“…The government, unlike many private health insurance plans, is working in the daylight. That’s a strength.”
The official, Dr. Donald M. Berwick, listed five reasons for what he described as the “extremely high level of waste.” They are overtreatment of patients, the failure to coordinate care, the administrative complexity of the health care system, burdensome rules and fraud.
If his estimate is right, Medicare and Medicaid could save $150 billion to $250 billion a year by eliminating waste, which he defines as “activities that don’t have any value.”
(emphasis mine)
Some of us thought these efficiency arguments seemed fishy all along. A line I used to hear frequently from my professors was “intuitively, we know that…” One of the things we intuitively knew was that incentives matter (greatly). When you don’t have incentives for efficiency, you won’t get it. Market-based entities that can literally vanish from existence if they aren’t competitive have that incentive. Government agencies do not. Neither do banks that are “too big to fail.”
“Correspondingly, a large literature finds that when state-owned companies are privatized, efficiency improves dramatically, and a smaller company is required to produce the same level of output. Mueller (2003) lists 71 studies that compared the performance of state-owned companies: in only 5 of these studies did state-owned companies outperform their counterparts in terms of efficiency.”
MIT. 66 out of 71 studies. Ex post facto analysis. Common sense. Personal experience (I have worked in both government and private offices.) I call this case closed.
Robert Reich laments that workers can no longer purchase the goods they manufacture.
“For most of the last century, the basic bargain at the heart of the American economy was that employers paid their workers enough to buy what American employers were selling.”
Seems to me like the iPad is a poster child for this issue. Sadly, those Foxconn workers are not making the kind of money that provides an iPad compatible lifestyle.
Given this, I wondered what would happen if we just did the right thing and:
1) Repatriated manufacturing from China back to the USA
2) Paid the people who built them a fair wage
How would this work out? Luckily someone has already done most of the math. If we paid Americans the current $32.53 an hour wage (Average U.S. manufacturing/mining/construction compensation) for producing an iPad, Apple would need to charge $1,144.02 in order to keep their margins the same.
Problem is, Reich apparently doesn’t like the current prevailing wage. Wonder what an iPad would cost if we bumped that hourly up to a reasonable rate? Maybe $2,000?
Hey, works for me. All ya gotta do is make sure that sales don’t drop due to the price increase, and that all potential competitors worldwide agree not to be unscrupulous and take their manufacturing overseas as well. Could be a problem is somebody else comes up with a decent tablet that undercuts them significantly.
PS: This post made me nostalgic for the good old days. I remember when my dad worked at Boeing way back when. Still have those happy memories of picking up that shiny new 737 back in ’67 when it came out. Loved flying that classic around the block. Good times. Too bad those days are now likely lost forever thanks to the greed of Airbus.
Are you a Web data geek? Are you into scraping sites? If so, you may be one of the many people who rely on the highly-regarded Needlebase to help you in your efforts. 
It’s not just housing bubbles that are caused by good intentions and market intervention. 
Harvard Professor Greg Mankiw 
Robert Reich 
