The Yahoo Effect

Lukas Biewald and Chris Van Pelt of Dolores Labs wrote a fun application called FaceStat. This application lets its users evaluate each other based on their photos. Unlike its famous spiritual ancestor Hot or Not, in FaceStat each person can choose which criteria he or she wants to be evaluated on, e.g. “am I liberal or conservative”, “do I seem trustworthy”, etc.

Everything was sunshine and puppies until the day Yahoo decided to link to FaceStat from their front page, sending masses of new visitors to the site. The FaceStat server gave a small whimper, rolled on its back and played dead. Incensed Yahoos took the site’s downtime personally and resorted to stalking tactics: they found the email and phone number of the site’s registered owner (Chris Van Pelt), and left him angry emails and phone messages. It’s a tough racket, the web business.

Defunct TV

After some frantic work over the weekend to add hardware and streamline the software, FaceStat was back online and able to handle the load. And what was that load? According to their amazing Google Analytics chart, they jumped from 10,000 pageviews per day to 800,000! That’s not a hockey stick, that’s a space elevator.

So what happened? They fell victim to one of the classic dangers of the web. The most famous is the Slashdot Effect, which happens when a website is linked to from Slashdot. But only slightly less well-known (despite being more potent) is the Yahoo Effect. Although they managed to recover fairly quickly, they lost valuable visitors during the time that their site was still on the front page of Yahoo, but inaccessible.

Unfortunately, building an immunity to this kind of problem is usually not cost-effective. There are two options, and both of them have drawbacks.

First, you can buy enough hardware in advance to survive the Yahoo Effect. But if you never get that link from the front page of Yahoo then you will have wasted a lot of money.

Second, you can use Cloud Computing to enable your application to use additional servers when needed. In Cloud Computing, your application runs on a variable number of servers that are owned by someone else; you can add or remove servers at a moment’s notice. The poster boy for this kind of service is Amazon’s Elastic Compute Cloud (EC2). Since you can add resources almost instantly, your application can handle vastly increased loads when needed, and you pay only for the resources you actually require at any given moment. This is a very attractive proposition, and indeed a representative of cloud computing management company RightScale was quick to leave a comment on Lukas Biewald’s blog suggesting their services (thus demonstrating that ambulance chasing isn’t just for lawyers anymore).

Although cloud computing is cost-effective from a hardware point of view, it has a different cost: you must design your application in advance to use these resources. This requires additional development time, and that’s also an up-front cost. Given the relative costs of programmers and hardware, it might be cheaper to buy additional servers than rearchitect the application.

So what’s an internet entrepreneur to do? If you’re starting a new application then definitely look into cloud computing to help your application withstand traffic spikes. Designing a new application to use cloud computing is easier than retrofitting it into an existing application. Another option is to use Google App Engine, which is Google’s entry in the scalable web applications space. But that requires a significant commitment to do things the Google Way ™.

Or just do what most of us (including FaceStat) do: build your application as quickly as possible, and worry about the traffic when you get it. It’s the time-honored way: people won’t respect you unless you’ve got war stories about overcoming vast amounts of traffic with nothing but a screwdriver and a SCSI differential cable.

Update – June 14, 2008

Eran Hammer-Lahav spent two years building Nouncer, a Twitter-like service, before deciding to shut down the project. One of his lessons from this experience is:

Many people criticize the typical path Web 2.0 applications take in their development: putting together a poorly executed site, gauging the market, and only upon success building the service to actually scale and accommodate the market. However, the cost of building scalability ahead of time is extremely high, and for most startup is cost prohibitive.

(Photo by Robbt)