Cookies are for Closers: Oren Hurvitz's Blog

Google Improves Privacy, Petulantly

2008-09-12T07:14:12Z

Google have announced that they’ll reduce the amount of time that they keep individually-identifiable information about searches from 18 months to 9 months. I would like to think that my previous post on this topic played a small part in this decision, but it looks like it was mostly due to pressure from the European Union.

In that post, I showed a request that I had sent to Google, asking them to reduce the amount of time they keep private data to just one month. I asked everyone who read the post to send a similar request to Google, and judging from the comments some people did so. For the record, I did receive a reply from Google, but it was just a standard email that didn’t actually address my points: it just reiterated their position on data retention. In fine Google tradition, it would appear that no human was involved in sending that response.

Although I would like to see Google reduce the retention period further, to one month, this is a big step in the right direction. Google deserves credit for listening to the public and changing their practices. It is therefore unfortunate that they chose to pepper this announcement with vague threats:

“Back in March 2007, Google became the first leading search engine to announce a policy to anonymize our search server logs in the interests of privacy. [...] Although that was good for privacy, it was a difficult decision because the routine server log data we collect has always been a critical ingredient of innovation.”
“When we began anonymizing after 18 months, we knew it meant sacrifices in future innovations in all of these areas [search quality, security, fighting fraud and reducing spam]. We believed further reducing the period before anonymizing would degrade the utility of the data too much and outweigh the incremental privacy benefit for users.”
“While we’re glad that this will bring some additional improvement in privacy, we’re also concerned about the potential loss of security, quality, and innovation that may result from having less data.”

What on earth could they mean?

Translation #1: “Ok world, you win, we’ll keep the data for less time. But you’re going to be sorry!”

Translation #2: “We’ve now reduced our retention period as far as humanly possible, and then some. Please don’t make us reduce it any more!”

Google are keeping this discussion (of how long to keep the data) at a superficial level: they throw a number (”18 months”), the European Union throws a number (”6 months”?), I throw a number (”1 month”). You, too, can become a highly respected privacy advocate by coming up with your own number (that no one else has claimed yet) and writing about it!

A more substantive discussion would require Google to reveal some of their cards: how much of a benefit to fraud protection do they derive from keeping this data for 9 months (vs. a shorter length of time)? How do 9 months of individually-identifiable information help them improve their algorithms vs. 1 month of such information, especially given that they will always have an unlimited amount of anonymized data?

Of course, Google will never reveal this information because it would hurt their competitive position. But an experienced programmer, well-versed in the art, can make some reasonable guesses.

Individually-identifiable information is most important for security, fraud prevention, and fighting spam. But since these are time-sensitive tasks, the information quickly loses its value. I believe that the residual value of this information is close to zero after a few weeks have passed.

The other use for this data, improving search quality, can be handled with anonymized data for the most part. One example that Google commonly give is their automatic spell checker. But they don’t need individually-identifiable information in order to figure out that people who search for “brittaney” really mean “britney”. Yes, I can envision some types of search quality improvements that would benefit from studying individually-identifiable information, but they are a minority, and Google can learn how to do that while keeping data for a shorter period of time. I therefore stand by my position that 1 month of private data would strike the right balance between privacy and security/fraud prevention/spam detection/search quality.

(Photo by lesprit_descalier)

Lookin’ For Tall Kids

2008-08-19T21:14:34Z

In the towns, in fields, in basketball courts, the talent scouts — looking for runners, looking for tall kids who can become championship sprinters.

Hey mister, that’s my kid over there! See him? He’s the one you’re looking for. Look how fast he is, he’ll break the world record.

No way, too short. Looks like he won’t be more’n five-ten, five-eleven. That doesn’t cut it anymore. Usain Bolt is six-five. That’s what I want. Give me ten tall kids, and I’ll find a champion.

Wait, wait! Tyson Gay is five-eleven. Maurice Greene is just five-nine. And they won plenty of races!

That’s ancient history. We seen Bolt at the 2008 Olympics. Ran the 100 meters, set a new world record, 9.69 seconds. First man under 9.70, and he didn’t even try hard. With those big strides, he overtook ever’one else, then slowed down at the end to wave at the crowd and beat on his chest. Now that’s a champion! All those stubby runners behind him looked like children. No, short runners are the past; tall ones are hot. I’m looking here today, tomorrow I’ll be in Oklahoma City.

God, if I could only get a hundred tall kids. I don’t care if they run or not. We’ll take ‘em, train ‘em, see how fast they can go. Ninety-nine kids out of a hundred won’t be good enough. Tall runners always have problems: their reaction time is slow, they can’t move their feet fast enough. That’s why everyone used to think that sprinters can’t be tall. But that all changed after we saw Bolt run in the Olympics. Now everyone wants tall runners, runners with the potential to be the next Usain Bolt.

It’s like we always thought automobiles should be big, as big as you can get ‘em, and then one day we’re told that small automobiles are better. Everyone’s dumping their big trucks and gettin’ small cars. Jus’ like that, the world turned upside down. Well, that’s how it is with runners now. We don’t want them short; short runners are finished. Only tall ones. Can’t find tall adult runners; we would have washed them out long ago. Gotta get ‘em young, train a whole new generation of runners. When we tell our grandchildren we saw runners under six feet win gold medals they ain’t gonna believe us. Christ, if I could only get five hundred tall kids! Find ‘em, sign ‘em up, get them to a trainer, collect the finder’s fee! If I had enough tall kids I’d retire in six months.

What do you mean, you don’t want to run? Now, look here. I’m tryin’ to help you become a star, and you took all this time. I might a signed three kids while I been talkin’ to you. I’m disgusted. Yeah, sign right here, and get your parents to sign over there. I’ll send you the details later.

Jesus, I wisht I had a thousand tall kids!

(With apologies to John Steinbeck)

(Photo by puck90)

Man Being Eaten By Alligator

2008-07-09T21:35:22Z

The Wilhelm Scream (originally called “Man being eaten by alligator”) is an inside joke in Hollywood: it’s a short but distinctive scream that has been used in hundreds of movies. Someone with too much time on his hands compiled this video with clips of many of the movies where the scream appears:

The more you watch this video the funnier it gets. Villains scream it, usually while flying through the air. Animated characters scream it. Every Star Wars movie features the scream, as do Star Wars spoofs. Sometimes even heroes scream it. It gets so ridiculous you want to scream it yourself. Oiaaaaargh! The next time you host a party, have a Wilhelm Scream Competition. The winner gets a Lacoste shirt.

Compare the Wilhelm Scream to Howard Dean’s famous scream during the 2004 Democratic presidential primaries, the one that ended his chances. He really nailed it! (Turn up the volume for full effect.)

(Via Freakonomics)

One Month is Enough

2008-09-12T08:51:46Z

No Photos! (photo by vinnie bezoomny)

Dear friends,

The first rule of crisis management is to get ahead of the story. Since my shameful secret is about to be revealed, I decided to break it here first. I’d rather you heard it from me than from the media:

In March 2008 I watched Rick Astley’s music video Never Gonna Give You Up on YouTube. It’s widely considered to be the most corny music video ever created. I have no excuse; I can’t even claim to have been RickRolled. I heard about the video, and willingly went and viewed it. It was me, just me, officer!

The reason for this confession is that Google is about to hand over to Viacom a complete list of every video watched by YouTube users:

[...] the judge granted a Viacom motion that records of every video watched by YouTube users, including their login names and IP addresses, be turned over to the entertainment giant.

The order prevents Viacom from using this information to target lawsuits at users. But it makes no sense to give this information to Viacom in the first place: Google could easily make this data anonymous, and they’ve asked Viacom to do just that. Viacom have said that they won’t use any personally identifiable data, but they haven’t replied to Google’s request directly. These mixed signals make me lunge for my tin foil hat: what could explain Viacom’s behavior? Perhaps, once they have the logs in their possession, they intend to ask the judge to allow them greater use of the data. Or perhaps the data will be “accidentally” leaked — after all, that sort of thing happens all the time.

But criticizing a media company like Viacom for ignoring users’ privacy is like berating a toddler for getting food all over themselves: it’s in their nature, and they’re going to keep doing it. Let’s beat up on Google instead, that never gets old. Google shouldn’t have kept this data around for Viacom to subpoena. Google deletes personally identifiable user data after 18 months, which isn’t enough to hide my Rick Astley obsession. Google’s track record on privacy is spotty in general. For example, after a lot of pressure they finally added a link to their privacy policy on the Google homepage in July 2008, not before bitching and moaning like a teenager whose parents have forced him to clean his room.

Google has some of the most sensitive data in the world; in particular, they know every search that a user makes. In their Privacy FAQ they list several good reasons why they need to keep this data:

To improve search results
To maintain the security of their systems
To prevent fraud and other abuses

It’s true that in order to achieve these goals Google needs to save the search logs. However, the problem isn’t that they keep the search logs; it’s that they keep personally identifiable information in the logs, which lets them (or anyone else, such as Viacom) associate searches and clicks with real people. Google keeps this information for 18 months, and that’s far too long. They could erase the personal information much sooner and still achieve all of the goals described above.

For example, Google use the search logs to find common spelling mistakes made by users, so that they can offer automatic suggestions for the correct spelling. This doesn’t require any personally identifiable information. Another use for the search logs is to detect click fraud. For this purpose it is indeed useful to look at the search and click history of individual users. However, the benefit of this personal data quickly diminishes with time. Data about click fraud that is over a month old should be considered prehistoric; the perpetrators are long gone from whatever IP they had been using.

Private Property (photo by Zervas)

Google’s privacy policy doesn’t say how long they keep search logs; probably forever. The only promise they make is to scrub out personally identifiable information after 18 months. Google are very vague about where this figure of “18 months” comes from; perhaps it has some religious significance. From Google’s Privacy FAQ:

Why are logs kept for 18 months before being anonymized?

We strike a reasonable balance between the competing pressures we face, such as the privacy of our users, the security of our systems and the need for innovation. We believe 18 months strikes the right balance.

It’s time we told Google: 18 months is too long. One month would strike the right balance between privacy, security and the need for innovation. With one month of personally identifiable information, Google will be able to catch all the fraud they are ever likely to catch. After that, it’s time to anonymize the data. The anonymized data is still useful for improving their search engine.

Go to Google’s Privacy Feedback page and ask them to reduce the amount of time they keep personally identifiable data in their logs. You could use a message such as this one:

Dear Google,I’m concerned about your data retention policy: you keep user identifiable information in your search logs for 18 months, and that’s too long. As we have seen with the recent lawsuit by Viacom, this information can easily fall into the hands of third parties. To protect my privacy and the privacy of the rest of your users, please reduce the amount of time you keep personally identifiable data to one month. Thank you.

Google isn’t alone in this. Microsoft also anonymizes its logs after 18 months. Yahoo makes do with just 13 months (how did they come up with that number? Perhaps it also holds occult significance). Ask.com, the fourth-largest search provider, gives its users the option of making completely anonymous searches. But we should focus on Google: where the market leader goes, the rest will surely follow.

Apple’s Engineers: an Unexpected Profit Center

2008-06-17T20:07:29Z

According to salary information collected by new startup Glassdoor, Apple pays its engineers significantly less than competing companies in Silicon Valley. Apple engineers make $89,000 a year, whereas Google engineers can buy four more Segways a year (pre-tax) with their $112,573 paycheck. Microsoft and Yahoo are closer to Google: both companies pay their engineers $105,000 a year. See TechCrunch’s review of Glassdoor for the data.

I wondered how much of a difference this salary disparity made to Apple’s bottom line, so I took a look at its annual 10-K filings from 2003 to 2007. Each of these reports includes, buried among its 170 pages, Apple’s net income and how much it spent on R&D. For simplicity I assumed that the R&D budget was entirely spent on salaries; this isn’t far off the mark in a hi-tech company like Apple.

If Apple were to pay its engineers the same salaries as Google then its R&D budget would increase by 26%. This amount (26% of the R&D budget) is how much Apple saves each year by paying below-market salaries. I calculated what Apple’s net income would have been if it had paid its engineers the same as Google, and these are the results:

Explanation:

All dollar values are in millions.
# Employees - from Apple’s 10-K.
R&D Budget - from Apple’s 10-K.
Adjusted R&D Budget - had Apple paid its engineers at the same level as Google, this would have been its R&D Budget.
Net Income - from Apple’s 10-K.
Adjusted Net Income - had Apple paid its engineers at the same level as Google, this would have been its Net Income.
Increase in Net Income - the magnitude by which Apple’s net income was higher that year compared to what it would have been had it paid salaries at the same level as Google.

The Adjusted Net Income is a good estimate, but it’s not completely accurate. For example, the increase in Apple’s R&D Budget would have meant that its expenses are higher, so it would have paid less taxes. But the overall trend is clear.

Here’s the Increase in Net Income in chart form:

In 2003 and 2004, the effect of underpaying its engineers made a huge difference to Apple’s bottom line. In 2003, these savings turned around Apple’s year: from a loss to a small profit. In 2004, they doubled the profit. However, once Apple’s earnings began to skyrocket in 2005, the effect of the R&D savings became much smaller: just 6% of the net income in 2007, for example.

Paying low salaries to its engineers was a lifesaver for Apple during its difficult times. But now that Apple is immensely profitable there’s no more excuse for this practice. In the TechCrunch article mentioned previously, the site’s owner Michael Arrington says: “Apple software engineers make only about $89,000, on average, but they get to create some of the most loved products on Earth.” I’m sure this warms their hearts. But an extra $20,000 a year would make their hearts downright toasty, and their spouses’ as well.

The Yahoo Effect

2008-09-12T08:36:06Z

Lukas Biewald and Chris Van Pelt of Dolores Labs wrote a fun application called FaceStat. This application lets its users evaluate each other based on their photos. Unlike its famous spiritual ancestor Hot or Not, in FaceStat each person can choose which criteria he or she wants to be evaluated on, e.g. “am I liberal or conservative”, “do I seem trustworthy”, etc.

Everything was sunshine and puppies until the day Yahoo decided to link to FaceStat from their front page, sending masses of new visitors to the site. The FaceStat server gave a small whimper, rolled on its back and played dead. Incensed Yahoos took the site’s downtime personally and resorted to stalking tactics: they found the email and phone number of the site’s registered owner (Chris Van Pelt), and left him angry emails and phone messages. It’s a tough racket, the web business.

After some frantic work over the weekend to add hardware and streamline the software, FaceStat was back online and able to handle the load. And what was that load? According to their amazing Google Analytics chart, they jumped from 10,000 pageviews per day to 800,000! That’s not a hockey stick, that’s a space elevator.

So what happened? They fell victim to one of the classic dangers of the web. The most famous is the Slashdot Effect, which happens when a website is linked to from Slashdot. But only slightly less well-known (despite being more potent) is the Yahoo Effect. Although they managed to recover fairly quickly, they lost valuable visitors during the time that their site was still on the front page of Yahoo, but inaccessible.

Unfortunately, building an immunity to this kind of problem is usually not cost-effective. There are two options, and both of them have drawbacks.

First, you can buy enough hardware in advance to survive the Yahoo Effect. But if you never get that link from the front page of Yahoo then you will have wasted a lot of money.

Second, you can use Cloud Computing to enable your application to use additional servers when needed. In Cloud Computing, your application runs on a variable number of servers that are owned by someone else; you can add or remove servers at a moment’s notice. The poster boy for this kind of service is Amazon’s Elastic Compute Cloud (EC2). Since you can add resources almost instantly, your application can handle vastly increased loads when needed, and you pay only for the resources you actually require at any given moment. This is a very attractive proposition, and indeed a representative of cloud computing management company RightScale was quick to leave a comment on Lukas Biewald’s blog suggesting their services (thus demonstrating that ambulance chasing isn’t just for lawyers anymore).

Although cloud computing is cost-effective from a hardware point of view, it has a different cost: you must design your application in advance to use these resources. This requires additional development time, and that’s also an up-front cost. Given the relative costs of programmers and hardware, it might be cheaper to buy additional servers than rearchitect the application.

So what’s an internet entrepreneur to do? If you’re starting a new application then definitely look into cloud computing to help your application withstand traffic spikes. Designing a new application to use cloud computing is easier than retrofitting it into an existing application. Another option is to use Google App Engine, which is Google’s entry in the scalable web applications space. But that requires a significant commitment to do things the Google Way ™.

Or just do what most of us (including FaceStat) do: build your application as quickly as possible, and worry about the traffic when you get it. It’s the time-honored way: people won’t respect you unless you’ve got war stories about overcoming vast amounts of traffic with nothing but a screwdriver and a SCSI differential cable.

Update - June 14, 2008

Eran Hammer-Lahav spent two years building Nouncer, a Twitter-like service, before deciding to shut down the project. One of his lessons from this experience is:

Many people criticize the typical path Web 2.0 applications take in their development: putting together a poorly executed site, gauging the market, and only upon success building the service to actually scale and accommodate the market. However, the cost of building scalability ahead of time is extremely high, and for most startup is cost prohibitive.

(Photo by Robbt)

Anatomy of a Con

2008-09-12T08:34:07Z

This is the tale of how I was conned at a conference. (As far as alliterative woes are concerned, I could have done worse: I could have been shafted at a shindig. Hoodwinked at a hootenanny. Mauled at a meal. You get the picture.)

Amsterdam, June 2000. The conference was about WAP. Do you remember WAP? It was an attempt to rewrite the entire web infrastructure from scratch for mobile phones. Instead of HTML we were supposed to use WML: a markup language which is almost, but not quite, entirely unlike HTML. WAP flopped, but not before dumping a sediment of useless software on every mobile phone, and an 800-page tome in my suitcase (it was given away at the conference).

But I didn’t care about any of that in 2000. This was the dot-com era before the bubble burst, the weather was sunny and Amsterdam beautiful. After the conference ended I had some time to walk around Amsterdam and take in the canals, the bikes, and the coffee shops. The next day I took a train to the airport, and that’s when I was conned and relieved of my briefcase, passport, plane ticket, camera, and various other items (but sadly, not the huge book).

Con Man

It was mid-morning, and the train was almost empty. I had an entire car to myself at first. After a few stops one other guy came in and sat across the aisle from me. He seemed quite ordinary: in his 30’s, some stubble, no distinguishing characteristics. He asked me something trivial about the stops that the train will make, but mostly just looked out the window and fiddled with his prepaid phone cards. (A note to my younger readers: in Ye Olden Days, before everyone had cellphones, people made calls using public phone booths. Phone cards were used to pay for these calls.)

A couple of stops before the airport Phone Card Guy jumped up as if he’d just noticed that this is his stop, and hurried out, dropping a few of his phone cards in his haste. I looked at the cards on the floor, and then around the train. There was no one else there. So I picked up the cards, went to the door of the train and shouted after him, “you dropped your phone cards!” Phone Card Guy was already some distance away from the train, but he came back and took the cards, thanked me, and walked away. While this was happening, a passenger that I hadn’t seen before came behind me and left the train through the doorway I was standing in. He looked like a businessman: he wore a suit, and was in his 50’s.

I returned to my seat, and the train started moving again. It was then that I noticed that my briefcase and camera were gone from the seat where I’d left them, and in a flash I realized what had happened.

In con movies, at this point we would see a quick succession of scenes from earlier in the movie, explaining how the con was put together and making us see everything in a different light. This is how it worked: Phone Card Guy established rapport with me, so that I’ll be motivated to go to the door of the train and tell him that he dropped his phone cards. Suit Guy was his accomplice: his job was to lurk one car over and watch to see when I had left my seat and had my back turned. At that point Suit Guy came into the car, grabbed what he could, and left through the same door I was standing at! Phone Card Guy had gone one way and Suit Guy the opposite way, so I was looking in the wrong direction and didn’t notice that Suit Guy was holding my briefcase. This was all timed so that the train started moving just as I realized what happened, so I couldn’t run after them or call for help.

I was so full of admiration for their smooth technique that I almost didn’t mind losing my stuff. Fortunately there was enough time for me to get replacement travel documents at the airport. They didn’t issue me a new passport on the spot, of course: instead they had me travel with the sort of papers that are normally used to transport pets. Wuf!

What I regret most is the loss of my camera, with its photos of Amsterdam. I hope the con men liked them.

LinkedIn Architecture

2008-06-09T21:34:15Z

At JavaOne 2008, LinkedIn employees presented two sessions about the LinkedIn architecture. The slides are available online:

These slides are hosted at SlideShare. If you register then you can download them as PDF’s.

This post summarizes the key parts of the LinkedIn architecture. It’s based on the presentations above, and on additional comments made during the presentation at JavaOne.

Site Statistics

22 million members
4+ million unique visitors/month
40 million page views/day
2 million searches/day
250K invitations sent/day
1 million answers posted
2 million email messages/day

Software

Solaris (running on Sun x86 platform and Sparc)
Tomcat and Jetty as application servers
Oracle and MySQL as DBs
No ORM (such as Hibernate); they use straight JDBC
ActiveMQ for JMS. (It’s partitioned by type of messages. Backed by MySQL.)
Lucene as a foundation for search
Spring as glue

Server Architecture

2003-2005

One monolithic web application
One database: the Core Database
The network graph is cached in memory in The Cloud
Members Search implemented using Lucene. It runs on the same server as The Cloud, because member searches must be filtered according to the searching user’s network, so it’s convenient to have Lucene on the same machine as The Cloud.
WebApp updates the Core Database directly. The Core Database updates The Cloud.

2006

Added Replica DB’s, to reduce the load on the Core Database. They contain read-only data. A RepDB server manages updates of the Replica DB’s.
Moved Search out of The Cloud and into its own server.
Changed the way updates are handled, by adding the Databus. This is a central component that distributes updates to any component that needs them. This is the new updates flow:
- Changes originate in the WebApp
- The WebApp updates the Core Database
- The Core Database sends updates to the Databus
- The Databus sends the updates to: the Replica DB’s, The Cloud, and Search

2008

The WebApp doesn’t do everything itself anymore: they split parts of its business logic into Services.
The WebApp still presents the GUI to the user, but now it calls Services to manipulate the Profile, Groups, etc.
Each Service has its own domain-specific database (i.e., vertical partitioning).
This architecture allows other applications (besides the main WebApp) to access LinkedIn. They’ve added applications for Recruiters, Ads, etc.

The Cloud

The Cloud is a server that caches the entire LinkedIn network graph in memory.
Network size: 22M nodes, 120M edges.
Requires 12 GB RAM.
There are 40 instances in production
Rebuilding an instance of The Cloud from disk takes 8 hours.
The Cloud is updated in real-time using the Databus.
Persisted to disk on shutdown.
The cache is implemented in C++, accessed via JNI. They chose C++ instead of Java for two reasons:
- To use as little RAM as possible.
- Garbage Collection pauses were killing them. [LinkedIn said they were using advanced GC's, but GC's have improved since 2003; is this still a problem today?]
Having to keep everything in RAM is a limitation, but as LinkedIn have pointed out, partitioning graphs is hard.
[Sun offers servers with up to 2 TB of RAM (Sun SPARC Enterprise M9000 Server), so LinkedIn could support up to 1.1 billion users before they run out of memory. (This calculation is based only on the number of nodes, not edges). Price is another matter: Sun say only "contact us for price", which is ominous considering that the prices they do list go up to $30,000.]

The Cloud caches the entire LinkedIn Network, but each user needs to see the network from his own point of view. It’s computationally expensive to calculate that, so they do it just once when a user session begins, and keep it cached. That takes up to 2 MB of RAM per user. This cached network is not updated during the session. (It is updated if the user himself adds/removes a link, but not if any of the user’s contacts make changes. LinkedIn says users won’t notice this.)

As an aside, they use Ehcache to cache members’ profiles. They cache up to 2 million profiles (out of 22 million members). They tried caching using LFU algorithm (Least Frequently Used), but found that Ehcache would sometimes block for 30 seconds while recalculating LFU, so they switched to LRU (Least Recently Used).

Communication Architecture

Communication Service

The Communication Service is responsible for permanent messages, e.g. InBox messages and emails.

The entire system is asynchronous and uses JMS heavily
Clients post messages via JMS
Messages are then routed via a routing service to the appropriate mailbox or directly for email processing
Message delivery: either Pull (clients request their messages), or Push (e.g., sending emails)
They use Spring, with proprietary LinkedIn Spring extensions. Use HTTP-RPC.

Scaling Techniques

Functional partitioning: sent, received, archived, etc. [a.k.a. vertical partitioning]
Class partitioning: Member mailboxes, guest mailboxes, corporate mailboxes
Range partitioning: Member ID range; Email lexicographical range. [a.k.a. horizontal partitioning]
Everything is asynchronous

Network Updates Service

The Network Updates Service is responsible for short-lived notifications, e.g. status updates from your contacts.

Initial Architecture (up to 2007)

There are many services that can contain updates.
Clients make separate requests to each service that can have updates: Questions, Profile Updates, etc.
It took a long time to gather all the data.

In 2008 they created the Network Updates Service. The implementation went through several iterations:

Iteration 1

Client makes just one request, to the NetworkUpdateService.
NetworkUpdateService makes multiple requests to gather the data from all the services. These requests are made in parallel.
The results are aggregated and returned to the client together.
Pull-based architecture.
They rolled out this new system to everyone at LinkedIn, which caused problems while the system was stabilizing. In hindsight, should have tried it out on a small subset of users first.

Iteration 2

Push-based architecture: whenever events occur in the system, add them to the user’s "mailbox". When a client asks for updates, return the data that’s already waiting in the mailbox.
Pros: reads are much quicker since the data is already available.
Cons: might waste effort on moving around update data that will never be read. Requires more storage space.
There is still post-processing of updates before returning them to the user. E.g.: collapse 10 updates from a user to 1.
The updates are stored in CLOB’s: 1 CLOB per update-type per user (for a total of 15 CLOB’s per user).
Incoming updates must be added to the CLOB. Use optimistic locking to avoid lock contention.
They had set the CLOB size to 8 kb, which was too large and led to a lot of wasted space.
Design note: instead of CLOB’s, LinkedIn could have created additional tables, one for each type of update. They said that they didn’t do this because of what they would have to do when updates expire: Had they created additional tables then they would have had to delete rows, and that’s very expensive.
They used JMX to monitor and change the configuration in real-time. This was very helpful.

Iteration 3

Goal: improve speed by reducing the number of CLOB updates, because CLOB updates are expensive.
Added an overflow buffer: a VARCHAR(4000) column where data is added initially. When this column is full, dump it to the CLOB. This eliminated 90% of CLOB updates.
Reduced the size of the updates.

[LinkedIn have had success in moving from a Pull architecture to a Push architecture. However, don't discount Pull architectures. Amazon, for example, use a Pull architecture. In A Conversation with Werner Vogels, Amazon's CTO, he said that when you visit the front page of Amazon they typically call more than 100 services in order to construct the page.]

The presentation ends with some tips about scaling. These are oldies but goodies:

Can’t use just one database. Use many databases, partitioned horizontally and vertically.
Because of partitioning, forget about referential integrity or cross-domain JOINs.
Forget about 100% data integrity.
At large scale, cost is a problem: hardware, databases, licenses, storage, power.
Once you’re large, spammers and data-scrapers come a-knocking.
Cache!
Use asynchronous flows.
Reporting and analytics are challenging; consider them up-front when designing the system.
Expect the system to fail.
Don’t underestimate your growth trajectory.

Run for office with Contendr

2008-09-12T08:32:43Z

The Obama compaign is hiring developers to create software for his presidential campaign. It was suggested to make this software open-source. But why stop there? Whenever a successful website comes along, someone invariably creates a service that lets anyone churn out a clone in five minutes:

Want your own social network? Ning.
Your own Digg? coRank.
A Wiki to call your own? Wetpaint.
Want to show the Twitter folks how to keep a site running? ReVou.
Starting a presidential campaign? Contendr!

He coulda been a contender, if only he'd had Contendr. (Marlon Brando in "On the Waterfront")

Suggested features:

Collect signatures to get the candidate’s name on the ballot by harvesting .sig’s from Slashdot and other forums.
Ask for campaign contributions with a tip jar on the website.
Spread the candidate’s message by link-spamming the appropriate sites: Instapundit for Republicans or Daily Kos for Democrats. Actually, link-spam both sites; everyone deserves to hear what you’ve got to say.

The name is available (but sadly, the domain is not). Act now, and help democratize the democratic process!

JavaOne 2008

2008-09-12T08:28:22Z

I was at JavaOne 2008 this month. I attended mostly server-related sessions, with a few J2SE and J2ME presentations thrown in to see how the other half lives. This is my photo report.

Moscone Center

JavaOne is held at Moscone Center in downtown San Francisco.

It was crowded!

People queued up outside the rooms until they were allowed to enter (about ten minutes before the session’s start time). This was surprisingly well-organized: Moscone Center staff stood outside the rooms with large signs containing the room’s number and made sure everyone stood in the right line. The Esplanade was especially crowded, so the staff took to standing at the end of the line, holding up their sign and shouting their room number every few seconds.

There was a space with bean bags and video games, which is apparently de rigeur in any geek gathering these days.

Men outnumbered women at least 15:1. This led to an unlikely scene.

The General Sessions were held in a huge auditorium.

The regular sessions were in smaller rooms. This guy utilized the time before the session to the fullest.

Smash Mouth concert

On Thursday night there was a Smash Mouth concert in the adjacent Yerba Buena Gardens.

Vringo BOF Session

My company, Vringo, presented a BOF called Real-World Challenges in Signing Java Platform, Micro Edition (Java ME Platform) Applications. The reason the title is so unwieldy is that Sun enthusiastically replaced the text in all the presentations with the One True Marketing Terminology. This turned a simple title such as “Real-World Challenges in Signing J2ME Applications” into the monstrosity above.

This photo shows our CTO, David Goldfarb (right), and mobile developer Chaim Kutnicki (left).