i202 Fall 2010 School of Information, UC Berkeley

Google recently revealed an overhaul of its back-end web indexing infrastructure, called Caffeine, making search results “50 percent fresher”. The old system was split into layers that did a series of batch processes on new Web content. Caffeine is able to add new pages or changes to existing pages directly into the index, without having to completely rebuild it. Many of the new technologies that make up Caffeine helped make Google Instant a reality.

The amount of information available on the Web is no less daunting than that of the “Library of Babel” as described by Borges. Google’s databases occupy “nearly 100 million gigabytes” and grows at a rate of “hundreds of thousands of gigabytes per day”. Luckily for us the indexing is automated and Google’s algorithms have been highly tuned to return relevant results, saving hapless Internet users from getting lost and throwing themselves over the edge as Borges feared.

More technical explanation of what's happening here: http://www.theregister.co.uk/2010/09/09/google_caffeine_explained/