Downloading the Web in an Afternoon

That's the equivalent of what Caltech recently accomplished.

Bandwidth is a measure of how much data can be sent across a network in a set amount of time. It's what you experience online when waiting for a web page to load.

But recently, Caltech reached a stunning milestone in how fast data can be sent, transferring 475 terabytes of data in 24 hours. That's fast enough to download the entire Web as indexed by Google in an afternoon. (Cool idea too. Can you imagine having your own personal cache of the web, updated every couple of hours in the background while you surf with no page load waits?)

OK, for math heads, here's the figures I used rounded to the nearest sanest numbers:

475 terabytes (the amount of data transferred in 1 day by Caltech), divided by 156 terabytes (the size of Google's web) = (roughly) 3.

More math? OK here is how I determined Google's size:

I did a Google search with my preferences set to return 100 pages. I searched for everything using *.* (Note: This search query doesn't work anymore. Use a creative seach of your own with common words like "home" or "welcome").
From those 100 hits, I added up the size of each page for a total of:
1721 k
and then divided that by 100 to get the average size of each page.
That gave me an average of 17 K per page
Multiply that by the number of pages on Google (9,180,000,000)
and that gives 156,060,000,000,000 Bytes or 156 terabytes.

How to Visualize Data

Did you know that 1 CD can hold the entire Human genome? That a Gigabyte of text printed out would fill the bed of a pickup truck with paper? That it takes 20,000 trees to print out 400 Gigabytes on paper? That 100 Terabytes is a high-side guess of Human brain storage space, slightly less than one days Web Page views from Google in 2009.

I've been collecting these data bits for a couple of years now, finding gems and doing my best math to see if they fall in line with everything else. It's pretty close. Close enough to give you some useful comparisons.

Bits

2 bits: any 2 choice decision. Yes/No Run/Stop...
3 bits: A color pixel. Only 3 bits are needed to represent any colour. Red, Blue and Green, from which all other colors are derived.
8 bits: 1 byte

Bytes

1 byte: Any character on your computer keyboard (not including your cat).
5 bytes: The average English word

Kilobyte

1,024 bytes often rounded to 1,000 bytes
1 Kilobyte: A joke or a couple of paragraphs
2 Kilobytes: Typewritten page
3.2 Kilobytes: The amount of data in the H1N1 Swine Flu virus
3.5 Kilobytes: The size of the first web page
5 Kilobytes: A Desktop Icon
10 Kilobytes: A page out of an encyclopedia
17 Kilobytes: The size of an average Web Page
50 Kilobytes: A (roughly) 4 by 6 inch image
100 Kilobytes: A low-resolution photograph
750 kilobytes: The file size necessary to categorize the entire range of human experience and interest. (11 categories and about 450 unique sub-categories) as indicated by the Yahoo! directory on November 3rd 2007. (I wrote a small program to fetch all the headings and categories, save it to a text file, then view the text file size.)

Megabyte

1,048,576 bytes,
1 Megabyte: Small novel, 3-1/2 inch diskette
2 Megabytes: 12 Megapixel Digital Photo, high resolution
3 Megabytes: The average mp3 song. (Rough rule of thumb: an mp3 plays about 1 megabyte per minute.)
4 Megabytes: A Non illustrated King James Bible. I downloaded a text version from Project Gutenberg and viewed the file size).
5 Megabytes: 1 minute of a YouTube High Quality video.
10 Megabytes: (Roughly) 1 minute MPEG movie.
20 Megabytes: Typical hard drive in the first desktop PCs
100 Megabytes: Roughly the text info contained in 1 meter (3 feet) of bookshelf
750 Megabytes: 1 CD. The Human genome (props to Max for his comment below)

Gigabyte

1,073,741,824 bytes,
1 Gigabyte: The bed of a pickup truck filled with paper.
7 Gigabytes: 1 DVD
10 Gigabytes: A 1 inch stack of CD's
28 Gigabytes: Tweets Per Day on Twitter as of Jan 1, 2011
30 Gigabytes: From My Life In A Terabyte - Roughly the entire collection of Gordon Bell's Gordon Bell's articles, books, correspondence (letters and email), CD's, memos, papers, photos, pictures, presentations, home movies, videotaped lectures, and voice recordings by 2003.
400 Gigabytes: 20,000 trees made into paper and printed.
500 Gigabytes: 100 DVD Movies

Terabyte

1 Terabyte: 1000 gigabytes
An 8 foot stack of CD's or about 150 DVD's. It would hold all 350 episodes of The Simpsons or all 238 episodes of Friends. About 2 years non stop MP3s. About 50,000 trees made into paper would be needed to print out a Terabyte of data. 250 million pages printed both sides, over 10 miles high. Roughly 250,000 MP3s (2 years non-stop listening). About 2 weeks of non-stop DVD movies. 500,000 digital camera pictures.
4 Terabytes: The YouTube record of U.S. user names and IP addresses including every record of every video watched by them as of 2008.
10 Terabytes: Enough to store everything you look at for a year, and could include a heart monitor, personal GPS, everything you type and every move of your mouse. From Charlie's Diary Shaping the Future.
45 Terabytes: All the videos on YouTube as of Aug 2006
100 Terabytes: High guess of Human brain storage space. The monthly growth of The Internet Archive in 2009. From this Google search
122 Terabytes: The size of one days Web Page views from Google in 2009 (7.2 Billion daily page views) X 17 Kilobytes (the size of the average web page).
150 Terabytes: Estimated size of all Web pages indexed by Google on Dec 8th 2005 (not including databases or video). (See this article for the figures I used)

Petabyte

1 Petabyte: 1 thousand Terabytes. Storage at this level signals the dawn of a new era with powerful implications to the sciences and Artificial Intelligence.
About 100 years of television. The amount of data storage space the Internet Archive had in 2004.
Roughly the amount of new video added to youTube every day in 2007
A stack of CD's 3 kilometers high
2 Petabytes: The amount of data Google processed every day in 2008
3.5 Petabytes: 2007 Estimated capacity of Google's Data centers in a box.
4 Petabytes: Estimated amount of Internet data stored in RAM by Google in 2006.
4.5 Petabytes: The capacity of The Internet Archive as of 2009. From this Google search
15 Petabytes: The amount of data the Large Hadron Collider (LHC) generates per year as of 2008.
20 Petabytes: Google daily total workload in January 2008. The storage capacity of all hard drives produced in 1995.
60 Petabytes: Estimated total size of Flickr photos by December 2011
200 Petabytes: The estimated amount of data contained within the Googleplex in 2006.

Exabyte

1000 Petabytes.
2.2 Exabytes: According to Charles Stross, all data recorded by our species in 2003
246 Exabytes: Total storage of the Internet in 2006.

Zettabyte

1000 Exabytes
1.8 Zettabytes: Estimated amount of total electronic data in existence by 2011

Yottabyte

1000 Zetabytes.

Technology, Computers and Innovation: Why Everything is Speeding Up

Our grandparents incredulously shook their heads at the moon landing, microwave ovens, nuclear bombs and computers. Theirs was the first generation to actually feel technology speed up. Today, crude replicators, new forms of Intelligence, immortality and the very reigns of evolution are within our grasp. Why IS everything speeding up?

Innovation MEANS Acceleration

Ideas trigger ideas. Step into any research lab, and you'd probably interrupt someone methodically confirming or disproving some theory. Boiling enormous amounts of information down to one bit of data. Either yes or no.

But once that tiny single grain of new truth is harvested, new knowledge, questions and ideas automatically spring forth. There’s suddenly a whole other set of things you can do. This is the classic feedback loop of discovery.

Time saving innovations create another self reinforcing loop. Think of all the things we use to save time. Our dishwashers, cars and computers free us from one set of demands but give birth to new urgencies. Our cars make it possible to live further from work. So we commute for an hour. Dishwashers make it possible to catch up on the Email we receive from all of those people around the world we could never have met any other way. So time saving solutions also incrementally speed up the pace of life. More events get crammed into shorter periods of time. Including world changing events.

As we shall see, these simple feedback loops have been speeding up for thousands of years. And we have arrived upon the scene in time to witness leisurely progress become super accelerated.

It's Been A Long Time Coming

"We didn't start the fire. It was always burning since the world's been turning." Billy Joel

The world seemed as still as stone for thousands of years and technology moved as slow as amber. Take a 100,000 year walk back with me. Notice ancient hands transform logs into levers, wheels and axles. Watch these three simple innovations morph into a wheelbarrow.

Flash forward as wheelbarrows transform into wagons going 2 miles per hour, chariots at 20 mph, cars at 100 mph, trains at 300 mph, planes at 2000 mph, rockets at 4000 mph, spacecraft at 18000 mph... Innovations beget ever more and powerful innovations because each advancement illuminates a new set of still greater possibilities.

Why Our World Is Noticeably Speeding Up

It was only last century that our grandparents incredulously shook their heads at the moon landing, microwave ovens that heated only the food and not the container, nuclear bombs and computers. Theirs was the first generation to actually feel technology speed up. A mere half generation later, crude replicators, new forms of Intelligence, immortality and the very reigns of evolution are suddenly within our grasp.

Computers Changed Everything

The BIG invention was computers because they are embedded with memory, logic and calculating power. Shreds of human intelligence. Things sped up dramatically since the computer (including computers) because each generation of computer chip could be harnessed to propagate the next. Now all of our world is speeding up in lockstep with the acceleration of computing.

Centuries ago the pool of innovations we sipped from was small and local. But on the Internet, our global ocean of knowledge and a worldwide pool of innovators are brought together.

Today's automobiles spring both from inventions, knowledge and processes as far back as fire and the wheel as they do from networked minds blanketing the globe.

Is There An End In Sight?

Most of tomorrow's inventions, from new drugs, to enhancing our abilities with embedded computing, to the progress we make with Artificial Intelligence, will be about coping with the acceleration. As a species, we CAN do more than survive. We could thrive and discover ourselves in ways we now believe impossible. But, as far as I can see, the acceleration will eventually go far beyond us. Soon AI will be as good at the same intelligence feedback loop we've been using since the evolution of our opposable thumbs - and accelerating.

Why You Should Know About The Digital Revolution

The computer revolution, the information revolution. You hear talk within the media about some revolution that is supposed to change everything.

Yes, there is a revolution so profound in it's implication, so sweeping in it's scope, and eventually so pervasive in it's effects, that it will change everything, but it is neither of these.

The two big revolutions we have seen so far are the Agricultural revolution which began about 3000 years ago, and the Industrial revolution which began about 300 years ago. I am convinced that we are simply in the heyday of the Industrial revolution, the final strokes of a mighty brush, which because of the Industrial revolution's drive to automate, simply paints faster and faster.

The acceleration of change we are witnessing is the logical outcome of the drive to automate causing history to unfold faster and faster.

Computers are simply machines which automate the processing of information. Because the most important underpinnings of our present world rely so heavily on information, (Economy, Government, Commerce, Business, Education Military, Media etc.), and because computers do that so quickly, all of these systems are speeding up, with change being the only constant. Therefore, these things do not represent something new, so much as the same unfolding process accelerating.

But the true revolution, the one that will change everything, the one with all of the wild cards, is just beginning. It is the digital revolution. At it's very core it IS new. It's about translating atoms into bits of data. Let's talk about that, and why you should know about it.

Shakespeare once spoke of a "tide in the affairs of men" and how valuable it is to recognize it early.

When you consider all of the things that you do which are repetitive, someone, somewhere is working on automating that process.

However, the digital revolution (the translation of real world experiences into bits or data) represents a still deeper shift in the "tide". That is because everything that can be digitized is being digitized. And everything can potentially be digitized. Sounds, images, movement, thought processes, information, knowledge, intelligence... In other words, everything that can be translated to data will eventually be data. Boring you say?

Well consider that the meaning of data has barely begun to dawn on us. Everything we sense is converted to data by our brain. It's called "sensory data". What we now call virtual reality is a vague shadow of what is to come. By then, our world will have completely shifted from the focus on atoms to bits of data.

Once things become digitized, their cost goes into free fall while rapidly accelerating in power and reach. Digital Cell phones loaded with digital cameras communicating across digital networks are leaching the mass from media spawning a new viable grass roots media. Notice how quickly digital devices; computers, phones and iPods spread and evolve. Or how the digitization of entertainment through MP3s or video on DVD have so quickly outstripped the recording and movie industries ability to stop it.

Meanwhile, look around at what physical things become vaporized due to the thrust of digitization: Paper, tape cassettes, photo development processes may quickly spring to mind.

Some cars have no physical link between the accelerator pedal and the engine. When you depress the accelerator, you are actually sending a digital signal to a computer. The pedal might as well be a mouse.

What about farm equipment and the land upon which it tills? Bio-engineering has already produced oranges that grow in vats. An entire "orange grove" sits inside a room, without trees, branches, leaves or dirt. What the digital revolution will do with Real Estate, it can do with practically anything else.

Look at all of the tasks that are repetitive where you work. These are the most fruitful to begin considering in a new light. The sheer amount of opportunities can be blinding at first glance, so start with customer service tasks. Specifically communication needs. Are there repetitive information requests that would better serve these people if they were automated? What about information that WOULD be very valuable, but because of distribution or production costs (printing etc.) it has never been fully implemented?

Now ask yourself how technology could make obsolete some of your equipment, products and or services. If it is conceivable to you, it is achievable. Pay attention to news releases about these areas. The best way I know of doing that right now is by using RSS. Have a robot scan hundreds of your favorite Websites and automatically deliver new information rather than surf for it.

(This is an excellent example of innovative people re-engineering the Web using these principles.)

So, if I have offered you any perspective at all, it is the direction in which to begin casting your flashlight.

For the best description of this emerging digital world, read "Being Digital" by Nicholas Negroponte. Lucid, easy to read and understand, it is at the top of my recommended list.

Warm regards,

Ted

The Dangerous Convergence of Technology and Warfare

Technology decentralizes power; military, political, economic, media, energy... all of it. Because once a medium becomes digitized, it must move with the laws of networked information. Distributed, connected, decentralized, democratized, Peer-to-peer ... Eventually grass roots. From the hands of the few to the masses.

The Internet is possible because of the computerized digitization of text, audio and video. The fight over music copyrights emerges from the digitization of audio. Pirated Hollywood movies can arrive online within hours of release because of the digitization of video.

The same pattern is also occurring within manufacturing which will soon give birth to desktop fabrication and crude (at first) replicators. A. I. will emerge as we get far enough down the road in digitizing intelligence. And this ongoing digitization of everything allows ever smaller groups to wage increasingly potent warfare.

This nexus of technology and warfare are probably the biggest challenge we face as a species. But the increasing acceleration and magnitude of political violence is also driving technological solutions to it. Ultimately the solution to global violence will be distributed as well, meaning we, the masses will somehow become the solution.

The Acceleration of Global Violence

There is an ugly economy to violence. It's cheaper to destroy than build, and to take by force than negotiate. And within every segment of the political spectrum there are those who would employ violence to achieve their ends.

Todays coordinated attacks from stateless groups around the globe cost orders of magnitude less than the burdens of rebuilding and forever heightened security.

Present political stability depends on managing wholesale violence with the threat or exercise of still greater violence. Otherwise the political structure dissolves.

The Decentralization of Global Violence

Technology makes increasingly powerful weapons cheaper and gives rise to new ways and means of destruction. And so small innovative groups become capable of increasingly cheap, massive, raw violence. In Madrid, cheap cellphones became sophisticated detonators. They could have been triggered by Email, remotely and anonymously, from anywhere in the world.

Where Is This Taking Us?

Technology democratizes power of every kind; military, political, economic, media, energy production; all of it. Centralized power becomes distributed from the hands of the few to the masses.

I do see the day when politics reflects the conscience of mothers, the security of children and all the things communities really want in order to thrive and grow functionally. There isn't much time though. Weapons of mass destruction are becoming open source, distributed and cheap.

Half of Test Subjects Never Suspected They Were Talking To A Computer

A clever twist on the Turing Test using a popular chatbot and some creativity. Roughly half of the test subjects never suspected they were actually talking to a computer.