Evolution of the Human Processor

More and more frequently, I hear people talking about the importance of processing huge datasets. These grand ideas often push the limitations of technology. I can understand the urge to pursue these ideas, but I feel that the mentality is misguided. Mother nature worked out a long time ago that processing data in this way wastes precious neurons. Like us, nature faces a limitation of resources, both in storage capacity and processing power. I've heard (admittedly, from a dubious source) that the human brain has a capacity of about 4 terabytes. This may sound like a lot, but it is not when you consider the storage requirement for all a video and audio over your whole lifetime. So how does the human mind deal with this flood of information?

By prioritising knowledge. Think about the structure and operation of the human eye. We have central vision, with which we continuously change focus between things that we consciously or subconsciously decide is important. We also have peripheral vision which allows us to detect important information within our current view. Between these two extremities, we have a gradual transition between very high resolution and very low resolution.

What we don't have is: a uniform grid of retinal receptors to process data from all directions equally. Unlike a bug, we have evolved to assign attention to very small subset of the information that surrounds us, and then to seek out further detail.

Over millions of years of evolution, nature has determined that all information should not be processed equally. It does this by extracting a large amount of information from a small field of view (think hunting), and only processing small amounts from our peripheral vision (sudden movements).

So how can we apply these observations to real-world problems?

This philosophy should be applied to any high-volume data processing scenario. Just remember a few concepts:

1) Devise rules for deciding what characterises valuable information
2) Formulate efficient methods to extract the valuable information.

In my next post I will discuss techniques for improving algorithm efficiency.

James.