Rajesh Jain: Time to build the memory extender

FUTURE TECH

In the past, the information available to us was limited by what we could remember, file on our own, ask others in a close circle, or access in a library. The internet changed it all by making available a vast library of inter-linked digital information.

Finding the information of value to us when we need it is still a challenge. In the beginning, we had Yahoo's editorial selection of the best sites on the web. Then, its editors updated site listings via incoming requests for additions as the web expanded.

Next came the first generation of search engines which moved beyond the directory approach and crawled millions of pages, indexing the text on these.

Google, representing the second generation of search engines, used the information about links between pages as part of its PageRank technology, to come out with a superior way of searching.

And yet, the problem of finding the right information in billions of web pages remains. Whether it is information in our personal or work life, there seems to be no end to the massive amount of information that we have to sift through. It is time to look at a new approach.

Imagine if we could bridge directories and search engines, making them much more customised based on our likes and the trails that we leave as we surf the internet, and also taking into account all that we write in emails, and elsewhere. This system would use our memory and knowledge as the starting point.

We would start by outlining our interest areas "� the topics that form the knowledge sphere of our lives. This is akin to the directory of topics "� only much more personally relevant to us. For example, in my case the main categories of this list would be something like this: affordable computing, ICT for development, emerging markets, enterprise software, information management, small and medium enterprises, new technologies and India.

Each topic needs to be drilled down further. What is needed is a hierarchy of topics, which helps in further defining our interests in greater detail. For example, my outline for affordable computing could look like this: hardware (thin clients, refurbished PCs, set-top boxes, personal digital assistants), software (terminal services, Linux, applications, language computing), and communications (the ethernet, wireless, satellite, broadband).

This hierarchy of topics serves as the basis for our interests. It gives a unique lens and context to the information that we browse on the web, write in emails and receive as attachments. These topics will evolve as our interests change and as we come across experts who may have done a better job in building a certain part of the information ecosystem.

This is an evolving information base "� built not by a centralised organisation, but in a distributed manner by each of us. Each of us would have a microcosm of the information space, created and updated continuously by what we did. It would ensure that our ideas would have a context, that we would never forget something, and that we could leverage on similar work done by millions of others like us. This is the real two-way web "� linking not just documents, but people, ideas and information.

In the previous column (Ice World, April 21), we discussed Vannevar Bush's vision of a memory extender (memex), which envisioned just such a machine that would allow us to deal with information overload by leveraging aspects of the human mind and its capability to deal with associations. The platform exists today to construct the memex "� built around millions of personal directories and weblogs.

The weblog becomes a personal journal which allows us to store links and abstracts from articles we find of interest on the web, along with our own ruminations. A personal directory provides a table of contents, a glimpse into the way our mind catalogues information. Two existing XML-based standards would help share directories and syndicate the blog posts that we create "� Outline Processor Markup Language (OPML) and Rich Site Summary (RSS).

In a sense, the memex is more about assimilation than aggregation, more value-added integration than scanning silos, more amplification than just presentation. The building blocks already exist in the form of the millions of blogs created by people like us. What is missing is the outline, a map of each of our minds. Once we start creating this in a distributed manner, we will see the emergent effects of a whole which is much bigger than sum of the parts.

In the past few years, various technologies like weblo-gs, RSS, OPML, web services, Google's API, visualisation tools and social software, combined with a deeper understanding of the science of networks, have laid the foundation for leveraging individual intellect to put together a information framework that is far beyond the capacity of any person or organisation.

The time for constructing the memex has now come.

(Rajesh Jain is managing director of Netcore Solutions Pvt Ltd. His weblog is http/www.emergic.org. He can be contacted at rajesh@netcore.co.in)