[Version 0.72 -
The essay is also available in Rich Text format and MS Word Format
This essay focuses on the conceptualization of the issues, comparisons of current technological developments to other historical/evolutionary processes, future of automated collaboration and its implications for economic and social development of the world, and suggestions of what we may want to pursue and avoid. Explanations of the workings of the technology and analysis of the current market are not my purpose here, although some explanations and examples may be appropriate. Please send your suggestions to sasha1@netcom.com
You can find an up-to-date version of the essay at http://www.lucifer.com/~sasha/articles/ACF.html
Automated Collaborative Filtering of information (ACF) is an unprecedented technology for distribution of opinions and ideas in society and facilitating contacts between people with similar interests. It automates and enhances existing mechanisms of knowledge distribution and dramatically increases their speed and efficiency. This allows to optimize knowledge flow in the society and accelerate the evolution of ideas in practically all subject areas. ACF also provides a superior tool for information retrieval systems that facilitates users' navigation in the sea of information in a meaningful and personalized way. This technology can be viewed as a semantic transport - a social utility that, after physical and data transports, transfers increasingly abstract and intelligent objects between previously isolated fragments of the social organism. As an artificial system that integrates and processes knowledge of multiple human participants, ACF represents an intermediate stage between human and purely artificial intelligence and lays the foundation for the future knowledge processing industry. This article discusses the premises and the historical analogs of ACF technology and suggests its possible uses as well as long-term economic and social implications.
Social mechanisms of knowledge distribution represent a formative factor for all spheres of social life. It is the advantages of sharing knowledge among individuals that, together with benefits of group work, led to the development of language, symbolic thinking, and specialization of labor. The rate of social progress is to a large degree determined by the availability of standardized and affordable communication tools. The transaction costs of the social communication infrastructure define the scales and interrelations of social institutions. Effective mechanisms for collecting and publishing aggregate opinions of the population are a crucial factor for democratic governance; similar mechanisms for establishing balanced product and share prices form a signaling foundation of a market economy.
While generalized, or aggregate, information is essential for balancing social processes on the macro-scale, it is usually not sufficient for suggesting optimal behavior to any particular person. For making efficient personal selections, people have to possess both necessary general knowledge and special information relevant to their particular situation. For collecting necessary information, one has to choose what objects to pay attention to. In early human history, each person was familiar with the whole environment and, after gaining experience with most available things and people, could decide what to explore further. However, this strategy cannot work in a more complex society, when one is faced with more objects and people than he can even sample. This situation requires exchange of personal experience among individuals and sharing personal advice on many particular issues. If a person needs to make a decision in an unknown situation, he can talk to his friends, and follow their suggestions. Here, one's circle of acquaintances effectively plays the role of an information filter, suggesting most relevant options and leads for further exploration.
With increasing variety of areas of expertise and value judgments, the opinions of a few chosen individuals and the averaged opinion of the society become insufficient for providing advice for all of one's decisions. In this situation, larger-scale collaboration in information filtering becomes increasingly important. Individuals seeking advice query people with similar interests whom they trust, collect their opinions and choose the options selected by the majority of the most knowledgeable people. This is very often the way we select places for vacation, books, movies, or restaurants. If none of the people we ask have any experience with a specific item, some of them may still know something about it, if they have heard about it from others. If someone cannot recommend an item of interest, they may still be able to refer us to another person who we may ask about it. This is the "word of mouth" method of information distribution in a society.
Querying people is a useful method for finding information when you know you need it, or when you think something new might have appeared. This method may be called "user pull", as you have to expend an effort to pull the knowledge out of the passive environment. However, it is not always sufficient, especially in cases when you do not know what questions to ask, or something totally new has happened, or your contacts find out something they couldn't have told you about before.
Active collaborative filtering helps you in such cases, by bringing you information you need when someone in your community discovers it. Information can "find" you in two ways. First, you can ask people to let you know whenever they learn something exciting, new, or relevant in some area. Second, people who know you can share information if they decide you can benefit from it. In both cases, after learning about your needs or preferences, members of the community take an active role in supplying you with important knowledge. We can say that the community actively "pushes" the information towards you.
Traditional methods of knowledge distribution become very inefficient when the size and complexity of a society far outpace the ability of anyone's circle of acquaintances to monitor events. The modern communication system transmits billions of messages daily, and many of them may be of great interest to you. You can also easily access any of the millions of available books, magazines, songs, movies or Internet resources. However, finding people who have looked at the things you may like and whose opinions you might trust can be very difficult. You do not know most of these people, so you can not hope that they will know - or care - to tell you when they find something of interest to you. Even if you could overcome these difficulties, it would still be impossible to retain opinions of thousands of people in your mind and decide which of them to follow.
The hope here rests on machines. In their evolution as knowledge carriers machines have repeated the early stages of the development of human information sharing techniques. They have been taking an increasing role in storage, transmission and averaging of opinions. Now it is time for machines to take on the more sophisticated tasks of automated information filtering.
When too much information exists, reliance on human connections for finding exactly what you want becomes inefficient, because people, smart as they are, have limited capacity to store and share information.
The traditional solution to this problem has been to store data outside the human mind, and let people search through it. Recently, with the advent of computer systems, machines also began performing searches for information. So now information seekers have two kinds of helpers: people who can understand your interests, but are slow and inefficient in handling data; and computers that are fast and cheap, but still too stupid to understand the meaning of the material and judge what is good and relevant. Still, we want help that is both fast and intelligent.
Attempts to speed up humans or smarten machines have not been too successful thus far. The solution that allows us to perform information searches with human intelligence and at machine speeds, however, is relatively simple: record people's opinions on the importance and quality of the material, and use these records to improve the results of computer searches. This technology is called Automated Collaborative Filtering (ACF). It can also be viewed as an example of Synthetic Intelligence (SI) - a system combining human and machine knowledge processing strengths. ACF allows people to find others with similar opinions, discover experts in a field, analyze the structure of people's interests in various subject domains, facilitate creation of interest groups, decentralize communication media, improve targeting of announcements and advertisements, and do many other useful things that, together with other intelligent technologies, promise to bring the information economy to new levels. As distribution of opinions does not require knowing the physical identity of the persons sharing and receiving advice, the automated filtering systems (with appropriate software tools) may become the most secure channel of knowledge sharing in history.
The creation of the physical transport system was, in a sense, analogous to the development of mobile organisms. The difference is that in an ecology, organisms move around looking for resources; in an economy, resources usually get delivered to organisms, which is a smarter way to connect them in complex systems. The important point here is that ultimately, the transport function shifts from the organisms to synthetic (technological) systems.
Transport systems modify the effective geometry of the environment. The important metric in the social space is how much time and resources it takes to move things from one place to another. We could create a map of a territory, "distorted" so that the distance between two points is proportional to the time (or cost) of traveling between these points. If we use such a map to track the evolution of the transportation system, we will see that with time, distances contract in all directions, most rapidly along the most important connections. Small improvements in the quality of vehicles and infrastructure reflect the general contraction of the space metrics; new roads and transport methods represent structural changes in geometry and even topology of the social space. We can visualize this impact as new dimensions added to the space or new shortcuts opened between various objects. Advanced transportation infrastructure creates a social circulatory system that helps transform a set of isolated production locales into an integrated economic environment.
Modern history: data/signal transports
Since there is a great variety of resources to move and travel to, a new problem arises - knowing what to get and how to handle it. Some of this knowledge is embedded in our bodies. In other cases, organisms' generic biological skills can be utilized by the social system. So humans spend a good part of their history memorizing and sharing ways to handle things - eventually overloading their ability to exchange information. People may be near the objects they need, but they can not see them directly, or get advice from others - and so they cannot utilize important resources. People may be able to create necessary artifacts and transport them to places where they are needed, but often they cannot do so, as they do not know what to create and where they should deliver the products. As a result, the social space is divided into a set of functional fragments reflecting existing information pools, even though good transportation systems make the physical environment relatively liquid.
To ameliorate this situation, people improve their artificial ways of storing and transmitting data. Efficient information exchange mechanisms re-integrate the economic space, and accelerate economic progress beyond what physical transport could do alone. These mechanisms also allow further leaps in complexity. Now, with today's sophisticated information technology that can transport data electronically as well as physically, people just have to decide who and what to contact; the technological tools allow people to maintain relationships with distant objects, organizations and friends as if they were located next to them.
Consequential benefits of this stage include greater economic efficiency, faster growth, demise of territorial sovereignty, and shift of importance to functional power centers. As conflicts migrate from territorial to functional structures, weapons change from guns to words and bank accounts. Informational entities are torn apart daily in intense economic and cultural competition, while physical bodies feel less pain. Deterritorialization of power structures and obsolescence of the territorial/physical warfare between advanced economies are among the benefits of the information revolution.
Topologically, the effect of fast signal transports is equivalent to the collapse of the social space metric along established social connections.
In the distant future, we may learn how to cost-effectively synthesize any physical artifact on the spot (e.g., by nanoassembly or manipulation of vacuum microstructure), and send high-bandwidth signals at light speed between any two points. Then, the physical space will effectively collapse, and its intrinsic topology will be of as little interest to system designers as details of physical representation of bits are to today's software engineers. The 15-billion-year struggle of functional structures for independence from physical substrate that is now experiencing such a magnificent growth spurt, will be mostly complete, allowing the world to finally concentrate on its real tasks - development of complex, balanced, flexible systems not limited by the shortcomings of any particular substrate.
Now we encounter the next obstacle: so much data, so many signals. Even the powerful machines are overloaded with data. Information traffic, partly due to its relatively low cost, becomes horribly inefficient. Many media, from newspapers to USENET, try to send everything to everybody. As a result, while people already read as much as they possibly can, most of the increase in information traffic is never read. A typical reader of New York Times may read 5% of the paper, and a typical reader of USENET may consume 0.01% of the daily news traffic coming to his server. (Multimedia gives a one-time increase in bandwidth demand, but the main tendency is still the same). The "Internet visionaries" debate whether increased capacities of physical cables may provide the projected increase in data transfer, and billions of dollars are spent on attempting to fix information problems using physical methods. That's a step back by a level! Instead, we should move a level forward and look at the message semantics. Then, we could transmit only the messages that are actually going to be read.
The problem here is that while machines store and pass messages, they do not know what to store and where to pass it. So, they process everything. Humans understand what to handle, and how to handle it. But people's semantic abilities are at least as limited as their skills in handling material objects and storing and transmitting data, so each person has to specialize in a few isolated areas of opinions and expertise. Communication between these areas is imperfect because of semantic barriers - most people do not know each other, the organizational mechanisms of knowledge transfer between domains are relatively weak, and terminology and interests differ (conditions of "reproductive isolation" in the memetic ecology).
The semantic space is fragmented, though the information environment is liquid. So here we face a task similar to the one that led to the creation of the [signal] communication industry: to build a semantic transport system that would take an increasing role in determining what data should be stored and where, who should get signals and from which sources, and how to structure social mechanisms.
In this scenario, machines begin by collecting data from people about what they know, have, like and want, and assist in figuring out what kinds of connections should be made. Then, the machines can suggest these connections to humans ("check out X!") or partly execute selected transactions ("Here is something you may want to know/do" - that is, machine chooses the source, and you decide whether you want to be the recipient, or vice versa). Machines can even act independently for your benefit, e.g., perform market transactions on your behalf, or upgrade your computer with a program liked by similar users - according, of course, to your preferences and instructions.
Consequences here are similar to those of the data revolution, but on another level: growing semantic liquidity and integration, higher effective complexity of the social environment, and even faster growth. At the advanced stage of this process, humans may play the role of "semantic sensors", assessing the meaning and value of objects in their vicinity, and letting machines organize the global architecture of value flows.
This is virtually opposite to the human role in a primitive physical economy: then, measuring and sensing devices (thermometers, clocks, watchdogs) played specialized roles in monitoring and enhancing the environment, while humans determined the system architecture and provided general support and coordination. But now, humans specialize in creating particular artifacts and assessing their quality, and the machine economy provides the general structure. (At this point, most of the architectural structure of the economy is non-biological, so its further de-humanization should be relatively painless.)
It is likely that semantic transport will play as important a role in the advanced information economy as signal and physical transports played in the industrial age. Twenty percent of the global economy by the year 2030? This may sound like an exaggeration. However, people already spend at least as large a share of their time and effort on collecting and sharing expertise. What is needed here is automation of this process. If we look at the acceleration of automation and data processing and extrapolate the trend, 30 years may seem sufficient for development of a knowledge processing industry. We can hardly expect that collection, distribution and sharing of expertise - the most valuable social resource in the coming era - would be done by unpaid and unorganized enthusiasts in their spare from other tasks time, without assessing costs and benefits. Most likely, effective cooperation among contributors and beneficiaries of knowledge flows will call for more wide-spread usage of value accounting, and ultimately build the wisdom economy of the future.
Trends in the evolution of transports
The distinctions of physical, data and semantic transports may seem somewhat forced, as would any other simple subdivision of a complex process. The fact remains that exo-biological systems take an ever increasing role in storing, transporting and processing structures valuable for biological individuals, and the involvement of the technology in these functions is ever more active and intelligent. A wheelbarrow filled with stone tablets already represents an external mobile transport of abstract references, although the message is still too deeply engraved in the medium, and the whole transport is passive and unaware of its function. During the course of history, value transports become increasingly abstract, changing physical carriers without human involvement, testing their own integrity and assuring delivery, processing the entrusted values, and ultimately, accepting a greater role in deciding what should be shared, where it should be sent, and how it should be processed. To build this new semantic transport, we need to create the global information infrastructure, with open standards for semantically rich descriptions of objects and their relations allowing for diversity of opinions on the same subjects and access control for different parties. We also need comprehensive feature and interest ontologies, to describe the structure of the environment, a variety of algorithms that would form the brain of this system, and an extensive amount of knowledge loaded into it by multiple participants.
The development of technology in any area goes through a similar set of stages. At each stage a new level of human abilities first becomes explicitly expressed in non-biological constructs, then independently carried, then carried out (all things that are started by humans and continue functioning on their behalf, from campfires to mousetraps to stock trading programs), then combined and processed, then get taken over. This has already happened, consecutively, with the handling of physical materials, energy, and raw information. Pursuit of AI, though a worthy goal, may have been an attempt to leapfrog the natural cooperative/mixed/synthetic stage in the area of object semantics. Now we are filling this gap. Human judgments have been expressed in artificial media for a long time, communication industry gave us efficient tools for transporting them. Automated processing of opinions is still in a very primitive stage (e.g., ballot counting systems), but ACF technology promises to significantly raise its level. After this is accomplished, the next layer of human functionality to be taken over will probably be semantic analysis of objects. The work here has already begun, with pattern recognition and signal processing techniques, and commonsense information analysis tools on the higher end. Existence of semantic transport infrastructure will allow immediate utilization of advances in automated semantic analysis, which should greatly accelerate its development. An analog here is text analysis tools that after the emergence of the World Wide Web became crucial for daily information searches of millions of people, and turned information retrieval from an area of orderly academic activity into an arena of intense commercial competition.
Strictly speaking, ACF, as well as data communications, starts with transporting references to existing values, rather than the values themselves. Reference transport optimizes the flows of underlying values. This allows references themselves to be recognized as values, and then the new mechanism turns into a value transport in the full sense. In the current society, though money can be easily passed from one agency to another, many crucial areas of economic and social activity are under-funded, as their importance is not recognized (though all necessary information may be available somewhere, it can remain unknown or ignored by people with money). In many cases, people have no incentive to share their knowledge with others, as in many areas there are no mechanisms to compensate them for sharing the knowledge. So instead of just telling other people what they know, which would be the least expensive way to provide the maximal social benefit, they attempt to realize that knowledge in something non-transportable, so that they can control the distribution of benefits and profit from it (though these implementations often could be realized faster and cheaper by other people if they knew about them). The creation of reward mechanisms for knowledge distribution, in terms of enhanced social reputation of the individual, as well as direct payment for advice based on the utility of previous opinions expressed by the same person, may give people incentives to share some of the knowledge - e.g., observations of the economy, or ideas for inventions - in the areas where they tend to keep information proprietary today. This would result in less antagonistic relationships between an individual and society, conservation of resources, and acceleration of progress.
ACF provides multiple functions. Let us examine them and see how they may be used.
People spend a lot of time trying to find the best objects to satisfy their interests. In the process, they waste a lot of effort looking through junk, find some occasional surprises (serendipity is a side benefit of bad searches), and miss many gems that might have made their lives much better. Email and Web access dramatically increased the efficiency of information searches and distribution, but did not modify the nature of the mechanisms. Improving this search process would increase quality of life and accelerate the adoption of new and worthy products and ideas. This would save resources and increase the rate of social and economic progress.
An interesting issue here is that recommendations delivered by ACF come from the person's affinity groups, and are more geared towards the person's interests than suggestions from mass-media channels, whose choice recommendations are linked to the power interests of a small number of commercial and political organizations. This could very well lead to decentralization and democratization of influential powers in the society, even for mainstream interests. I would like to stress that I divide interests into fringe and mainstream here, not people. For example, my interest in philosophy may be fringe, but my health and nutrition interests may be quite typical. I would expect that the ACF-enhanced system would be more likely to recommend apple juice over Coca-Cola than would commercial advertising. The effect on the economy should be a greater incentive for companies to tailor both their products and marketing messages to consumer interests rather than to manipulate people's opinions. "Producer push" in value exchanges would give some space to "consumer pull".
Implications for fringe interests are still more promising. People who are interested in rare objects or obscure issues usually get no social messages targeted at them at all. ACF may provide them with advice whose quality matches that of wider spheres of interest. ACF can also benefit people who have interests in the mainstream subjects, but who have opinions different from those of others - they may be trying to do something new, or have finer or exotic tastes. For these individuals, ACF can deliver advice corresponding to their preferences. Such functions will allow ACF systems to support the diversity of tastes and interests and encourage their development, while preventing alienation.
Some caveats exist here as well. If people's expertise and tastes in some area are underdeveloped, a straightforward ACF system might provide them with the advice of their "laggard" affinity group that may give them support but not help in developing beyond their current level.
The hopes here are:
Soon, ACF technology as an information search method will join the ranks of established information retrieval tools, and will make information retrieval systems more intelligent and adaptable. Increasingly semantics-rich information systems will include elements of ACF functionality as integral parts, and will use them to adjust their communications with humans and each other, building representations of the environment more suitable for their tasks. This includes negotiation of "consensus ontologies" for different subject areas, and building models of people's interests that would help refine searches for information. Later, automated content analysis tools based on "common sense knowledge bases" may aid humans in evaluating item semantics.
The ACF methods should not necessarily be based on the straightforward matching of people's interests. One group of people can be interested in materials that another group considers offensive or controversial; educators would pay special attention to things that students find too difficult or boring; researchers may be interested in problems that have puzzled their colleagues or have not received sufficient attention; social scientists and market analysts would study subjects experiencing rapid changes in levels of public interest or shifts of opinions; reviewers would look for new or unknown material in promising areas; fashion watchers may pay attention to things rising in popularity. As a result, the ecology of knowledge may establish productive information exchanges among groups that previously didn't find much in common. (Let us hope that this information will not be used for hostile purposes too often).
This is a very important function of ACF, just as it is of many social mechanisms. Finding people who share your interests is important for finding further directions in life, starting social and economic activities, forming friendships and families, getting advice on important personal decisions, and feeling more confident and stable in the social environment. Many people abandon ideas of businesses or social undertakings, or feel depressed and alienated only because they cannot find other individuals who share their interests, even though those people may pass them in the street, or could be reached in a few seconds with a phone call. ACF can help you find kindred spirits wherever you go - just express your preferences, and you will find that the person in the next theater seat shares your interests in the arts, that your neighbor at a business banquet table is thinking of hiring somebody with your qualifications, and that the person in the airplane seat next to yours is single, fits all your dating criteria and shares your political affiliations and musical tastes. Some day, you might even be able to find people sharing such bizarre interests as the historical subject of social alienation in the pre-ACF era.
The social functions of ACF can be supplemented by other Automated Collaboration (AC) techniques, such as group scheduling, multi-user games, teleconferencing, groupware, etc., that would facilitate all kinds of activities among like-minded people.
The mechanism that refers people to things and ideas they may like, may also work the other way, to help things and ideas find people who may like them. So it may be not clear whether the whole process will help people pursue things, or help things to pursue people. In any thing-person encounter, the beneficiary is some interest that may or may not reside within that person. If this interest is friendly to the recipient of the connection, we have the previously described optimistic scenarios. However, the interest may easily be hostile or manipulative. One may imagine personalized ads that would suggest snake oil to the people who are most likely to be gullible, at the maximum price they could pay for it, or political campaigns where the message delivered to each citizen will say exactly what this person "needs" to hear to decide in favor of this particular politician, and various other versions of "personalized truth" and manipulative messages. Considering that the development of ACF, just as most other technologies, is propelled by pursuit of commercial gains and power more than by search for truth and general social progress, this potential is quite alarming.
Historically, the development of physical and signal transports has led to democratization of these areas; travel and communication are widely available in developed countries, and the gap between the elite and the general populace now is narrower than ever. However, the development of the industrial and information economies was far from peaceful and democratic, and was characterized by huge inequalities and other unpleasant phenomena, from wars over resources to mass brainwashing. In the end, the social powers came to a new equilibrium, and the grass-roots power of the increasingly well informed populace balanced the stronger but indiscriminate influence of new political and economic elites.
Could the new industry of personalized knowledge delivery overpower the existing mechanisms of grass-roots communications and upset this historical balance? Maybe we will be lucky, and the logic of the next stage of technological development will magically coincide with general interests of humankind and the needs of social progress and stability. But, this has never happened before. Or, perhaps, people will wake up and pass their development powers to benevolent visionaries. This has never happened either...
People will find themselves surrounded by a host of intelligent reactive information systems. Most people will have no idea of what is going on, and everybody will just hope that they can find some value in the new situation, while tolerating the bad effects. Governments and businesses will develop semi-intelligent solutions with predictable economic efficiency (different for these two), but mostly with localized benefits and random social effects. People who prefer emotional stability to intellectual growth will form comfortably stagnant communities and shield themselves from disturbing and challenging influences. Fringe sociopathic people with perverse interests will for the first time find secure and efficient ways to organize and communicate. The governments will cite the "urgent necessity" of protecting some obscure parts of the population from whatever the officials consider criminal, and pressure the system to incorporate snooping technologies and censoring rules. Politicians and marketers will do their best to brainwash the population. Socially responsible thinkers will play about as little a role as they ever did.
Still, ACF can do something for visionaries, too: it can help them form their own common interest groups on the semantic fringes of the society, and the wise folks will at least have some pleasure jointly observing the events.
ACF systems will provide a responsive semantic network designed by people and for people, as an expression of their interests and aspirations. Ideas coming from incompetent or malevolent forces will get caught in the global memetic sieve before they could influence a sizable part of the population. Good and useful ideas and opinions will proliferate and be distributed with unprecedented speed to all who need them, while people who produce and recommend these ideas will earn great respect from their audiences, further increasing their ability to influence events.
Life will most likely be more complex than these scenarios, especially if one considers that ACF will never be the sole factor shaping the society. Whether ACF may soon start offering strong positive contributions to the lives of many people depends largely on the vision and efforts of the people who develop and promote this technology.
A common problem in an advanced society, and one that has not received sufficient attention because of lack of commercial interest, is that of ubiquitous junk. In the good old cave times, keeping track of one's few belongings was easy; today, people seem to be drowning under piles of personal possessions. The problem becomes even more difficult with data objects, as low delivery costs and virtually unlimited storage potential make the accumulation of one's intellectual possessions overwhelming, while efficient handling of personal information becomes more and more crucial.
A typical personal computer today contains several thousand files, many of which have hundreds or thousands of records. Without effective management, these information repositories are becoming "write-only memory". The evolution of managing methods here is repeating the path of larger systems. Originally, people just remembered all their files. Then, the software allowed organized storage and searches for data. The recent generation of PIMs - Personal Information Managers - allows for more flexible and meaningful organization of personal information. However, the task of classifying and managing thousands of aging pieces of information on a PC becomes a challenge even for a seasoned professional. The "Miscellaneous" category acquires thousands of diverse materials; searches by keywords become difficult ("What was that thing that I got recently from some good friend of mine that I thought I'd do something with?" - try this with keywords!) In many cases, one doesn't even remember to look for something...
Intelligent software agents promise to alleviate these problems by remembering all activity on a computer, deriving patterns from the user's behavior to help classify and prioritize messages and otherwise manage their "external memory". But here we face the usual hard problems of building good AI. At this point, PC intelligence is far too limited to build reasonable classifications of data and figuring out which messages are important, except for a few cases with usual topics and regular correspondents. Even with good intelligence, users wouldn't receive very useful advice, if an agent's knowledge is derived only from one user's limited personal experience.
The first generation of software for managing personal resources has already appeared on the market. The next stage may be expected to include elements of ACF, such as recorded opinions of human experts in one's sphere of interests, recommendations of like-minded people, and object classification and information retrieval rules derived from their personal information-handling patterns by their own software agents. From this stage, the software servants may be constantly at work exchanging experiences with each other, generalizing it for related groups of tasks, negotiating subject ontologies, etc. With time, most of the "semantic traffic" will consist of messages created by one machine for the consumption of others, as has already happened with storage, transportation and processing of physical materials, energy and information.
With good information protection technologies, people could trust the large servers with storage of their personal data, to ensure its security and accessibility from anywhere in the world. Then, the search for personal information can be provided by the same tools as the global or company-wide search, with consideration of access rights.
Eventually, the knowledge of our habits and preferences will be distributed in, or available to, all our tangible possessions, from the toaster to the walls of our houses. As appliances acquire the physical and intellectual skills necessary for physical mobility - make the transition analogous to that between plants and animals - the benefits of information processing we have been discussing will extend to include manipulation of physical objects. Mobile possessions will rearrange themselves and other appliances for our convenience, discover and bring us things we may consider of interest, take us to destinations, and drive along the paths, that we are likely to enjoy the most.
With a natural progression of a human from a biological creature to a relatively individual tool manipulator to a part of a technological system, the perception of human value shifted correspondingly - from bodily strengths, to the amount of personal possessions, to the value of knowledge and experience necessary for taking part in an integrated economy. In today's economy, reputation - the perception by the community of the utility of one's socially productive skills - often represents the single most valuable asset of a person or an organization and plays an important role in most social and economic interactions. However, there has been little intentional effort to widen the use of this important social tool. ACF technology can make a considerable contribution here, collecting, maintaining and distributing knowledge of people's reputations.
Many people feel suspicious of plumbers and car mechanics, as they tend to under-perform and overcharge. However, these people do not have different DNA from everybody else's, and their occupations are not legally limited to the members of the National Crook Guild. The problem is that these businesses have little incentive to perform more responsibly. The customer does not have sufficient expertise to personally control the work of these people, and usually has no access to the opinions of previous customers regarding their services. The business then does not feel motivated enough to give the customer the best service and price, as the excess charges would enrich it directly, while the negative opinion of the customer is unlikely to repel other potential clients.
This situation would change dramatically if the opinions on previous jobs became publicly accessible. If you need to get any service, you would ask the ACF system for a list of business evaluations, and select the service based on the combined knowledge of people similar to you in income, place of residence, service requirements, and cultural preferences.
The system is not necessarily immune from abuse, as bad servicemen might ask their friends to enter fake references for their businesses. However, people whose recommendations appear misleading, would see their own reputations - and influences - decrease in the system. This would provide an incentive to people to be truthful, and will make sure that people who tend to give bad advice, will not harm others too much. If the system rewards the providers of good advice (would you pay 25 cents for a reference to the most experienced and honest car mechanic in your area?), then people with the most valuable consumer expertise may find themselves employed as paid consumer advisors. If the system can transport values, it can transport rewards as well - collection and distribution of incentives for providing useful references and reviews may become a key part of a collaborative system ensuring "positive value feedback".
Direct references to the company and product quality, coming from independent sources and tied to the interests of the particular user, seem far superior to the current simple method of building product reputation - branding. The "brand awareness" is often built by massive advertising that wins one company or product a place in the limited consumer memory, at the expense of other brands. This familiarity with the brand often depends more on the size of the company and its advertising budget than on the quality of any particular product. Companies can exploit brand popularity gained from previous successes to push new products with non-optimal value/cost combinations to a market that does not know how to choose better, and prevent new rivals with cheaper and superior products from gaining the recognition they deserve. As better mechanisms for product recommendations appear, we may see brand advertising drift towards obsoletion, as the products will be judged and recommended based on their actual values and costs. The general reputation of a company or a product line would still be important, as they would provide additional guarantees of consumer service and product quality, which is especially important for introduction of new products to the market. However, this reputation will be built more on the consumers' aggregate opinion on the quality of the product, than on the all-too-familiar non-informative ads. In many cases, especially for new products, in addition to - or even instead of - the producer's reputation, the consumer would rely on the reputation of the critics - such as Consumer Reports, Environmental Protection Agency or his own friends, to decide whether he should purchase this product from this particular company. This will also give incentives to the critics to improve the quality of their product - advice - on the market of ideas.
Signal economy, as most other things, works well on a certain scale. In the absence of intelligent message targeting, the larger the signal environment, the more there is competition for public attention, and the louder the signals should be to be heard. People need to hear the whole message to assess its meaning, but they may turn away from it if it does not catch their attention. So the message has to be loud, sparkling and shocking from the very beginning. It needs to SCREAM to be heard. Assuring this becomes the essence of the marketing profession. Meanwhile, the audience learns to turn away from a message if it's attention is not arrested within a few seconds. As a result, the attention span of the populace takes a historically unprecedented plunge, while the social environment becomes filled with shocking people and shocking issues, which do not necessarily have anything to do with real values or problems. Perceptual shocks satisfy consumers by tickling their natural senses, and reinforce the dominance of perception manipulators over value creators in both political and economic decision-making bodies. Oftentimes, these bodies are called to "handle" the photogenic non-issues they themselves made up in the minds of the populace. John Perry Barlow once called this "government by the hallucinating mob". At this point it does not look like changing the situation is perceived by anybody as being in their interest. I would not try to convince you that ACF or any other technology can be a social panacea, but it still may help fix some of the problems created by the previous generation of communication tools - the mass media. The consumers may soon learn that expensive TV ads may be fun to watch, but modestly looking recommendations from their friends and independent experts delivered by the ACF system promise more relevant products of better quality and at a lower price - and they are actually worth attentively watching all the way through. This may start some far-reaching trends.
The society that does not have the benefit of enlightened visionary governance, is prone to be corrupted by unexpected side-effects of technological innovations (such as mass media) whose negative influences it is not intelligent enough to counter. By the same token though, this society also does not have the strength to resist when the arrow of the technological imperative turns in the positive direction. So, unlike a socium controlled by visionary villains, our situation at least has a chance to be improved by dumb luck...
So far, we discussed using people's opinions on existing things. However, people can as easily express their opinions on the things they want to bring into existence, or things they hope will never happen. They can tell how much they want to have a concert next Tuesday night, whether they think they need a bus service in their neighborhood, whether a translation of some book into their language is desirable, or whether they would join a mailing list on cactus collecting if it existed - and also how much resources (time, money or other projects) they would sacrifice to make sure this project is implemented (or prevented) and who they would trust to perform the work. If there is a social agreement, the project would start. The role of ACF in this process would be distribution of the initial suggestions to people who care about them, and collection of opinions about the utility of each suggestion and the best ways to implement it. Meaningful targeting of all messages would assure that the suggestions would be distributed only to people who do want to take part in the project, so the expense of time would be minimal. Actually, not all interested people have to take part. Each person can define trust quorum criteria - for example, any 5 of his personal friends, or at least 20% of the community, or any group whose total reputation exceeds a certain threshold - and delegate them the right to commit his personal resources to the project on his behalf (maybe, with certain limitations). The outcome here is a variable-geometry representation and voting scheme in which nobody's resources are committed without their personal approval, and the most appropriate people, according to the best knowledge of the community, are trusted with evaluating the decisions and carrying out the projects. Comparisons of this scheme in efficiency with existing methods of collecting and allocating public resources may be left as an exercise to the reader. An interesting question is whether today's methods of extraction of resources from unwilling "participants" and delegation of power to "representatives" untrusted by many, may still appear efficient for any purpose. An ideal construction of any social mechanism, in my opinion, is one that combines the benefits of large-scale integration with maximal local and personal freedoms. After a certain stage of development of economy and communication industry, people started enjoying such mechanisms in their economic life; maybe, ACF will be the technology that will have the same effect on the political system and other decision-making processes.
Filtering the world for your attention is important, but it results in just simple decisions, whether or not you should see a given item. After that decision is made, lots of options remain in constructing your interaction with an object. Presentation of information can be adjusted in a variety of ways, based on what the ACF system learns from the behavior of other people. Texts you see on the screen can be shown to you with the parts that you are likely to be most interested in, highlighted, and uninteresting details shrunk or masked. Terms that you are not likely to understand may be converted to hyperlinks leading to explanations appropriate for your level of expertise and linguistic and cultural preferences. Similar transformations may be applied to sounds, images and any other interface parts you may encounter, based on the experience the system has gained from interacting with similar people. (More radical ideas on possible transformations of one's view of the world and artificially enhanced perception are described in my essay on Intelligent Information Filters and Enhanced Reality.)
The computational features of the knowledge economy will be based on many heuristic parameters that can be continuously adjusted for various purposes. Slight changes in computation factors can temporarily move access traffic away from congested communication lines, change the balance between stability and serendipity of information searches, shift advice on entertainment choices toward healthier activities at the time of a flu epidemic, support inter-group communications or increase attention to third-party interests in periods of social instability, or steer students towards the choice of subjects that are likely to become understaffed in the near future. Of course, there are more direct methods of interference, such as censoring message traffic or forcing selected connections. Any private party could also help adjust the workings of the system, for example, by subsidizing prices that certain people have to pay for certain references (e.g., teenagers - for reading advice, or abuse victims - for counseling references). The US government has one parameter now - the prime rate - that it can more or less frequently modify to assure stability of the market, and some other legal instruments - like taxes and tariffs - to periodically readjust things. The information market can be fine-tuned in real time (many times a second if it is necessary) along numerous structural parameters. This creates wide opportunities for flexible and intelligent regulation. And tremendous opportunities for abuse... A system designed in the interests of the society as a whole should facilitate good influences and prevent bad ones. For that, it should be very carefully constructed.
The applications mentioned above move beyond mere information filtering. Further extensions of this technology may include other methods of manipulating and utilizing semantic assessments of items and their features by multiple human and machine participants of the system. For better understanding of the item and user spaces and resolution of ambiguities in user-item encounters where several interests of the user interact with multiple features of the item, we may want to cluster item features and user interests into functional vectors in the semantic space. (Visualization tools and graphical navigation utilities in this space may appear useful). The affinity factors of both items and users to these "semantic eigenvectors" may provide compact and meaningful representations of their essential features and assure more efficient communication of their meaning. Such "semantic encoding" of information about entities represented in the system may allow both dramatic improvements in information flows and significant reduction in bandwidth requirements. The representations can range from simple aggregate value assessments to arbitrarily complex structured descriptions of people, objects, and their relations. The exchange and usage of this high-level information may be regulated by "semantic communication protocols". Possible uses of this technology are virtually unlimited. Terms such as "Filtering systems", "Recommender systems", and "Semantic transport" describe just some of the particular applications in the large emerging field of distributed knowledge processing.
Building the semantic transport system on the planetary scale may appear the most complex project ever undertaken in human history, with the greatest impact on social structures and with numerous technological challenges and social controversies on all levels. As we know from experience with economic and political structures, it is extremely important to build a proper architectural foundation that would assure efficiency and stability of an emerging social mechanism.
Semantic Web - a vision of the future of intelligent Web by Alexander Chislenko
Resnick, Paul. "Filtering Information on the Internet". Scientific American, March 1997. An overview of non-collaborative automated filtering system (PICS). Online copy: http://www.sciam.com/0397issue/0397resnick.html
Community and Personalization. IBM Systems Journal, Vol. 35, No 3&4, 1996. A set of articles discussing theory and practical implementations of community news systems, user modeling issues, and content-aware media. Online version: http://www.almaden.ibm.com/journal/sj/mit/sectionb/benderi.html
Communications of ACM, 1997, vol. 40(3). A series of articles edited by Hal R. Varian and Paul Resnick, discussing various aspects of recommender systems and their economic consequences.
Marvin Minsky, The Society of Mind
David Gelertner, Mirror Worlds. Oxford, 1992. The author suggests using reactive representation technology for mediating our contacts with the world. (Today, this function is performed by the outer layers of an individual human mind based on the knowledge loaded from the outside world.)
Olaisen, Johan, Patrick Wilson and Erland Munch-Pedersen, editors. 1996 Information Science: From the Development of the Discipline to Social Interaction. Oslo: Scandinavian University Press. - A series of articles on topics ranging from information overload to conceptual change and theory of social interaction.
Howard Bloom, "The Lucifer Principle". The book contains interesting discussions on society as a distributed cognition system; the author will develop these ideas in his next book, "The Irrational Invention Machine".
Hans Moravec, "The Mind Children". Harvard Univ. Press, 1990. Moravec's ideas on the future of human and machine intelligence will be further developed in his new book tentatively named "The Mind Age".
Alexander Chislenko, "Networking in the Mind Age". Telepolis, August 1996. Essay on the future of distributed intelligence. Online copy: http://www.lucifer.com/~sasha/mindage.html
Robin Hanson, Could Gambling Save Science? Encouraging an Honest Consensus. Social Epistemology 9(1):3-33,45-48, 1995. A market-based mechanism of establishing and using scientific reputations, reaching consensus on controversial theoretical issues, and choosing projects for research. Online version: http://uhall590d-1.sph.berkeley.edu/gamble.html
Nick Szabo, "Smart Contracts". Extropy #16. An article on security aspects of automated collaboration. An update is available online at http://www.best.com/~szabo/smart.contracts.2.html
Alexander Chislenko, Intelligent Information Filters and Enhanced Reality. Extropy #16 (vol.8, no.1) 1996. Hypertext version: http://www.lucifer.com/~sasha/EnhancedReality.html
Peter Russell, The Global Brain Awakens
Jeffrey S. Rosenschein, Gilad Zlotkin. Rules of Encounter: designing conventions of Automated Negotiation among Computers. MIT Press, 1994.
Lenat, D. B. and R. V. Guha. Building Large Knowledge-Based Systems: Representation and Inference in the Cyc Project
Lenat, D. B. "CYC: A Large-Scale Investment in Knowledge Infrastructure." Communications of the ACM 38, no. 11 (November 1995).
Information Institutions: The Technological Imperative.
American Library Association. Fall, 1997. A concise version of this article
will appear in this book, together with other related works.
Online
information:
http://www.lucifer.com/~sasha/refs/ALAbook.html
Being on Line, Net Subjectivity. Lusitania Press, 1997. ISBN 1-882791-04-5 A collection of articles exploring a diverse set of issues on consciousness and subjectivity on the Net.
World Wide Web resources
Freely Available Information Filtering Systems
Consumer Democracy - consumer opinion and product ratings site.
http://www.sims.berkeley.edu/resources/collab/ - an extensive collection of references to ACF- related resources on the Web compiled by Hal. R. Varian.
Qualitative Decision Theory page
http://www.ideosphere.com/fx/main.html - Foresight Exchange - a pilot project for balancing people's opinions on things that may happen in the future.
http://www.cyc.com CYC - a multi-contextual commonsense knowledge base development led by Douglas Lenat.
feedback -- other essays -- home