ROUT, the first search engine, and NS,
the first network news service
Les Earnest <les at cs.Stanford.edu>
2013 January 28
In March 1961, based on my observations of Air Force System
438L, namely a misapplication by IBM of
batch processing to military intelligence at the headquarters of the Strategic
Air Command near Omaha, Nebraska, I initiated the development of what was evidently
the first search engine, though that term did not come into use until about 40
years later. There were no online document collections nor
general purpose computer networks then but it was clear to me that it would
make sense to take texts directly into a computer from a telegraphic or other
such channel and index them automatically so that they could be retrieved based
on content. I initially proposed such a system under the name Project ROTGUT
(Retrieval Of Teletype Generated Unformatted Text) but for some reason MITRE
Corp. management didn’t like that name, so I shortened it to ROUT (Retrieval Of
The idea behind ROUT was to create an inverted index based
on keywords. As new messages were received they would be searched for keywords
and for each such keyword a list of all the documents that used it was
compiled. Users could then specify the messages they were interested in by
giving a Boolean combination of keywords, possibly with a date constraint, and
all matching messages would be promptly retrieved. The retrieval process was
quick because performing Boolean operations on document lists using “and”,
“or”, or “not” operations promptly produced a list of qualifying texts.
Though my original motivation for creating ROUT was to deal
with intelligence messages, for convenience I wanted to do the testing with
unclassified material but couldn’t find any online document collections until I
stumbled onto a solution. The Air Force’s Project Bluebook had
been collecting reports of Unidentified Flying Objects (UFOs) for years and had
baskets full of Teletype tape from all over the world and some of which made
interesting reading. I understand that they are now available online in Facebook but I will never go there.
No sooner did I get that project
started than I was reassigned to work at the Headquarters of the Central
Intelligence Agency (CIA) in Langley,
Virginia. I then followed the development
of ROUT from afar. As far as I know there were no articles written about it in
public media, though there were a number of unclassified (but proprietary)
reports, which I still have:
L. Earnest, Intelligence
message retrieval experiment (Project ROTGUT), MITRE Working Paper W-3779,
1961 Mar. 16.
C. Justice, W. Aldrich, H. Lynch, R. Rander,
A description of the ROUT System,
MITRE Working Paper W-5802, 1963 Jan. 22.
L. Earnest, Document
Retrieval Development, MITRE Corp. Memo 600.5-33, 1963 Oct. 15.
Final report on the evaluation of the
ROUT document retrieval system – Proj. 438L, MITRE Tech. Memo.
TM-3869, 1963 Nov. 12.
NS News Service
Beginning in 1974 a program called
NS (for News Service), created by Martin Frost with input from John McCarthy
and me, began operating on our computer at the Stanford
Artificial Intelligence Lab (SAIL). It indexed and stored stories from both the
Associated Press and New York Times newswires and allowed users to search for
recent stories using essentially the same retrieval scheme used in the ROUT
search engine discussed above, namely Boolean combinations of keywords.
Actually there were two differences. Whereas ROUT used preselected
keywords, NS regarded all words as keywords except for a short list of
extremely common words such as “a, an, the.” Also whereas ROUT had a thesaurus
for searching synonyms of retrieval terms we didn’t provide that in NS.
NS also allowed users to leave
standing queries so that when a relevant story arrived it would automatically
send an email notification to the requester. An auxiliary program called HOT
allowed users to watch both newswires in real time as stories arrived and were
displayed concurrently on separate parts of the user’s display.
The cost of providing this news
service was rather low inasmuch as the newswires treated us as a college
newspaper and charged only around $25 per month. NS was widely used by people
on ARPAnet for general news information until one of the news services found
out and forced us to provide it only to SAIL people.
Aside from providing a general
news service, NS played an important role in dealing with the Three
Mile Island nuclear accident in 1979. The emergency response team
at Lawrence Livermore Lab found that they needed up-to-the-minute information
on developments at the site but couldn’t get it until we provided them with NS
Also, during the Tiananmen
Square protests in 1989, Chinese students in the U.S.
wanted to pass information to friends in China
but there were no Internet connections there at that time. I set up NS to
locate news about China
and forward it to a student distribution list so that they could print the
stories and fax them home.
Library scientists soon started
using computers to identify and located books and articles. For example, in
1975 Gerard Salton and colleagues developed a vector
space model for automatic indexing but as far as I know systems such as that
didn’t turn into network search services until later.
G. Salton , A. Wong
, C. S. Yang, “A vector space model for automatic indexing,” Comm. ACM, v.18 n.11, p.613-620, Nov.
After the creation of ARPAnet
and Internet but before the Web, several search engines were developed
including Archie, Veronica, Jughead, WAIS and Gopher. The Yahoo! search engine
appeared in the early 1990s, shortly after Gopher. Altavista,
developed by DEC, was evidently the first search engine to use a web browser
and began operating at the end of 1995. It was later purchased by Yahoo!.
Google started later
that decade and soon moved into a dominant position. Both Yahoo! and Google
were spinoffs from Stanford University
computer groups other than SAIL. Marissa Mayer, who reportedly came out of the the Symbolic Systems group there and became an early Googler, has now become CEO of Yahoo!.
A number of other search engines have appeared over the years and continue to
compete with each other.