The World Wide Web Wanderer, also simply called The Wanderer, was a Perl-based web crawler that was first deployed in June 1993 to measure the size of the World Wide Web. The Wanderer was developed at the Massachusetts Institute of Technology by Matthew Gray, who also created back in 1993 one of the 100 first web servers in history, www.mit.edu.[1] The crawler was used to generate an index called the Wandex later in 1993. The Wanderer charted the growth of the web until late 1995.
The Wanderer was probably the first web robot, and, with its index, clearly had the potential to become a general-purpose WWW search engine. The author, Matthew Gray, does not make this claim.[2] Elsewhere, it is stated that the purpose of the Wanderer was not to be a web search engine.[3]
References
- ^ SEO, Doctor (2022). "Evolution of SEO". www.doctor-seo.net.
- ^ Matthew Gray's home page - Pertinent page on Matthew Gray's section on MIT site
- ^ Brian LaMacchia's PhD thesis, section 1.2.3