A web crawler (also known as a web spider or web robot) is a program or automated script which browses the World Wide Web in a methodical, automated manner. Search engine for locating information on WWW, it indexes all the words in a document, adds them to a database, then follows all hyperlinks and indexes and adds that information also to the database. Also called web spider, spider.