To prevent unwanted content within the lookup indexes, webmasters can instruct spiders not to crawl particular files or directories in the regular robots.txt file in the basis directory with the area. In addition, a site might be explicitly excluded from a search engine's databases by using a meta tag distinct to robots (commonly ). When a online s