Crawlability Testing Tool

Robot.txt Generator

Website URL:

User Agent: All User Agents Mediapartners-Google Googlebot Googlebot-Image Googlebot-Mobile MSN Search Yahoo! Slurp Yahoo MM Yahoo Blogs Ask/Teoma GigaBlast Nutch Alexa/Wayback Baidu Naver MSN PicSearch

Allowed or Refused: Default Allow Refuse

Crawl Delay: Default (No delay) 5 seconds 10 seconds 1 minute 2 minutes

Sitemap:

Robot.txt:

 

About 'Robot.Txt Tester' Tool

A Robot.txt Generator is a tool that assists site owners in creating a file called "robots.txt" that instructs search engine bots or "spiders" on how to crawl and index the pages of their domain.

The robots.txt file is a basic text file that lives in the root directory of a domain and includes web robot or spider directions. Website owners may simply test and change the robots.txt file by utilizing a Robot.txt Generator, which does not require significant understanding of web construction and code.

The robots.txt file is significant as it may prevent search results for indexing pages on a website that the owner does not want to appear in search engine results. A site owner, for example, may wish to restrict search results from indexing pages containing sensitive material, duplicated data, or pages that are currently under construction.

A Robot.txt Generator often includes a viewer interface via which website owners can insert the pages or directories they want to exclude from search engine crawling and designate which search engine robots must be excluded. Once the robots.txt file has been created, it may be posted to the website's root directory to take effect.