Robots.txt Generator

Robots.txt Generator

Build robots.txt rules for search engines and AI crawlers, and test any URL against them.

Live preview

Used for the Sitemap line and the upload address.

Start from

Your robots.txt

Open to all crawlers

Sitemap Not added
AI training Allowed
Bot groups
1
Lines
—
Size
—
  1. Start from
    Allow all
    Preset
  2. Rules
    —
    Paths and bots
  3. Sitemap
    —
    Helps discovery
  4. Upload to
    /robots.txt
    Root of your host

Build your robots.txt

All crawlers

The User-agent: * group. Every bot without its own group follows it.

Common blocks

AI crawlers

Checked bots get Disallow: /. Blocking training bots does not affect normal search.

Training crawlers

AI search crawlers

Fetches a user asked for

Rules for one bot

A bot with its own group ignores the All crawlers group, so its rules are copied in unless you turn that off.

No bot groups yet.

Sitemaps

Full URLs, one per line. Filled in from your website address until you edit it.

robots.txt 

        

        

          
Test a URL
Checks the file above the way Google reads it.

          

            
            
          

          

        

      

    

  



	
	



	
	



			

			
		
	



	
 
	


	





	
                
                	Scroll to Top