Robots.txt Validator

About this tool

Paste a robots.txt file to check it for common issues: recognized directives, presence of a User-agent, and a Sitemap line.

Example

Input: User-agent: *\nDisallow: /admin

Output: Valid robots.txt

Privacy

This tool runs entirely in your browser. Your data is processed locally and never uploaded to a server.

How to use this tool

1. Paste your robots.txt content into the input box.
2. Run the validator to check the file for common robots.txt issues.
3. Review the results for recognized directives, User-agent lines, and Sitemap lines.
4. Fix any warnings or errors in your robots.txt, then validate again.

Step by step, what happens

1. The tool reads the pasted robots.txt text line by line.
2. It checks whether directives look like valid robots.txt directives, such as User-agent, Disallow, Allow, and Sitemap.
3. It looks for at least one User-agent line, since a robots.txt file needs a target crawler group to be useful.
4. It checks whether a Sitemap line is present and flags missing or unusual patterns that commonly cause problems.
5. It returns clear feedback so you can correct the file and recheck it quickly.

Worked example

Here is a simple robots.txt file that allows all crawlers, blocks one folder, and includes a sitemap reference.

Input: User-agent: * Disallow: /private/ Sitemap: https://example.com/sitemap.xml

1. The validator sees a User-agent line, so the file has a valid crawler target.
2. It recognizes Disallow and Sitemap as expected robots.txt directives.
3. It finds no obvious syntax problems in this short file.
4. It reports the file as valid or with no common issues, depending on the tool's output style.

Tips and common mistakes

• Always include at least one User-agent line, even in a simple file.
• Make sure directive names use the expected spelling and punctuation, especially the colon after each directive.
• Use a full sitemap URL in the Sitemap line, not just a path.
• If you manage multiple groups, check that each User-agent section is followed by the rules you intended for that crawler.

Frequently asked questions