Robots.txt Generator builds crawl directives for search engines, including user-agent rules, disallowed paths, and sitemap references. Use it when preparing a site for indexing.
Create a basic robots.txt file with crawl rules and a sitemap reference. Everything runs locally in this tab.
User-agent: *
Yes. The tool is free and runs directly in your browser with no account required.
No. Processing happens locally in the browser tab. Your input is not sent to a server.
Building a robots.txt file with user-agent rules, disallowed paths, and a sitemap reference before launching or relaunching a site.
It must be placed at the root of your domain, e.g. https://example.com/robots.txt. It won't be read by crawlers if placed in a subdirectory.
Disallowing a path stops crawling, but a previously indexed page can still appear in search results (e.g. via links) unless it's also blocked with a noindex meta tag or removed via search console tools.
Yes. Add separate User-agent blocks (e.g. 'Googlebot', 'Bingbot') each with their own Allow/Disallow rules, or use 'User-agent: *' for all crawlers.
Yes, adding a 'Sitemap: https://example.com/sitemap.xml' line helps search engines discover your sitemap, though submitting it via search console is also recommended.
'Disallow: /' under 'User-agent: *' blocks all crawlers from your entire site, which can remove it from search results over time. Double-check rules before publishing.
No, it's optional. Without one, crawlers assume they can access everything. A robots.txt file is only needed if you want to restrict or guide crawler access.
