Zellio.io

Robots.txt Generator

Build crawl rules and a sitemap link for search engines.

Processed locally — nothing is uploaded

About Webmaster File Generator

One thing worth knowing before you use it: robots.txt controls crawling, not indexing. A blocked URL can still surface in results when something else links to it, and suppressing that needs a noindex tag instead.

Build a valid robots.txt with crawl allow/disallow rules and a sitemap reference to guide search engines around your site. The file is generated in your browser, ready to copy into your site root.

Build a robots.txt with crawl rules, a valid XML sitemap from your list of page URLs, and a .htaccess with HTTPS and www redirects plus common security options. These three files are the backbone of a well-configured, search-friendly website.

Copy the output and drop each file into your site's root. A correct robots.txt and sitemap help search engines crawl efficiently, while .htaccess rules enforce HTTPS and clean redirects.

Learn how this works

A Practical Guide to Everyday Developer Encoding & FormattingHow and when to use Base64, JSON formatting, hashing, JWTs, regex and cron — with free in-browser tools for each task.11 min read

Frequently asked questions

Does blocking a page in robots.txt remove it from Google?

No, and this is the most common and costly misunderstanding. Disallow stops crawling, not indexing — a blocked URL can still appear in results from external links, just without a description. To remove a page, let it be crawled and serve a noindex tag instead.

Where does the file have to live?

At the root of the domain, exactly at /robots.txt. Crawlers look nowhere else, so a copy in a subdirectory has no effect at all. Each subdomain needs its own.

Should I list my sitemap in it?

Yes — a Sitemap: line pointing at the full URL is a cheap, universally supported way to help every crawler find it, including ones with no equivalent of Search Console.

Common Use Cases

  • •Keeping crawlers out of admin, cart or internal search URLs
  • •Pointing every search engine at your sitemap in one line
  • •Writing crawl rules that differ per user agent

More Webmaster File Generator tools

Webmaster File GeneratorXML Sitemap Generator.htaccess Generator

Related tools

Base64 Encode / DecodeConvert text or files to and from Base64 instantly in your browser.Code FormatterBeautify, minify and validate JSON, HTML, CSS, SQL, XML and YAML.Regex Tester & BuilderTest patterns against text and build common regex from presets.Slug / URL EncoderClean broken URLs or turn any title into a safe URL slug.