Zellio.io

Webmaster File Generator

Generate robots.txt, sitemap.xml and .htaccess files in seconds.

Runs locallyWorks offlineShare link carries settings, never your data
Send output toBase64 Encode / DecodeCode FormatterRegex Tester & BuilderBuild a workflow from thisOutput stays on this page until you send it.

Frequently asked questions

What does robots.txt do?

It tells search-engine crawlers which parts of your site they may or may not request, and points them to your sitemap. It guides crawling but is not a security control. robots.txt can tell Google and other search engines to ignore admin pages, private directories, or duplicate content, helping them crawl efficiently instead of wasting time on unimportant pages. The sitemap reference tells search engines where to find your important URLs. Note that robots.txt is only advisory — malicious crawlers ignore it, so never rely on it for security. Use .htaccess or login authentication to protect sensitive content.

Why generate an XML sitemap?

A sitemap lists your important URLs so search engines can discover and index them more reliably, which is especially helpful for new or large sites. Search engines like Google can often find pages through links, but a sitemap guarantees they won't miss important pages, especially for new sites with few external links. Large sites with hundreds of pages benefit especially, since a sitemap gives search engines a roadmap of your content structure. You submit a sitemap URL to Google Search Console and Bing Webmaster Tools to accelerate indexing and ensure your important pages are discovered quickly.

Where do these files go?

robots.txt and sitemap.xml belong in your site's root directory; .htaccess goes in the directory you want its rules to apply to on Apache servers. For a site like example.com, place robots.txt at example.com/robots.txt and sitemap.xml at example.com/sitemap.xml so search engines know exactly where to find them. The .htaccess file controls Apache server behavior for that directory and all subdirectories, so place it in your web root (usually public_html) to apply HTTPS redirects and security rules to your entire site. Test that files are accessible at their expected URLs before assuming they're working correctly.

Pro tips

  • Never promote a staging robots.txt to production. `Disallow: /` removes a whole site from search, and recovery takes weeks of recrawling.
  • Use a noindex tag, not a robots.txt block, when the goal is keeping a page out of results rather than out of the crawler's fetch queue.
  • List the sitemap URL in robots.txt as an absolute URL; it is the one directive that does not take a path.
  • Test .htaccess redirect rules on a copy first. A malformed rule takes the whole site down with a 500, not just the affected path.
  • Keep the sitemap to canonical URLs only — listing redirects and noindex pages spends crawl budget for nothing.

About Webmaster File Generator

One thing worth knowing before you use it: robots.txt controls crawling, not indexing. A blocked URL can still surface in results when something else links to it, and suppressing that needs a noindex tag instead.

Build a robots.txt with crawl rules, a valid XML sitemap from your list of page URLs, and a .htaccess with HTTPS and www redirects plus common security options. These three files are the backbone of a well-configured, search-friendly website.

Copy the output and drop each file into your site's root. A correct robots.txt and sitemap help search engines crawl efficiently, while .htaccess rules enforce HTTPS and clean redirects.

The most consequential thing to understand about robots.txt is what it does not do. It requests that compliant crawlers avoid fetching a URL; it does not remove that URL from an index. A blocked page that other sites link to can still appear in results, listed without a description, precisely because the crawler was told not to fetch it and therefore could not read the noindex tag that would have removed it. Blocking and deindexing are different operations, and using the first when you meant the second is why pages linger in search results long after someone believed they had removed them.

Two details cause most robots.txt mistakes. Rules attach to the most recently declared User-agent line, so a directive placed under the wrong block silently governs a different crawler than intended. And paths are prefixes rather than patterns, so `Disallow: /admin` also blocks `/administrator` and anything else beginning with those characters. The file also governs only the host it is served from, so a subdomain needs its own.

Common use cases

  • Launching a site and needing the three configuration files without hand-writing syntax you use once a year.
  • Adding a sitemap reference so search engines discover pages that internal linking reaches poorly.
  • Setting up HTTPS and canonical-host redirects on Apache hosting.
  • Blocking crawlers from admin, search-result or duplicate-parameter paths that waste crawl budget.
  • Producing a starting point to hand to a developer, rather than a vague request.
How it comparesSearch Console and the equivalents at Bing test a robots.txt against real URLs, which is the right way to verify one and requires the site to be verified first. This is the authoring step that comes before that: getting a syntactically correct file to upload in the first place.