About Webmaster File Generator
One thing worth knowing before you use it: robots.txt controls crawling, not indexing. A blocked URL can still surface in results when something else links to it, and suppressing that needs a noindex tag instead.
Generate an Apache .htaccess with HTTPS and www redirects, directory-listing blocks and custom error pages. The rules are built in your browser, ready to paste in.
Build a robots.txt with crawl rules, a valid XML sitemap from your list of page URLs, and a .htaccess with HTTPS and www redirects plus common security options. These three files are the backbone of a well-configured, search-friendly website.
Copy the output and drop each file into your site's root. A correct robots.txt and sitemap help search engines crawl efficiently, while .htaccess rules enforce HTTPS and clean redirects.
The most consequential thing to understand about robots.txt is what it does not do. It requests that compliant crawlers avoid fetching a URL; it does not remove that URL from an index. A blocked page that other sites link to can still appear in results, listed without a description, precisely because the crawler was told not to fetch it and therefore could not read the noindex tag that would have removed it. Blocking and deindexing are different operations, and using the first when you meant the second is why pages linger in search results long after someone believed they had removed them.
Two details cause most robots.txt mistakes. Rules attach to the most recently declared User-agent line, so a directive placed under the wrong block silently governs a different crawler than intended. And paths are prefixes rather than patterns, so `Disallow: /admin` also blocks `/administrator` and anything else beginning with those characters. The file also governs only the host it is served from, so a subdomain needs its own.
Common use cases
- Forcing HTTPS or a single canonical www/non-www host
- Setting up 301 redirects after restructuring URLs
- Adding cache-control or security headers on shared hosting