Robots.txt Generator

Create a robots.txt file to control how search engines and AI bots crawl your website. Use presets or build custom rules.

boltQuick Presets

robot_2Rule 1

User-Agent

Allow

check_circle

Disallow

Crawl-delay (seconds, optional)

mapSitemap & Host

Sitemap URLs

Host (optional, for Yandex)

descriptionrobots.txt

User-agent: *
Allow: /

infoQuick Reference

User-agent: *Applies to all crawlers

Disallow: /path/Blocks crawling of this path

Allow: /path/Overrides a broader Disallow

Sitemap: URLTells bots where your sitemap is

Crawl-delay: NWait N seconds between requests

Pro Tip

Place your robots.txt file at the root of your domain: https://example.com/robots.txt. Search engines only check this exact location.

How it works

Configure Rules

Choose a preset or build custom rules. Select which bots to target, and which paths to allow or block.

Add Sitemap

Add your sitemap URL so search engines can discover all your pages efficiently.

Download & Deploy

Copy or download the generated robots.txt and place it at the root of your website.

Frequently Asked Questions

Common questions about this tool and how it works.

A robots.txt file is a plain text file placed at the root of your website that tells search engine crawlers which pages they are allowed or not allowed to access. It follows the Robots Exclusion Protocol, a standard supported by all major search engines.