Robots.txt Generator

Create a `robots.txt` file to control which parts of your site search engine crawlers can access. This helps manage crawl traffic and prevent indexing of duplicate or private content.

1. Add User-Agents: Specify rules for all crawlers (`*`) or target specific bots (e.g., `Googlebot`).
2. Set Rules: For each user-agent, add "Allow" or "Disallow" rules for specific paths.
3. Add Sitemap: Optionally, provide the URL to your XML sitemap.
4. Copy Code: The `robots.txt` content will be generated in real-time. Copy it and save it to the root directory of your website.