Use the robots.txt Generator to create a simple crawler rules file with allow or disallow settings and an optional sitemap URL for search engines, locally.
The robots.txt Generator creates a basic robots.txt file based on your preferences. You can choose to allow or disallow all crawlers and specify a sitemap URL. The resulting file can be pasted directly into your site’s root to guide search engine bots in a straightforward way.