Robots.txt example file
A lot of people look for example robots.txt files. Rightfully so, because robots.txt are easy to mess up.
In this article we'll cover a simple and a slightly more advanced example robots.txt file.
A simple robots.txt file
Here's an example of a simple robots.txt file that:
- Allows all crawlers access
- Lists the XML sitemap
User-agent: * Disallow: Sitemap: https://www.example.com/sitemap.xml
Audit your robots.txt and find out right away!
A slightly more advanced robots.txt file
Here's an example of a slightly more advanced robots.txt file example that:
- Prevents Googlebot from accessing
/shoes/ - Prevents Bingbot from accessing
/socks/ - Lists the XML sitemap
User-agent: googlebot Disallow: /shoes/ User-agent: BingBot Disallow: /socks/ Sitemap: https://www.example.com/sitemap.xml
Read the full Academy article to learn everything about Robots.txt for SEO: The Ultimate Guide
Share Article Details
About the authors
Steven van Vessum
Steven is Conductor's Director of Organic Marketing. This means he's involved in everything SEO, Content Marketing and Community related. Right where he wants to be. He gets a huge kick out of letting websites rank and loves to talk SEO, Content Marketing and Growth.
