Robots.txt Generator for Blogger & WordPress

Enter Your Blog URL
Please select your platform:
#Robots.txt Generated by User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /cgi-bin/ Disallow: /wp-content/plugins/ Disallow: /wp-admin/ Disallow: /search/ Sitemap:

What Is Robots.txt & How Is It Useful

Robots.txt is a text file that helps the bots/crawlers of the search engines such as Google, Bing, and Yandex to crawl and index your blog effectively.

It is a set of instructions or rules which the crawlers & search engine bots visiting your blog/website follow.

The Robots.txt file is also used by crawlers from other websites, such as social media sites like Facebook and Twitter, and SEO sites like online keyword research tools.

The primary function of the Robots.txt file is to guide the bots/crawlers about which portion of your blog should be available on the search engines and which posts/pages should not appear on the search.

For example, a WordPress would want all her articles to show up on the search engine but would definitely not want other pages like the blog’s internal search pages to show up on Google!
Go Up