Google Search Console Integration
Get powerful insights with actionable steps to help you improve your sites rankings in search results.
Explore All Our Powerful SEO Features
Here are all the features that make All in One SEO the best WordPress SEO plugin.
Improve your WordPress SEO Rankings with AIOSEO
Get the most powerful All in One WordPress SEO plugin and improve your website search rankings today.
Even more powerful SEO features
- Easily add title, meta description, keywords, and everything you need for on-page SEO optimization
- Complete support for Google Knowledge Graph and Schema rich snippets
- Easily connect with Google Search Console, Bing, Yandex, Baidu, and Pinterest webmaster tools
- Advanced eCommerce SEO support for WooCommerce, so you can optimize product pages, product categories, and more.
- Automatically notify search engines like Google and Bing about changes to your site.
How to Create and Optimize Robots.txt File on WordPress
Are you having trouble creating or editing a robots.txt file for your WordPress website? If yes, then you don’t have to look any further. This guide will teach you how to create a robots.txt in WordPress that’s perfectly optimized by using All in One SEO, the best WordPress plugin for SEO in the market.
What Is Robots.txt?
Robots.txt is a text file that has instructions, also known as directives, for the search engine bots. Website owners use it to prevent search engines from crawling unnecessary pages on their website. It also gives tips to search engines on how to best navigate and index your website.
Here’s a typical WordPress robots.txt example:
User-agent: * Allow: /wp-admin/admin-ajax.php Disallow: /wp-admin/ Disallow: /wp-content/uploads/wpforms/ Disallow: /xmlrpc.php Sitemap: https://aioseo.com/video-sitemap.xml Sitemap: https://aioseo.com/sitemap.xml Sitemap: https://aioseo.com/sitemap.rss
Before we discuss the robots.txt file in the example above, let’s take a look at the main components of a robots.txt file.
- User-agent: User-agent is what search bots use to identify themselves. You can use this directive to target specific search engine bots. For example, if you only want Google to crawl your website, then you should add googlebot in place of *. The * indicates that all of the search engines like Google, Bing, Yahoo, and others are allowed to crawl your website.
- Allow: This indicates that the search engine is allowed to crawl this specific section of the website.
- Disallow: Using this directive, you can tell search engines not to access certain pages or files on your website.
Going back to the robots.txt example, we have allowed all search engines to crawl our website using the user-agent: * directive. We have also allowed and disallowed URLs based on how we want Google to crawl our website.
By using the robots.txt file, you can also submit multiple sitemaps. In our robots.txt file, we have submitted a separate video sitemap we have on our website. This makes it easier for search engines to find, crawl, and index videos of our website.
A sitemap is a navigational map of your website. It tells the search engines of how many pages you have on your website. With AIOSEO’s WordPress Sitemap Generator you can easily create and submit your sitemaps.
A lot of people also ask, “Where is robots.txt in WordPress?” in case they want to make edits.
The WordPress robots.txt location is in the root directory of your WordPress website. You can view the file by connecting to your site using an FTP client or cPanel.
But you won’t have to go that far if you’re using AIOSEO. It lets you edit and view your robots.txt file from your WordPress dashboard. No need for a separate WordPress robots.txt plugin. We will discuss more about it in detail later in this guide.
You can also view it online by typing in your domain name followed by the text “robots.txt.” For example, https://www.example.com/robots.txt.
Now that you know what a robots.txt file is and what it looks like, let’s talk about why it’s important for your website.
Why Do You Need a Robots.txt File For Your WordPress Website?
Robots.txt allows you to control how search engines crawl and index your website. Without the robots.txt file, search engines will crawl all of your website, even the pages that aren’t important.
If you don’t stop the search bots from crawling unnecessary pages, they will use up all of your crawl budget and might not crawl the pages you want them to crawl.
Crawl budget is the level of attention a search engine like Google gives to your website. This is based on how frequently search engines like to crawl your website and how frequently your website can be crawled.
If you’re wasting your crawl budget, Google and other search engines won’t be able to efficiently crawl your website and that could hurt your website’s performance.
Each website has a limited crawl budget. Search engine bots only crawl a certain number of pages during their crawl session.
If the bot doesn’t crawl all of your website in the first crawl session, then it will come back and crawl the remaining pages in the next session.
Depending on your website, the next crawl sessions might not be for days. This can slow down your website’s indexation, and your important pages will not get crawled and indexed on search engines. This can lead to a huge loss in traffic and business.
You can quickly fix this by disallowing your website’s unnecessary pages, like your WordPress admin page, plugin files, and themes folder. This will help save your crawl budget and improve your indexation rate.
Now let’s get to the part where we tell you how to view, edit, create, and submit robots.txt file using AIOSEO.
How to Create a Robots.txt file for your WordPress Website?
All in One SEO (AIOSEO) makes it very easy to create and edit the robots.txt file on WordPress.
All you need to do is head over to your WordPress dashboard and go to All in One SEO » Tools.
Then, you will be then taken to the Robots.txt Editor page. From here, you can easily add or edit the robots.txt file using the form.
With AIOSEO, you won’t have to worry about formatting the robots.txt file. All you have to do is add directives to the fields, and AIOSEO will do the rest.
Once you’ve added the directives, click on Save Changes to complete the setup.
Now that you’ve created your robots.txt file, you need to test it to see if it has any errors.
Test Your WordPress Robots.txt File
To test your robots.txt file, simply go to the robots.txt tester tool and sign in with your Google search console account.
Then type “robots.txt” in the URL field and click on Test.
If the robots.txt is file validated, then you will get the Allowed message. If not, then you will get the errors and warnings highlighted in the box.
And you’re done! You’ve now successfully created and validated your WordPress robots.txt file.
We hope that this guide helped you learn how to create an optimized robots.txt file for your WordPress website using AIOSEO.
Frequently Asked Questions and Resources
Do you have a question about AIOSEO? See the list below for our most frequently asked questions. If your question is not listed here, then please contact us.
All in One SEO is perfect for business owners, bloggers, designers, developers, photographers, and basically everyone else. If you want to optimize your WordPress SEO, then you need to use All in One SEO Pack.
All in One SEO is a WordPress Plugin. In order to use All in One SEO pack, you must have a self-hosted WordPress site. That’s all.
Absolutely not. You can optimize your WordPress SEO without any coding knowledge. All in One SEO is one of the most beginner friendly WordPress SEO plugin.
Absolutely not. All in One SEO is carefully built with performance in mind. We have developed everything with best practices and modern standards to ensure things run smooth and fast.
Yes, All in One SEO has full translation and localization support. All in One SEO is translated into 10 languages with many more on the way.
No. All in One SEO is a WordPress SEO plugin, so it will NOT work on sites that do not use WordPress. Additionally, All in One SEO is not compatible with the WordPress.com platform. You must be using a self-hosted version of WordPress to utilize All in One SEO.
Yes, you can use All in One SEO on client sites in two ways. Either you can purchase the Pro / Agency license OR purchase an appropriate license for each client sites. You can also refer your clients to purchase directly from our site using your affiliate link to earn a 20% commission.