Google Search Console Integration

User-agent: *
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-admin/
Disallow: /wp-content/uploads/wpforms/
Disallow: /xmlrpc.php
Sitemap: https://aioseo.com/video-sitemap.xml
Sitemap: https://aioseo.com/sitemap.xml
Sitemap: https://aioseo.com/sitemap.rss

Before we discuss the robots.txt file in the example above, let’s take a look at the main components of a robots.txt file.

• User-agent: User-agent is what search bots use to identify themselves. You can use this directive to target specific search engine bots. For example, if you only want Google to crawl your website, then you should add googlebot in place of *. The * indicates that all of the search engines like Google, Bing, Yahoo, and others are allowed to crawl your website.
• Allow: This indicates that the search engine is allowed to crawl this specific section of the website.
• Disallow: Using this directive, you can tell search engines not to access certain pages or files on your website.

Going back to the robots.txt example, we have allowed all search engines to crawl our website using the user-agent: * directive. We have also allowed and disallowed URLs based on how we want Google to crawl our website.

By using the robots.txt file, you can also submit multiple sitemaps. In our robots.txt file, we have submitted a separate video sitemap we have on our website. This makes it easier for search engines to find, crawl, and index videos of our website.

A sitemap is a navigational map of your website. It tells the search engines of how many pages you have on your website. With AIOSEO’s WordPress Sitemap Generator you can easily create and submit your sitemaps.

A lot of people also ask, “Where is robots.txt in WordPress?” in case they want to make edits.

The WordPress robots.txt location is in the root directory of your WordPress website. You can view the file by connecting to your site using an FTP client or cPanel.

But you won’t have to go that far if you’re using AIOSEO. It lets you edit and view your robots.txt file from your WordPress dashboard. No need for a separate WordPress robots.txt plugin. We will discuss more about it in detail later in this guide.

You can also view it online by typing in your domain name followed by the text “robots.txt.” For example, https://www.example.com/robots.txt.

Now that you know what a robots.txt file is and what it looks like, let’s talk about why it’s important for your website.

Why Do You Need a Robots.txt File For Your WordPress Website?

Robots.txt allows you to control how search engines crawl and index your website. Without the robots.txt file, search engines will crawl all of your website, even the pages that aren’t important.

If you don’t stop the search bots from crawling unnecessary pages, they will use up all of your crawl budget and might not crawl the pages you want them to crawl.

Crawl budget is the level of attention a search engine like Google gives to your website. This is based on how frequently search engines like to crawl your website and how frequently your website can be crawled.

If you’re wasting your crawl budget, Google and other search engines won’t be able to efficiently crawl your website and that could hurt your website’s performance.

Each website has a limited crawl budget. Search engine bots only crawl a certain number of pages during their crawl session.

If the bot doesn’t crawl all of your website in the first crawl session, then it will come back and crawl the remaining pages in the next session.

Depending on your website, the next crawl sessions might not be for days. This can slow down your website’s indexation, and your important pages will not get crawled and indexed on search engines. This can lead to a huge loss in traffic and business.

You can quickly fix this by disallowing your website’s unnecessary pages, like your WordPress admin page, plugin files, and themes folder. This will help save your crawl budget and improve your indexation rate.

Now let’s get to the part where we tell you how to view, edit, create, and submit robots.txt file using AIOSEO.

How to Create a Robots.txt file for your WordPress Website?

All in One SEO (AIOSEO) makes it very easy to create and edit the robots.txt file on WordPress.

All you need to do is head over to your WordPress dashboard and go to All in One SEO » Tools.

Then, you will be then taken to the Robots.txt Editor page. From here, you can easily add or edit the robots.txt file using the form.

With AIOSEO, you won’t have to worry about formatting the robots.txt file. All you have to do is add directives to the fields, and AIOSEO will do the rest.

Once you’ve added the directives, click on Save Changes to complete the setup.

Now that you’ve created your robots.txt file, you need to test it to see if it has any errors.

Test Your WordPress Robots.txt File

To test your robots.txt file, simply go to the robots.txt tester tool and sign in with your Google search console account.

Then type “robots.txt” in the URL field and click on Test.

If the robots.txt is file validated, then you will get the Allowed message. If not, then you will get the errors and warnings highlighted in the box.

And you’re done! You’ve now successfully created and validated your WordPress robots.txt file.

We hope that this guide helped you learn how to create an optimized robots.txt file for your WordPress website using AIOSEO.