How to Use Crawl Cleanup to Increase Search Engine Crawl Quota

When Google (and other search engines) begin to index your site, it's common for them to pick up many unwanted URLs that come from your RSS feeds.

Google sees these URLs as unique and tries to index them separately. While this doesn't harm your rankings (Google is smart enough to figure these things out), it does take up the crawl quota allotted to your site and in turn, could cause delays in indexing.

To combat this, we've introduced a new Crawl Cleanup feature inside our Advanced Search Appearance settings to allow our users to fine-tune what Google can pick up.

Disabling RSS Feeds Tutorial Video

Check out our video on How to Disable RSS Feeds in WordPress below:

Enabling Crawl Cleanup

To enable the Crawl Cleanup settings, click on Search Appearance in the All in One SEO menu and then click on the Advanced tab.

Scroll down to the bottom of the page and enable the toggle for Crawl Cleanup.

Crawl Cleanup section on the Search Appearance Advanced tab

Once enabled, you'll be presented with options for managing your RSS feeds on the RSS Feeds tab.

RSS Feeds Settings

WordPress includes many RSS feeds on your site, including feeds that are not necessary at all. If you enable Crawl Cleanup, we automatically disable most feeds, keeping your main site feed and some additional feeds that are important.

Global RSS Feed

The global RSS feed is how users subscribe to any new content that has been created on your site. This is enabled by default with Crawl Cleanup and we do NOT recommend disabling it.

Global RSS Feed setting under Crawl Cleanup

Global Comments RSS Feed

The global comments feed allows users to subscribe to any new comments added to your site. This is disabled by default with Crawl Cleanup.

Global Comments RSS Feed setting under Crawl Cleanup

Static Posts Page Feed

If you are using a static page for your posts (i.e. https://yoursite.com/blog/) Then this option will appear. This is enabled by default with Crawl Cleanup and we do NOT recommend disabling it.

Static Posts Page Feed setting under Crawl Cleanup

Author Feeds

The authors feed allows your users to subscribe to any new content written by a specific author. This is enabled by default with Crawl Cleanup.

Author Feeds setting under Crawl Cleanup

Post Comment Feeds

The post comments feed allows your users to subscribe to any new comments on a specific page or post. This is disabled by default with Crawl Cleanup.

Post Comment Feeds setting under Crawl Cleanup

Search Feed

The search feed allows visitors to subscribe to your content based on a specific search term. This is disabled by default with Crawl Cleanup.

Search Feed setting under Crawl Cleanup

Attachments Feed

The attachments feed allows users to subscribe to any changes to your site made to media file categories. This is disabled by default with Crawl Cleanup.

Attachments Feed setting under Crawl Cleanup

Paginated RSS Feeds

The paginated RSS feeds are for any posts or pages that are paginated. This is disabled by default with Crawl Cleanup.

Paginated RSS Feeds setting under Crawl Cleanup

Post Type Archive Feeds

This controls which post type archive feeds are enabled. No post type archive feeds are enabled by default with Crawl Cleanup.

Post Type Archive Feeds setting under Crawl Cleanup

Taxonomy Feeds

This controls which taxonomy feeds are enabled. Only the Categories feed is enabled by default with Crawl Cleanup.

Taxonomy Feeds setting under Crawl Cleanup

Atom Feed

This is a global feed of your site which is output in the Atom format. This is disabled by default with Crawl Cleanup.

Atom Feed setting under Crawl Cleanup

RDF/RSS 1.0 Feed

This is a global feed of your site which is output in the RDF/RSS 1.0 format. This is disabled by default with Crawl Cleanup.

RDF/RSS 1.0 Feed setting under Crawl Cleanup

Unwanted Bots Settings

On the Unwanted Bots tab, you can control which bots are blocked from crawling your site.

Unwanted Bots settings in All in One SEO

You can select each of the crawlers to block them, or you can select All AI Crawlers to block all the ones listed.

Internal Site Search Cleanup Settings

On the Internal Site Search Cleanup tab, you can control the URLs that are used for the search pages on your site. This can help reduce crawling and combat SEO spammers that use known spam attacks.

Internal Site Search Cleanup settings in All in One SEO

Filter Search Settings

The first setting is Filter Search which, when enabled, will display the settings for filtering the search URLs.

Filter Search settings in All in One SEO

The Filter Search settings are:

  • Max. Number of Characters – This setting will limit the length of internal site search queries to reduce the impact of spam attacks and confusing URLs. We recommend entering a number between 1 and 50 characters. The default is set to 50 characters.
  • Emojis and Other Special Characters – This setting will block internal site searches which contain complex and non-alphanumeric characters such as emojis.
  • Common Spam Patterns – This setting will block internal site searches which match the patterns of known spam attacks.

Redirect Pretty to “RAW” URLs Setting

The next setting after Filter Search is Redirect Pretty to “RAW” URLs. This will redirect pretty search URLs such as /search/ to the WordPress default of /?s=. This can consolidate search URLs and reduce crawling of these URLs.

Block Crawling of Internal Site Search URLs Setting

The last setting is Block Crawling of Internal Site Search URLs. This will add a Disallow rule to the robots.txt created by WordPress to prevent crawling of URLs for search pages. This reduces unnecessary crawling of your site.