What Is LLMs-full.txt? Your Guide to Controlling AI Crawlers

AI tools from companies like Google and OpenAI are always learning from websites. But have you ever wondered if you can control how they crawl your content?

The good news is, you can. LLMs-full.txt gives you that control, allowing you to set very specific rules for AI web crawlers.

In this guide, I'll explain what llms-full.txt is and why it's so important. I'll also show you the easiest way to create one for your WordPress site using an llms-full.txt generator.

First, What Is LLMs.txt?

To understand the whole picture, we first need to look at a new proposed standard called llms.txt.

This is a text file you can add to your website. It's designed to give specific instructions to large language models (LLMs), which are the engines that power AI tools like Google's Gemini and OpenAI's ChatGPT.

While it looks similar to a robots.txt file, its purpose is much more strategic. An llms.txt file is designed to help your site get discovered and understood correctly by AI.

Here’s what it looks like:

Think of llms.txt as a special tour guide for AI. Instead of letting AI crawlers wander your site aimlessly, llms.txt shows them exactly where to go and highlights your most valuable information. 

You can use it to:

• Guide AI to Your Best Content: You can include a special sitemap in your llms.txt file. This tells AI, “Hey, if you want to learn from me, start with these important pages.”
• Grant Clear Permission: You can use an “Allow” rule to explicitly grant AI crawlers permission to access and learn from certain parts of your site.
• Set Clear Boundaries: You can still use a “Disallow” rule. This is useful for telling AI to ignore sections that aren't valuable for training, like admin pages, while keeping your high-quality blog posts open for discovery.

In short, llms.txt is designed to help you build a productive relationship with AI, ensuring your best content is discovered and used correctly.

So, What Is LLMs-full.txt? The More Detailed Version

Now that we know llms.txt is, let's talk about llms-full.txt.

LLMs-full.txt is a larger, more detailed file that contains a machine-readable version of all your website's content. 

This file is often formatted using Markdown, which is a simple way of formatting text that uses plain characters to define structure. In other words, it makes it easier for AI agents to crawl and understand your site.

This is perfect for AI because it removes all the complex HTML code, CSS styling, and JavaScript that make up a normal webpage. It leaves only the pure content and its basic structure (headings, lists, bold text, etc.).

So, why would you create a file like this?

• It's More Efficient: It's much faster for an AI crawler to download one large, clean Markdown file than it is to visit and process thousands of individual pages. This can reduce the load on your server.
• It Provides a “Clean Feed”: By using Markdown, the llms-full.txt file contains just your pure content, ensuring the AI learns from your high-quality information without any distractions like menus, ads, or sidebars.
• You Control the Narrative: By providing this file, you are proactively giving the AI the exact content you want it to discover, in its cleanest and most complete form.

In short, llms-full.txt is an advanced method for site owners who want to efficiently provide a full and accurate version of their content for AI.

LLMs.txt vs. LLMs-full.txt: How AIOSEO Handles It

All in One SEO (AIOSEO) is a WordPress SEO plugin with a built-in LLMs.txt Generator. I’ll tell you more about it in the next section, but I want to quickly point out how it handles the two file types differently for your WordPress site. 

Spoiler alert: The key difference is the amount of content each file provides to the AI.

LLMs.txt: The “New & Noteworthy” Feed

What it is: In AIOSEO, this file contains the full content of your most recent posts. You get to choose exactly how many URLs per post type to include (e.g., 500, 1,000, or more).

Why it's useful: This creates a lightweight, fast-updating feed. It's perfect for AI crawlers to quickly see your latest content and stay up-to-date, without needing to process your entire site every time. 

It's like the “New Arrivals” shelf at the front of the bookstore.

LLMs-full.txt: The “Complete Library” Feed

What it is: This file contains the full content from all of your public posts.

Why it's useful: This is the comprehensive version. It’s designed for when an AI model needs a deep, complete understanding of your entire content portfolio. 

This is like giving the AI access to every book in your library.

AIOSEO's generator creates both files for you. This gives you the flexibility to use your main llms.txt rules to point different AI crawlers to the feed that makes the most sense for them.

Related reading: See our top picks for the best llms.txt generators for WordPress. 

How to Create an LLMs-full.txt in WordPress (The Easy Way)

If you’re a WordPress user, you’ll be happy to know that All In One SEO (AIOSEO) generates llms.txt and llms-full.txt files for you. This means no coding or complicated Markdown. 

AIOSEO is the best SEO plugin for WordPress. It makes on-page and technical optimizations simple, so you can start ranking higher in search results. 

One of the ways it helps in the age is AI is with its LLMs.txt Generator, which creates llms.txt and llms-full.txt files automatically for your website. 

Here’s how to do it:

Step 1: Navigate to All in One SEO Sitemaps

When you’re in WordPress, select All in One SEO from the main left menu. Then, click Sitemaps.

Step 2: Open the LLMs.txt Generator

From here, select the LLMs.txt tab. This will open the editor where you can control all of your AI-related rules and content feeds.

You’ll notice that the first thing you see is a brief explanation of llms.txt and a toggle button for enabling it on your site. The default setting is on. I recommend leaving it as is.

Step 3: Locate the LLMs-full.txt Settings

Continue scrolling down the page until you see the toggle button for llms-full.txt.

Like llms.txt, this setting is also toggled to on as its default.

In the screenshot above, you’ll also see another toggle button for “Convert posts to Markdown.” This is the markup language we discussed earlier that makes it easier for LLMs to crawl and index your site.

It’s also on automatically because it’s less load on your server.

Step 4: Adjust Your Settings

Here’s where you can start choosing what you want to show to LLMs and how you want AIOSEO to create your llms-full.txt file.

First, you can customize the title and description of your website. You can either type in your text or leave the default smart tags, which pulls preset text, such as your site title or tagline. 

Next, you can tailor the following settings:

• URLs per Post Type/Taxonomy: This number represents how many URLs per post type and taxonomy to include in the llms.txt and llms-full.txt files.
• Post Types: The default setting is to include all post types. If you uncheck this box, you can pick just posts or pages.
• Taxonomies: These are how WordPress groups your content, mainly your categories and tags. The default setting is to include them, but you can exclude them by unchecking the box.

Finally, you can choose what to exclude. This can be either specific posts, pages, or terms. 

Once you’ve made your selections, click Save Changes. 

And that’s it! 

AIOSEO will now automatically generate and manage both llms.txt and llms-full.txt files for you, helping AI agents discover the content you want them to see. 

Want to view your llms-full.txt file?

Just click Open llms-full.txt.

This will open a new tab in your browser with your file.

LLMs-full.txt: The File of the Future

You now have a powerful way to guide how AI discovers and indexes your website. Instead of just blocking AI crawlers, you can proactively show it your best, most valuable content in a clean, easy-to-read format.

By using AIOSEO’s LLMs-full.txt Generator, you can create both a “recent posts” feed and a “complete library” feed, turning a complex technical task into a few simple clicks.

Now that you've taken control of how AI crawlers see your site, a great next step is to master your robots.txt file to guide traditional search engine bots. Check out our Ultimate Guide to WordPress Robots.txt to learn more.

You can also explore our top picks for the best AI SEO tools and learn how to rank in Google’s AI Overviews.

Finally, for more simple WordPress tutorials and SEO tips, subscribe to our YouTube Channel. You can also follow us on X (Twitter), LinkedIn, or Facebook to stay in the loop.

FAQs About LLMs-full.txt

Disclosure: Our content is reader-supported. This means if you click on some of our links, then we may earn a commission. We only recommend products that we believe will add value to our readers.