Maintaining your site’s Search Engine Optimization (SEO) is crucial to driving organic traffic its way. However, there are some pages – such as duplicate content or staging areas – that you may not want users to find through search engines.
Fortunately, there’s a way to prevent engines like Google from accessing specific pages and displaying them to searchers. By editing your site’s robots.txt file, you can maintain control over which content shows up in Search Engine Results Pages (SERPs).
In this article, we’ll discuss what a robots.txt file is and why it’s important to your site. Then we’ll share how to easily create and edit your robots.txt file within WordPress. Let’s get started!
What Is a Robots.txt File?
A robots.txt file includes instructions for search engines about how to discover and extract information from your website. This process is called ‘crawling’. Once a page has been crawled, it will be indexed so the search engine can quickly find and display it later.
The first thing a search engine crawler does when it reaches a site is look for a robots.txt file. If there isn’t one, it will proceed to crawl the rest of the site as normal. If it does find that file, the crawler will look within it for any commands before moving on.
There are four common commands found within a robots.txt file:
- Disallow prevents search engine crawlers from examining and indexing specified site files. This can help you prevent duplicate content, staging areas, or other private files from appearing in SERPs.
- Allow enables access to subfolders while the parent folders are disallowed.
- Crawl-delay instructs crawlers to wait a certain amount of time before loading a file.
- Sitemap calls out the location of any sitemaps associated with your website.
Robots.txt files are always formatted in the same way, to make their directives clear:
Each directive starts by identifying the ‘user-agent’, which is usually a search engine crawler. If you want the command to apply to all potential user-agents, you can use an asterisk (*). To target a specific user-agent instead, you can add its name. For example, we could replace the asterisk above with “Googlebot”, to only disallow Google from crawling the admin page.
Understanding how to use and edit your robots.txt file is vital. The directives you include in it will shape how search engines interact with your site. They can help you by hiding content you want to steer users away from, benefiting your site’s overall SEO.
How to Test a Robots.txt File
You can check to see if you have a robots.txt file by adding “/robots.txt” to the end of your site’s URL in your browser. This will bring up the file if one exists. Just because your file is there, however, doesn’t necessarily mean it’s working correctly.
Fortunately, testing your robots.txt file is simple. You can just copy and paste your file into a robots.txt tester. The tool will highlight any mistakes within the file. It’s important to note that changes you make in a robots.txt tester’s editor will not apply to the actual file – you’ll still have to edit the file on your server.
How to Create a Robots.txt File in WordPress
If your site lacks a robots.txt file, you can easily add one within WordPress by using the Yoast SEO plugin. This saves you the hassle of creating a plain text file and uploading it to your server manually via File Transfer Protocol (FTP).
Step 1: Navigate to Yoast SEO Tools
To start, you’ll need to have the Yoast SEO plugin installed and activated. Then you can navigate to your WordPress admin dashboard, and select SEO > Tools from the sidebar:
This will take you to a list of helpful tools that Yoast users can access to efficiently enhance their SEO.
Step 2: Use the File Editor to Create a Robots.txt file
One of the tools available in the list is the file editor. This enables you to edit files related to your website’s SEO, including your robots.txt file:
Since your site does not have one yet, you’ll want to select Create robots.txt file:
This will take you the file editor, where you’ll be able to edit and save your new file.
Step 3: Edit the Default Robots.txt File and Save It
By default, a new robots.txt file created with Yoast includes a directive to hide your wp-admin folder, and allow access to your admin-ajax.php file for all user-agents. It’s recommended that you leave this directive in the file:
Before saving your file, you can also add any other directives you wish to incorporate. In this example, we’ve disallowed Bing’s crawlers from all our files, added a crawl delay of ten milliseconds to Yahoo’s crawler (Slurp), and directed crawlers to the location of our sitemap. When you’re finished making your own changes, don’t forget to save them!
How to Edit a Robots.txt File in WordPress
If you tested your robots.txt file manually or by using the Google Search Console, and you found errors, you’ll want to correct them. You can do this easily, without even leaving WordPress.
Step 1: Access Your Robots.txt File With Yoast SEO
First, you’ll need to access Yoast’s file editor. You can do this the same way we described earlier, when we were creating a new robots.txt file. With the Yoast SEO plugin installed and activated, navigate to SEO > Tools in your WordPress dashboard, then select the file editor.
On the resulting page, you’ll see options for each of your site’s files that are editable within Yoast, including your robots.txt file. Make sure you’re editing the correct file before you get started.
Step 2: Make Changes to Correct Any Mistakes
If you found mistakes in your robots.txt file while going through it by hand, you can easily correct them and save your changes in the file editor. Yoast will save the updates to your server without any extra effort required on your part.
Robots.txt file testing tools will highlight problem areas for you, as we mentioned earlier. However, these tools don’t make changes to the file on your server. You’ll need to correct any errors the test highlighted in Yoast’s file editor. Then, click on Save changes to robots.txt when you’re done.
Need Help With WordPress?
If your site runs on WordPress, editing your robots.txt file is made easy with the Yoast SEO plugin. You also get access to an entire plugin directory full of other tools for customizing your site, in order to make it your own.
Here at WP Engine, we provide helpful resources for creating and customizing your WordPress site, right down to its fonts and colors. We also offer WordPress hosting plans, and back them up with a variety of key features to help you build the site you need!