WordPress Robots.txt — Setup Guide

Default WordPress behavior

If you visit yourdomain.com/robots.txt and haven't created a physical file, WordPress will serve a virtual robots.txt that looks like this:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

This is a minimal configuration. It blocks the admin area but doesn't include a sitemap directive or block other low-value pages.

Recommended WordPress robots.txt

For most WordPress sites, this configuration provides a good balance of crawl efficiency and content discovery:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /search/
Disallow: /?s=
Disallow: /wp-json/
Disallow: /wp-login.php

Sitemap: https://yourdomain.com/sitemap.xml

What each rule does

• Disallow: /wp-admin/ — Blocks the admin dashboard (crawlers don't need to see login pages)
• Allow: /wp-admin/admin-ajax.php — Permits access to AJAX functionality that themes and plugins depend on
• Disallow: /search/ — Blocks internal search result pages (low-value for indexing)
• Disallow: /?s= — Blocks search URLs with query parameters
• Disallow: /wp-json/ — Blocks the REST API endpoint (optional, but reduces crawl waste)
• Disallow: /wp-login.php — Blocks the login page

How to add robots.txt to WordPress

There are three main methods:

Method 1: SEO plugin

Plugins like Yoast SEO, Rank Math, and AIOSEO include a robots.txt editor. In Yoast, go to SEO → Tools → File Editor. In Rank Math, go to Rank Math → General Settings → Edit robots.txt.

Method 2: Theme file editor

Go to Appearance → Theme File Editor. Create a new file called robots.txt in your theme directory. This method is less reliable because the file may be removed when you update your theme.

Method 3: FTP or hosting control panel

Upload a physical robots.txt file to your site's root directory (the same directory as wp-config.php). This is the most reliable method because it persists through theme and plugin updates.

Common WordPress robots.txt mistakes

Blocking CSS and JavaScript

Some site owners block /wp-content/ to hide plugin and theme files. This prevents Googlebot from rendering your pages correctly, which can hurt rankings. Never block CSS, JavaScript, or image files.

Blocking admin-ajax.php

If you block the entire /wp-admin/ directory without allowing /wp-admin/admin-ajax.php, many themes will break. Forms, comments, and dynamic content may stop working.

Forgetting the sitemap

Always include a Sitemap directive pointing to your XML sitemap. This helps search engines discover your content efficiently.

Testing your WordPress robots.txt

After creating your robots.txt, test it using the NodSEO Robots.txt Generator or Google Search Console's robots.txt Tester. Verify that:

• Important pages are accessible (not blocked)
• Admin and low-value pages are blocked
• The sitemap URL is correct
• CSS, JavaScript, and images are allowed