How To Fix Soft 404 Errors In WordPress – An Important Change You Need To Make

In this post I’m going to outline a very important SEO (search engine optimisation) flaw that’s affecting hundreds of thousands of WordPress installations around the world, and more than likely yours too.

This is a flaw that could be drastically hindering your search engine rankings, meaning your website will be getting a lot less visitors than it probably should be getting.

Thankfully I have a simple fix for you that will have it remedied in under 5 to 10 minutes.

What is a Soft 404 error?

When you visit a website the server that hosts the website sends some messages to your web browser which typically you can’t see.

One of the messages it sends it something called a “HTTP Status Response Code”.

There’s a whole bunch of different codes, but the two important ones that we’re interested here are 404 response codes and 200 response codes.

A 200 response code basically means that everything’s OK, and that the server was able to return the page you were looking for.

A 404 code on the other hand means that the page you were looking for could not be found, and so an error page will be displayed to the user.

However, the big problem comes when your website displays a “nothing found” error page but fails to send a 404 code to the browser.

And that’s what a Soft 404 error is – it’s when a website is telling you (the visitor) that there’s nothing there, but the server is telling the browser that everything’s OK.

Why is this a BIG problem?

A Soft 404 error won’t cause too big of a problem to the users of your website, but it can cause havoc with search engines and can result in your site getting penalized.

This is because search engines use “robots” to detect the content on your website, and they rely on the response codes to tell them what’s there.

Therefore if a search engine robot detects that a page on your website is telling it there’s nothing there, but the response code from the server is telling it that there is – then the robot gets confused & as a result it doesn’t know what to do with the page.

Should it list the page in the search engine, or should it leave it out?

Since it’s not a human it can’t make a logical decision, and so as a result it will most likely leave the page out of search results.

But on top of that, the search engine will then usually lower the rankings of all the pages on your website – because what’s to say that the other pages aren’t delivering incorrect response codes too?

So until you get it fixed, you’re going to most likely be stuck with low search rankings, meaning you could be missing out on hundreds if not thousands of visitors.

With that being said, fixing it, and fixing it as soon as possible is very important.

Is it affecting your blog?

The first thing you need to do is determine whether or not your blog is displaying the correct or incorrect response codes.

It’s possible that your blog will be fine, as the problem seems to be on a theme-to-theme basis, however I’ve noticed it’s affecting hundreds of thousands of blogs all across the world so it’s definitely best to check your blog.

If you’re signed up to Google Webmaster Tools, which as a webmaster you should be, then you might have received a message like the one shown below:

If you have, then I think you can probably gather that it’s pretty evident your website is affected.

However, if you’re not signed up to Google Webmaster Tools, or Google hasn’t alerted you to an issue yet, then here’s what you can do to check…

It’s clear from the data shown below (taken from GWMT of affected site) that the WordPress Soft 404 error persists on empty category pages, and empty search result listings:

So if you’re not a user of Google Webmaster Tools you’ll first need to run a search that returns no results, like so:

At least I sure hope that your blog delivers no results for a search term like that, otherwise you’ve got a lot bigger problems than Soft 404’s LOL. ?

Once you’re taken to the WordPress “Nothing Found” page, you need to copy the link from your browser’s address bar.

Once you’ve copied the link, head over to this website and run the check to see what HTTP Status Code is returned by your server.

If the code returned is NOT a 404 status code then you have a big problem & you need to resolve it ASAP, so keep reading…

How can I fix my Soft 404 Errors in WordPress?

It seems that by default WordPress only seems to send 404 HTTP Response Headers when a page isn’t found, and not when results aren’t found.

In fact, since there is content on the “No Results” page and the page technically can be found, you wouldn’t typically issue a 404 response code…

Usually you’d be just fine running things this way, and so to make it clear this isn’t a dig at WordPress (or theme authors) for bad functionality…

The issue has actually come to light because of spammers causing chaos (as usual) and posting links to your site that don’t exist.

So in order to protect your WordPress installation you simply need to add a small bit of code to one of the files. The code that you’re going to add will firstly detect if the user is on a category or search page, and then will detect if there’s any posts to display – if there isn’t, a 404 response code will be returned.

The code needs to go into your theme’s functions.php file, and here it is for you to copy & add:

[callout style=”orange” title=”Code To Copy & Add To functions.php File” centertitle=”false” align=”center”]

// Function to force a 404 header on empty pages with no posts
// Fixes soft 404s from empty search results or empty categories
function force_to_404() {
if (have_posts()) { // Do we have posts?
return FALSE;
}
header( ‘HTTP/1.0 404 Not Found’ );
$GLOBALS[‘wp_query’]->is_404 = TRUE;
return TRUE;
}

// Add force 404 function to global header
add_action(‘get_header’,’fix_soft_404′);

function fix_soft_404() {
if (is_search()) { // Is request on search page?
if (force_to_404()) {
return; // Issue relevant header code
}
} else if(is_archive()) { // Is request on archive page?
if (force_to_404()) {
return; // Issue relevant header code
}
}
}

[/callout]

Be sure to add it to your child theme, otherwise it will get wiped out when your theme updates & you’ll be faced with the same issue once again.

Once the code is added your blog should now start delivering 404 response codes on empty category pages & search pages with no results.

All the other pages on your website that actually exist should deliver a 200 “OK” response code.

To double check things are working you should head back over to this website and check the response codes once again.

Not sure how to add the code?

Adding the snippet of PHP code provided above to your theme’s functions.php is the most robust way to solve this problem as it will make sure that your blog is delivering the correct headers to the browser.

However, understandably if you’re not an advanced user then making a change like that may seem a little daunting. If this is the case & you’re a little worried then there is an alternative method you can use which simply involves adding a little bit of text to your robots.txt file.

So in the root directory of your WordPress installation there should be a file named robots.txt – and if there isn’t, simply go ahead & create one.

All you need to do is add the following snippet of text & save the file:

[callout style=”orange” title=”Code To Copy & Add To robots.txt File” centertitle=”false” align=”center”]

User-agent: *
Disallow: /search/
Disallow: /?

[/callout]

Note – if you already have a robots.txt file with content in, only add the lines above that are bold.

Whilst this wont change the header response codes that your pages are delivering, it will simply tell search engines to simply ignore them, and not to bother indexing them. This isn’t an issue since there’s no real reason to have your search result pages listed in the Google search index anyway.

If you want to be super cautious you could implement both methods onto your website – but there’s no real need to, adding one or the other will do just fine.

Telling Google you’ve made the changes…

After physically making the changes to your blog there’s one last thing left for you to do, and that’s to alert Google of the fact that you’ve actually remedied the issue.

To do this you will need a Google Webmaster Tools account, so if you didn’t already sign up for one above then you’ll definitely need to do it now.

After logging into Google Webmaster Tools, from the menu on the left click on “Crawl” and then select “Crawl Errors”.

From this page you’ll be presented with a list of the errors that were detected on your site. It’s a simply case of selecting them & hitting “Mark As Fixed“.

After doing that, give Google a bit of time to re-crawl your website & then you should notice improvements over the coming days or weeks.

I really hope this article was useful to you, and if you were faced with a Soft 404 issue yourself I hope it helped you out. Feel free to leave any comments or questions below.

Psst – here’s an update.

1 day after adding the above PHP snippet to the themes functions.php it seems like all the Soft 404 issues have been cleared up…

Happy days 🙂