Content appears on search results only if it has been indexed by the search engine (e.g. Google). Indexing is a term that traditionally refers to a key in the back of a book that references the locations of information contained within the book. In this case, it refers to the process of adding web pages to Google’s index (effectively, what shows up in Google search results). Web pages are indexed once they are found and identified by Google, which is done using Spiders to crawl websites.
Why index websites?
Websites – including pages, images, videos, etc. – all need to be indexed in order to be visible in Google search results. Using relevant keywords can help your site to be indexed correctly, and this in combination with high-quality content and other factors (page load speed, responsiveness) can help boost your search rankings.
Why would you want to exclude content from search engines?
All WordPress pages and posts are indexed by default, however, if you want to optimise how your site ranks in search results, you don’t want to be indexing everything. Here are a few examples of what not to index.
Sensitive and members-only content
You might have some pages that you want to exclude from search results, e.g. ones that contain sensitive content. Without locking down private content with passwords and usernames, you risk exposing it in public search results.
Keep any pages that require a login off search engine results pages.
Duplicate content
Think about whether you have any duplicate content on your site. For example, do you have two versions of the same web page, one standard, and one printer-friendly? Duplicate content will have negative repercussions with Google and result in reduced traffic to both pages. You can fix this by adding a ‘noindex’ tag to the pages that you want to hide from Google’s search results.
‘Thank you’ pages
I.e. pages that visitors are directed to after converting on one of your other pages. These should only be accessed in response to a certain action, e.g. completing a sign-up form or downloading some content and should not show up in search results.
Archived posts
Archived content is archived for a reason so don’t let your old posts show up in search results.
How to stop Google from indexing your web page
Hiding certain pages from Google’s Spiders is fairly simple. You can use a “noindex” tag on any page that you don’t want to go into Google’s index.
By default, all pages are set to “index” so you need to tell Google not to index your page. To do this, add the below code into the <head> of any page that you do not want search engines to crawl
<meta name=”robots” content=”noindex” />
In this case, you are still allowing Google to follow any links on your page. If you want top block this from happening as well, use a “nofollow” tag too.
<meta name=”robots” content=”noindex,nofollow”>
Do you need support with development? We are a team of freelance developers based in Manchester UK, on hand to help with projects of any size or budget. Get in touch to discuss your project.
