If your website isn't showing on Google at all then this may be why

We all know that if we can get our website high in the rankings on Google, then that is our holy grail. It's simply the best online marketing we can do without lots of recurring costs.

Throughout the years I have had many potential clients and friends come to me saying that their website isn't appearing anywhere on the search engines. Usually there's no problem, it just takes time and patience and they are expecting too much, too quickly. But every now and again it was true, and this was the reason.

Building your own website is not the easiest thing to do, there are so many things you need to know, especially nowadays... Creating a responsive, semantically correct and accessible site takes the skills of someone who is an experienced professional, and even if you do employ someone then generally the chances are that the cheapest web developer or development house will be the one that's favoured. Obviously, the cheapest is not always the worst but it's likely that they are either more inexperienced, or they are hastily building your site to move on to the next one. I am not saying that if you go cheap, then your site will not be found by Google, but I am saying that people make mistakes and mistakes are more likely to happen if the developer is inexperienced or in a rush. 

But surely, nobody can make a mistake that makes the search engines ignore you completely... Can they? 

Well yes, they can... It doesn't happen very often. But it does happen, and I have seen it

It's true that in most cases, even if a website is badly coded, has lots of repeated terms and is totally inaccessible, it should still be spidered by Google and other search engines even if they rank your site so low that they might as well not know you exist, but there is one thing that can happen that will make every major search engine brush past you without even looking at you. 

Badly created Robots.txt files

If you aren't a developer, then it's unlikely you would even have heard of a robots.txt file, but it's very helpful. This simple text file can do lots of things including:

1. Blocking search engine access to entire areas of your site. For example, you may have a development section, or part of the site you aren't happy with. you can tell the search engines not to look at that just yet.
   
   
2. Pointing the search engines to more prominent areas of your site instead of indexing more mundane shopping carts, thank you and login pages.
   
   
3. Keeping search engines from internal search pages, which can cause problems if they end up within infinite loops etc.
   
   
4. Preventing certain files from being indexed such as PDF files or images. 
   
   and
   
   
5. Specifying the location of sitemaps, which are other clever files on your website which tells the search engines where all your pages are, even the ones you can't navigate to without knowing the URL. 

Having a robots.txt file isn't crucial but most developers agree that it's a good idea to have them, if for no other reason than Google complains within it's webmaster tools if you don't, nobody wants to aggrieve the mighty Google so it's always best to put the file in but tell search engines that they are allowed to view every page on the site. 

If you have one on your site then you will find it at https://www.yourwebsite.co.uk/robots.txt so in this example your will find mine at https://www.yorkshiretechy.co.uk/robots.txt and a really simple one will look like this:

User-agent: *
Disallow:

It may not be obvious by looking at it, but this means that any search engine can look at your site. The User-agent is the search engine, if I put Googlebot instead of the star(*) then it would only allow Google to search. The star means 'everyone'

The Disallow line means, please don't look at anything that I enter after the word Disallow. I could enter /thanks.html which means that the search engines can't look at that page or I could even add whole sections /admin/. 

The problem comes when a developer accidentally adds a simple slash at the end 

User-agent: *
Disallow: /

This simple addition means disallow every page on the site. You can see how easy it is when creating this file to make the simple mistake of adding a slash (/) as website developers use the slash all the time. 

So if it's been a while and Google, Bing or one of the other search engines hasn't picked up your site at all, check your robots.txt and if it says "Disallow: /" somewhere in there... Then that's the reason. 

It's rare, but it DOES happen

I really could do with your help! 😃 If you find this article interesting, could you please do me a favour by either sharing it or commenting below, I would love to hear yours and other peoples' thoughts on this subject. And if this or any other content on the site has helped you and you would like to show your appreciation, then you can always buy me a coffee ☕️ It would make the time I put into this more than worthwhile! Thank you 😃

26 Mar 2024
Author: Craig Pickles (YorkshireTechy)