Navigation ⇓

How Search Engines Work

18th March 2016
Frog with a magnifying glass

Search engines are amazing. You can type any subject into a search box and information appears as if by magic on your screen. A wealth of information is at our fingertips. We use search engines for so many different things: product reviews, research, asking for directions, job hunting, planning holidays, finding local services...the list is endless.

So, if you have a website it'll appear in search results when someone looks for your products or services, right? Maybe...

Let's have a look at how search engines work.

When you search online, you aren't searching 'the web', you're searching an index of the web, for example, Google's index, or Bing's index. But, content online changes all the time. New websites and blogs are published, people make changes to existing websites, new social media accounts are registered, and so on. Search engines keep their indexes up to date by using software programmes called 'spiders'.

Graphical depiction of a web spider

Spiders 'crawl' the web and report back what they find. They start by crawling previously indexed sites to see if anything has changed. Then they look for outbound links from those sites and follow them to see what's there. This process is repeated again and again and whatever the spiders find they report back to the search engine's servers, which update the index to include any new or modified content.

When someone conducts an online search, the search engine returns the results from its index that it believes best matches the person's query.

So, what's stopping my site appearing in the results?

Well, it could be any of the following:

What can I do about it?

If your site isn't indexed it may simply be that it's new and hasn't been crawled yet. Other potential reasons could be excessive loading times, or duplicate content that has confused the spiders. There are also some more technical issues that could be the cause and will need addressing by your web developer.

The other three problems can be tackled with Search Engine Optimisation, which is making your site spider friendly. This ensures that your site will be indexed correctly. In addition, by making your content relevant to your target audience you can improve your chances of ranking higher in their search results.