Whether you are considering hiring an SEO agency, or doing SEO yourself, getting your head around the basics is going to pay dividends long term! Understanding how search engines work gives you the knowledge to effectively communicate with an external SEO agency or ideate a successful SEO strategy on your own.

So, before you begin trying to improve how well your website ranks, start here, with an article optimised for you (not for search engines).

Here is what we are going to cover:

  1. What search engines are and why they give librarians a run for their money.
  2. How search engines find your site, despite never even meeting you.
  3. How search engines determine how well your site ranks on their platform if it ranks at all.

What are search engines?

Search engines could be thought of as the world’s hardest working librarians – finding, reading, and cataloguing every page, on every website of the internet, so that we can easily find the answers to important questions like “Who can fix my air conditioner?”, or “my dog just ate carrot, is it safe for my dog eat a carrot?”.

Just like a librarian organises books by genre, search engines organise web pages by topics. Unlike books though, pages on a website are not bound together, which has its advantages, particularly for those wondering if that carrot they dropped is the last thing their dog will ever eat. While a librarian may be able to recommend you a book about safe dog food, that still requires a stressful navigation to the so-called carrot chapter.

Search engines on the other hand can filter through every page on the internet to give you the most relevant and helpful webpage for your query. In just a few short milliseconds, an entire page specifically about how good carrot is for dogs (with an included history on both dogs and carrots for good measure). Ladies and gentlemen, the future!

How search engines find your site

Search engines are constantly looking for new websites and webpages to add to their search results and share with the world. It’s never ending. But before they serve any website or webpage to the web, search engines need to do two things:

1. Crawl the site

Crawling is the process of browsing the web and discovering new webpages – just as a human would find your site).

2. Index the site

Indexing is the process of reading and forming an understanding of the content on a website and its webpages then adding it to a database – this could be thought of like cataloguing).

Thankfully, this isn’t done by some poor human, it’s by a program called a crawler. Crawlers (AKA Spiders, Robots, or Bots) explore the web following links from one webpage to another and forms connections between them. There are many crawlers out there using the web, but there is one that everyone on the planet has come to know fairly well …

Googlebot

Googlebot is Google’s crawler, and while it navigates the web in the same way as other crawlers, Googlebot is smarter. It prioritises crawling efficiently, respects websites, renders pages to understand user experience, and feeds data directly into Google’s search algorithms. Which is why almost everyone agrees that Google is better at answering queries than other search engines like Bing or Yahoo.

How search engines discover a site for the first time

If you have just launched a new website, there are a few ways that search engines might find you for the first time.

Direct submission

Search engines provide tools for manual submission so that website owners can directly submit their URLs to be crawled.

 

Domain registrars

Sometimes, domain registrars (Companies that sell domain names) have agreements with search engines to inform them about new domain registrations.

Following links

Search engines are constantly crawling known websites. If they discover links to a new domain from a page they already know, the crawler can follow that trail and begin exploring the new domain.

Social media

Search engines often monitor social media for new domains and links being shared.

How Google determines how well a site ranks

Search engines all have their own unique ways of deciding which content deserves to rank higher. Rating systems that they use to rate a website’s performance. Search Engines Optimisation Specialists call these ‘Ranking Factors’ and each of them plays a role in whether your site or your competitors comes first. For the sake of saving you time, let’s just focus on Google for now – at the end of the day, it’s probably the search engine you are most interested in.

Google’s ranking factors

While no one truly knows exactly what Google’s algorithm is looking for, SEOs have developed a good understanding over time. Most agree that when Google is looking for an answer to a query, it is searching for up to date, relevant content, written by an expert, that covers the topic thoroughly but efficiently. If you wish to know exactly how we optimise for each of these things, we’re working on a more comprehensive article to go a bit deeper on the subject – for now, here’s an overview.

Relevance

Google prioritises content that directly answers the searcher’s question or goal, this equates to more than just looking at keywords (though keywords are still a very most important part of this). Each time Google updates its algorithm, it gets closer to the end goal of all search engines; to grasp the searcher’s true intent so that it can provide the most relevant content for any given search query.

What is relevant content?

To appear in relevant searches content needs to answer specific questions quickly, thoroughly, and in a format that is most useful to the reader. This could mean using keywords strategically, but it also involves understanding the content format that searchers will find most helpful (e.g., guides, reviews, or quick facts). SEO specialists will analyse what types of content performs well for target topics and aim to match that format.

Finally, Google filters search results again, using personalisation factors like location to ensure a user’s search results are extremely relevant, ranking local businesses that include clear signals about their service area within their content.

Expertise and content depth

Google wants to recommend knowledgeable sources, especially for sensitive topics covering anything medical or financial. It assesses your Expertise, Authoritativeness, and Trustworthiness (E-A-T) through backlinks from reputable sites and your depth of content coverage on a subject.

A site that other websites have referenced as a source (through backlinks), serving content that covers all aspects of a topic signals to Google that they are an expert.

Freshness

It should come as no surprise that up-to-date information is important to Google, especially in industries that are evolving rapidly – think technology, politics, medicine, etc. It’s why an SEO specialist will likely recommend regularly refreshing your content, as this can significantly improve your rankings. Even in seemingly stable industries, the competitive nature of search results rewards content that builds upon what others have done.

That’s why it’s always worth revisiting your content periodically to check for new additions on your competitors’ sites and incorporate anything you may be missing.

User Experience (UX)

UX is becoming more important to Google as time goes on. In fact, Google’s ability to measure the UX of a site is one of the biggest differentiators, and a big reason why it is the leading search engine today.

While most people understand the basics of good user experience, like avoiding slow load times and ensuring your site works well on mobile phones, an SEO specialist often takes things a step further. They’ll concentrate on finding ways to answer questions clearly and quickly, with clear heading structure. They will break down content for easy scanning and use anchor links at the top of your pages to help users find important information.

Sites with a long history of SEO will also benefit from deep networks of strategically interlinked pages, guiding users and supporting certain pages with related content that dives deeper into topics. These are all reasons why SEO can take so long, and why it is so hard to catch up when you are behind.

The value of understanding search engines

Search engines are incredibly useful tools. They have the potential to provide incredible value to people using them to search, but for those looking to be found, their reasoning can seem vague and complicated. But there is a good reason ranking systems are complicated, because if they weren’t they would be manipulated even more than they already are.

At the end of the day, Google isn’t trying to help you, it is trying to help your customers. The sooner you ensure your site is communicating properly with search engines, the sooner you can start communicating with more customers – Ever wondered how many people are searching for your business?

Talk to us about talking to Google.

Alistair Hinchliffe

SEO Service Lead