Whether you are considering hiring an SEO agency, or doing SEO yourself, getting your head around the basics will pay dividends long term. Understanding how search engines work gives you the knowledge to effectively communicate with an external SEO Agency or ideate a successful SEO strategy on your own. So, before you begin trying to improve how well your website ranks, start here, with an article optimised for you (not search engines).

Here is what we are going to cover:

What are search engines?

Search Engines could be thought of as the worlds hardest working librarians – finding, reading, and cataloguing every page, on every website of the internet, so that we can easily find the answers to important questions like “Who can fix my air conditioner?”, or “Is it safe for my dog to eat carrot?”.

Just like a librarian organises books by genre, search engines organise web pages by topics. But unlike books, pages on a website are not bound together, which has its advantages, particularly for those wondering if their dog can eat carrot. While a librarian may be able to recommend you a book on safe dog food, that still requires a stressful navigation to the carrot chapter. Search engines on the other hand can filter through every page on the internet to give you the most relevant and helpful webpage for your query in a few milliseconds, an entire page specifically on how good carrot is for dogs (with an included history on both dogs and carrots for good measure). Ladies and gentlemen, the future!

How search engines find your site

Search engines are constantly looking for new websites and webpages to add to their search results and share with the world. It’s never ending. But before they serve any website or webpage to the internet, search engines need to do two things:

  1. Crawl the site: Crawling is the process of browsing the web and discovering new webpages – just as a human would find your site).
  2. Index the site: Indexing is the process of reading and forming an understanding of the content on a website and its webpages then adding it to a database – this could be thought of like cataloguing).

Thankfully, this isn’t done by some poor human, it’s done by a program called a crawler. Crawlers (AKA Spiders, Robots, or Bots) explore the web following links from one webpage to another and forms connections between them. There are many crawlers out there using the web, but there is one that everyone on the planet has come to know fairly well.

Googlebot

Googlebot is Google’s crawler, and while it navigates the web in the same way as other crawlers, Googlebot is smarter. It prioritises crawling efficiently, respects websites, renders pages to understand user experience, and feeds data directly into Google’s search algorithms. Which is why almost everyone agrees that Google is better at answering queries than other search engines like Bing or Yahoo.

How search engines discover a site for the first time

If you have just launched a new website, there are a few ways that search engines might find you for the first time.

  • Direct submission: Website owners can directly submit their URLs to search engines. Google (https://www.google.com/webmasters/tools/submit-url), Bing, and others provide tools for manual submission.
  • Domain registrars: Sometimes, domain registrars (companies that sell domain names) have agreements with search engines to inform them about new domain registrations.
  • Following links: Search engines are constantly crawling known websites. If they discover links to a new domain from a page they already know, the crawler can follow that trail and begin exploring the new domain.
  • Social media: Search engines often monitor social media for new domains and links being shared.

How Google determines how well a site ranks

Search engines each have their own unique website performance rating systems to deciding which content deserves to rank higher. Search Engines Optimisation Specialists (SEOs) call these ‘Ranking Factors’ and each of them plays a role in whether your site or your competitors comes first. To save you time, let’s just focus on Google for now – at the end of the day, it’s probably the search engine you are most interested in.

Google’s ranking factors

While no one truly knows exactly what Google’s algorithm is looking for, SEOs have developed a good understanding over time. Most agree that when Google is looking for an answer to a query, it is searching for up to date, relevant content, written by an expert, that covers the topic thoroughly but efficiently. If you want to know exactly how we optimise for each of these things, we’re working on a more comprehensive article to go a bit deeper on the subject – for now, here’s an overview.

Relevance

Google prioritises content that directly answers the searcher’s question or goal, this equates to more than just looking at keywords (though keywords are still the most important part of this). Each time Google updates its algorithm, it gets closer to the end goal of all search engines; to grasp the searcher’s true intent so that it can provide the most relevant content for any given search query.

What is relevant content?

To appear in relevant searches content needs to answer specific questions quickly, thoroughly, and in a format that is most useful to the reader. This could mean using keywords strategically, but it also involves understanding the content format that searchers will find most helpful (e.g., guides, reviews, or quick facts). SEO specialists will analyse what types of content performs well for target topics and aim to match that format. Finally, Google filters search results again, using personalisation factors like location to ensure a user’s search results are extremely relevant. They also rank local businesses that include clear signals about their service area within their content.

Expertise and content depth

Google wants to recommend knowledgeable sources, especially for sensitive topics covering anything medical or financial. It assesses your Expertise, Authoritativeness, and Trustworthiness (E-A-T) through backlinks from reputable sites and your depth of content coverage on a subject. When a site thoroughly addresses all aspects of a topic, it will lead other websites to reference it as a source (through backlinks). This in turn signals to Google that they are an expert.

Freshness

It should come as no surprise that up-to-date information is important to Google, especially in industries that are evolving rapidly – think technology, politics, medicine, etc. It’s why an SEO specialist will likely recommend regularly refreshing your content, as this can significantly improve your rankings. Even in seemingly stable industries, the competitive nature of search results rewards content that builds upon what others have done. That’s why it’s always worth revisiting your content periodically to check for new additions on your competitors’ sites and incorporate anything you may be missing.

User experience

User experience is becoming more important to Google as time goes on. In fact, Google’s ability to measure the user experience of a site is one of the biggest differentiators, and a key reason it is the leading search engine today.

While most people understand the basics of good user experience, like avoiding slow load times and ensuring your site works well on mobile phones, an SEO specialist often takes things a step further. They’ll concentrate on finding ways to answer questions clearly and quickly, with clear heading structure. They will break down content for easy scanning and use anchor links at the top of your pages to help users find important information. Sites with a long history of SEO will also benefit from deep networks of strategically interlinked pages, guiding users and supporting certain pages with related content that delves deeper into topics. These are all reasons why SEO can take so long, and why it is so hard to catch up when you are behind.

The value of understanding search engines

Search engines are incredibly useful tools. They have the potential to provide incredible value to people using them to search, but for those looking to be found, their reasoning can seem vague and complicated. But there is a good reason ranking systems are complicated, because if they weren’t they would be manipulated even more than they already are.

At the end of the day, Google isn’t trying to help you, it is trying to help your customers. The sooner you ensure your site is communicating properly with search engines, the sooner you can start communicating with more customers – Ever wondered how many people are searching for your business?

Talk to us about talking to Google.

Alistair Hinchliffe

SEO Specialist