Does Organic CTR Impact SEO Rankings? [New Data]

Posted by larry.kim

[Estimated read time: 13 minutes]

Does organic click-through rate (CTR) data impact page rankings? This has been a huge topic of debate for years within the search industry.

Some people think the influence of CTR on rankings is nothing more than a persistent myth. Like the one where humans and dinosaurs lived together at the same time — you know, like in that reality series “The Flintstones”?

Some other people are convinced that Google must look at end user data. Because how in the world would Google know which pages to rank without it?

Google (OK, at least one Google engineer who spoke at SMX) seems to indicate the latter is indeed the case:

Next time I hear a Googler say "we don't use query/click data in rankings," I'm pointing to https://t.co/qETZpGEbm9 pic.twitter.com/m9DO3cBVXb

— Rand Fishkin (@randfish) March 17, 2016

I also highly encourage you to check out Rand Fishkin’s Whiteboard Friday discussing clicks and click-through rate. In short, the key point is this: If a page is ranking in position 3, but gets a higher than expected CTR, Google may decide to rank that page higher because tons of people are obviously interested in that result.

Seems kind of obvious, right?

And if true, we ought to be able to measure it! In this post, I’m going to try to show that RankBrain may just be the missing link between CTR and rankings.

Untangling meaning from Google RankBrain confusion

Let’s be honest: Suddenly, everyone is claiming to be a RankBrain expert. RankBrain-shaming is quickly becoming an industry epidemic.

Please ask yourself: Do most of these people — especially those who aren’t employed by Google, but even some of the most helpful and well-intentioned spokespeople who actually work for Google — thoroughly know what they’re talking about? I’ve seen a lot of confusing and conflicting statements floating around.

Here’s the wildest one. At SMX West, Google’s Paul Haahr said Google doesn’t really understand what RankBrain is doing.

If this really smart guy who works at Google doesn’t know what RankBrain does, how in the heck does some random self-proclaimed SEO guru definitively know all the secrets of RankBrain? They must be one of those SEOs who “knew” RankBrain was coming, even before Google announced it publicly on October 26, but just didn’t want to spoil the surprise.

Now let’s go to two of the most public Google figures: Gary Illyes and John Mueller.

Illyes seemed to shoot down the idea that RankBrain could become the most important ranking factor (something which I strongly believe is inevitable). Google’s Greg Corrado publicly stated that RankBrain is “the third-most important signal contributing to the result of a search query.”

Illyes also said on Twitter that: “Rankbrain lets us understand queries better. No affect on crawling nor indexing or replace anything in ranking.” But then later clarified: “…it does change ranking.”

I don’t disagree at all. It hasn’t. (Not yet, anyway.)

Links still matter. Content still matters. Hundreds of other signals still matter.

It’s just that RankBrain had to displace something as a ranking signal. Whatever used to be Google’s third most important signal is no longer the third most important signal. RankBrain couldn’t be the third most important signal before it existed!

Now let’s go to Mueller. He believes machine learning will gain more prominence in search results, noting Bing and Yandex do a lot of this already. He noted that machine learning needs to be tested over time, but there are a lot of interesting cases where Google’s algorithm needs a system to react to searches it hasn’t seen before.

Bottom line: RankBrain, like other new Google changes, is starting out as a relatively small part of the Google equation today. RankBrain won’t replace other signals any time soon (think of it simply like this: Google is adding a new ingredient to your favorite dish to make it even tastier). But if RankBrain delivers great metrics and keeps users happy, then surely it will be given more weight and expanded in the future.

RankBrain headaches

If you want to nerd out on RankBrain, neural networks, semantic theory, word vectors, and patents, then you should read:

• Getting Your Head Around Google’s RankBrain by David Harry
• RankBrain: What Do We Know About Google’s Machine-Learning System? by Virginia Nussey (concentrate on Marcus Tober’s SMX presentation recap in the section “Machine Learning Ranks Relevance”)
• How Google Works: A Google Ranking Engineer’s Story by Kristi Kellogg

To be clear: my goal with this post isn’t to discuss tweets from Googlers, patents, research, or speculative theories.

Rather, I’m just going to ignore EVERYBODY and look at actual click data.

Searching for Rankbrain

Rand conducted one of the most popular tests of the influence of CTR on Google’s search results. He asked people to do a specific search and click on the link to his blog (which was in 7th position). This impacted the rankings for a short period of time, moving the post up to 1st position.

But these are all transient changes. Changes don’t persist.

It’s like how you can’t increase your AdWords Quality Scores simply by clicking on your own ads a few times. This is the oldest trick in the book and it doesn’t work.

The results of another experiment appeared on Search Engine Land last August and concluded that CTR isn’t a ranking factor. But this test had a pretty significant flaw — it relied on bots artificially inflating CTRs and search volume (and this test was only for a single two-word keyword: “negative SEO”). So essentially, this test was the organic search equivalent of click fraud. Google AdWords has been fighting click fraud for 15 years and they can easily apply these learnings to organic search. What did I just say about old tricks?

Before we look at the data, a final “disclaimer.” I don’t know if what this data reveals is definitively RankBrain, or another CTR-based ranking signal that’s part of the core Google algorithm. Regardless, there’s something here — and I can most certainly say with confidence that CTR is impacting rank. For simplicity, I’ll be referring to this as Rankbrain.

A crazy new experiment

Google has said that RankBrain is being tested on long-tail terms, which makes sense. Google wants to start testing its machine-learning system with searches they have little to no data on — and 99.9 percent of pages have zero external links pointing to them.

So how is Google able to tell which pages should rank in these cases? By examining engagement and relevance. CTR is one of the best indicators of both.

Head terms, as far as we know, aren’t being exposed to RankBrain right now. So by observing the differences between the organic search CTRs of long-tail terms versus head terms, we should be able to spot the difference:

We used 1,000 keywords in the same keyword niche (to isolate external factors like Google shopping and other SERP features that can alter CTR characteristics). The keywords are all from my own website: Wordstream.com.

I compared CTR versus rank for 1–2 word search terms, and did the same thing for long-tail keywords (4–10 word search terms).

Notice how the long-tail terms get much higher average CTRs for a given position. For example, in this data set, the head term in position 1 got an average CTR of 17.5 percent, whereas the long-tail term in position 1 had a remarkably high CTR, at an average of 33 percent.

You’re probably thinking: “Well, that makes sense. You’d expect long-tail terms to have stronger query intent, thus higher CTRs.” That’s true, actually.

But why is that long-tail keyword terms with high CTRs are so much more likely to be in top positions versus bottom-of-page organic positions? That’s a little weird, right?

OK, let’s do an analysis of paid search queries in the same niche. I use organic search to come up with paid search keyword ideas and vice versa, so we’re looking at the same keywords in many cases.

Long-tail terms in this same vertical get higher CTRs than head terms. However, the difference between long-tail and head term CTR is very small in positions 1–2, and becomes huge as you go out to lower positions.

So in summary, something unusual is happening:

• In paid search, long-tail and head terms do roughly the same CTR in high ad spots (1–2) and see huge differences in CTR for lower spots (3–7).
• But in organic search, the long-tail and head terms in spots (1–2) have huge differences in CTR and very little difference as you go down the page.

Why are the same keywords behaving so differently in organic versus paid?

The difference (we think) is that RankBrain is boosting the search rankings of pages that have higher organic click-through rates.

Not convinced yet?

Which came first: the CTR or the ranking?

CTR and ranking are codependent variables. There’s obviously a relationship between the two, but which is causing what? In order to get to the bottom of this “chicken versus egg” situation, we’re going to have to do a bit more analysis.

The following graph takes the difference between an observed organic search CTR minus the expected CTR, to figure out if your page is beating — or being beaten by — the expected average CTR for a given organic position.

By only looking at the extent by which a keyword beats or is beaten by the predicted CTR, you are essentially isolating the natural relationship between CTR and ranking in order to get a better picture of what’s going on.

We found on average, that if you beat the expected CTR, then you’re far more likely to rank in more prominent positions. Failing to beat the expected CTR makes it more likely you’ll appear in positions 6–10.

So, based on our example of long-tail search terms for this niche, if a page:

• Beats the expected CTR for a given position by 20 percent, you’re likely to appear in position 1.
• Beats beat the expected CTR for a given position by 12 percent, then you’re likely to appear in position 2.
• Falls below the expected CTR for a given position by 6 percent, then you’re likely to appear in position 10.

And so on.

Here’s a greatly simplified rule of thumb:

The more your pages beat the expected organic CTR for a given position, the more likely you are to appear in prominent organic positions.

If your pages fall below the expected organic search CTR, then you’ll find your pages in lower organic positions on the SERP.

Want to move up by one position in Google’s rankings? Increase your CTR by 3 percent. Want to move up another spot? Increase your CTR by another 3 percent.

If you can’t beat the expected click-through rate for a given position, you’re unlikely to appear in positions 1–5.

Essentially, you can think of all of this as though Google is giving bonus points to pages that have high click-through rates. The fact that it looks punitive is just a natural side effect.

If Google gives “high CTR bonus points” to other websites, then your relative performance will decline. It’s not that you got penalized; it’s just you’re the only one who didn’t get the rewards.

A simple example: The Long-tail Query That Could

Here’s one quick example from our 1000-keyword data set. For the query: “email subjects that get opened,” this page has a ridiculously high organic CTR of 52.17%, which beats the expected CTR for the top spot in this vertical by over 60%. It also generates insanely great engagement rates, including a time on page of over 24 minutes.

We believe that these two strong engagement metrics send a clear signal to Google that the page matches the query’s intent, despite not having an exact keyword match in the content.

What does Google want?

A lot of factors go into ranking. We know links, content, and RankBrain are the top 3 search ranking factors in Google’s algorithm. But there are hundreds of additional signals Google looks at.

So let’s make this simple. Your website is a house.

This is a terrible website. It was built a long time ago and has received no SEO love in a long time (terrible structure, markup, navigation, content, etc). It ranks terribly. Nobody visits it. And those poor souls who do stumble across it wish they never had and quickly leave, wondering why it even exists.

This website is pretty good. It’s designed well. It’s obviously well-maintained. It addresses all the SEO essentials. Everything is optimized. It ranks reasonably well. A good amount of people visit and hang out a while since, hey, it has everything you’d expect in a website nowadays.

Now we get to the ultimate house. It has everything you could want in a website — beautifully designed, great content, and optimized in every way possible. It owns tons of prominent search positions and everyone goes here to visit (the parties are AMAZING) again and again because of the amazing experience — and they’re very likely to tell their friends about it after they leave.

People love this house. Google goes where the people are. So Google rewards it.

This is the website you need to look like to Google.

No fair, right? The big house gets all the advantages!

Wrong!

So now what the heck do I do?

A bunch of articles say that there’s absolutely nothing you can or should do to optimize your site for Rankbrain today, and for any future updates. I couldn’t disagree more.

If you want to rank better, you need to get more people to YOUR party. This is where CTR comes in.

It appears that Google RankBrain has been “inspired by” AdWords and many other technologies that look at user engagement signals to determine page quality and relevance. And RankBrain is learning how to assign ratings to pages that may have insufficient link or historical page data, but are relevant to a searcher’s query.

So how do you raise your CTRs? You should focus your efforts in four key areas:

1. Optimize pages with low “organic Quality Scores.” Download all of your query data from Google Search Console. Sort your data, figure out which of your pages have below average CTRs, and prioritize those — it’s far less risky to focus on fixing your losers because they have the most potential upside. None of these pages will get any love from RankBrain!
2. Combine your SEO keywords with emotional triggers to create irresistible headlines. Emotions like anger, disgust, affirmation, and fear are proven to increase click-through rates and conversion rates. If everyone who you want to beat already has crafted optimized title tags, then packing an emotional wallop will give you the edge you need and make your listing stand out.
3. Increase other user engagement rates. Like click-through rate, we believe you need to have higher-than-expected engagement metrics (e.g. time on site, bounce rate — more on this in a future article). This is a critical relevance signal! Google knows the expected conversion and engagement rates based on a variety of factors (e.g. industry, query, location, time of day, device type). So create 10X content!
4. Use social media ads and remarketing to increase search volume and CTR. Paid social ads and remarketing display ads can generate serious awareness and exposure for a reasonable cost (no more than $50 a day). If people aren’t familiar with your brand, bombard your target audience with Facebook and Twitter ads. People who are familiar with your brand are 2x more likely to click through and to convert.

Key summary

Whether or not RankBrain becomes the most important ranking signal (and I believe it will be someday), it’s smart to ensure your pages get as many organic search clicks as possible. It means more people are visiting your site and it sends important signals to Google that your page is relevant and awesome.

Our research also shows that achieving above-expected user engagement metrics result in better organic rankings, which results in even more clicks to your site.

Don’t settle for average CTRs. Be a unicorn among a sea of donkeys! Raise your organic CTRs and engagement rates! Get optimizing now!

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!