seo

Posted by ronell-smith

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

Posted by Dom-Woodman

What will you learn from this post?

• How to get lots of Search Console data quickly and easily
• How to run a Python script

And who can do it? Hopefully, it should be accessible to any beginner.

Why do we use the API to get Search Console data?

At Distilled, we often want to use Google Search Console data, but getting it from the interface is incredibly clunky:

• You’re limited to the top 1,000 queries
• You have to apply each filter one at a time
• The interface is slow
• And if you want to do this regularly, you have to repeat this process often.

We can get around that by using the API. Now we can get up to 5,000 queries at a time, we can apply multiple filters instantly, and we can run multiple queries quickly and easily.

We do this with Python.

Why is it useful to be able to run Python scripts?

Being able to run scripts is incredibly valuable. There are lots of amazing scripts out there, both on Github and written by other people in the industry; using them, you can pull down data more quickly and faster than you otherwise could.

We’ll be using Python for this tutorial because it’s a very popular language, particularly when working with large amounts of data.

Crucially, you don’t need to be able to write in Python to use the scripts — you just need to understand some basics about how to use them.

With APIs you can pull data from all sorts of exciting places, far more quickly than through the user interface. You can also often get more data.

How do we run Python?

If you’re on a Mac or a PC, I’d recommend downloading Anaconda. That will get you set up and running with Python 3, and save a lot of fiddling around.

If you need administrator permission to install things on your work computer, then make sure you only install Anaconda for your user, not all users. If you try and install for all users, then you’ll need administrator permission.

Then we’re going to need a good shell (a command line interface, the place where you can run the script from). Mac has Terminal installed by default; on Windows, I would recommend Cmder.

Go ahead and install that.

(The rest of this tutorial is shown in Windows, but the same basic steps should be fine for a Mac!)

Double-check that Python has installed correctly

First open up the shell, type in python and hit enter.

Exit python by typing in exit().

Download our example script

For this example we’ll be using the search console script, written by one of our consultants, Stephan.

You can download it from his Github here. I’m not going to include a full tutorial on Git in this (although it’s a very useful tool for coding), so if you’re unsure how to clone a repository, just download the zip file:

Running our example script

Once we’ve downloaded the example script (and unzipped the folder, if necessary), we need to navigate in our shell to the folder where we just downloaded the script..

The command line functions like the Windows File Explorer or Finder that you normally use. Just like file explorer has a specific folder open, so does the command line, so we need to navigate to the folder where we have the script downloaded.

A command line shell functions a lot like a file explorer, only everything happens through text. You don’t get a mouse or a GUI.

Some command line basics

To change folders you’ll need some command line basics, most notably these two super-important commands:

1. cd [path]
2. ls -g

The first navigates you to the path given. (Or, if you use .. as your path, takes you a step backwards “cd ..”)

The second lists all the files and folders in the directory you’re currently in.

That’s all you need, but to make yourself faster there are two other things that are useful to know:

Hitting tab will cause the shell to try and complete the path you’re typing.

Suppose you’re in a folder with two files:

• Moz_1990_ranking_data.txt
• Moz_180_rankings.txt
• 180_rankings.txt

If you type:

• 180 and hit tab: It will autocomplete to 180_rankings.txt
• Moz and hit tab: It will autocomplete to Moz_

Secondly, hitting the up key goes through all the commands you’ve used. The reason a lot of people enjoy using the shell is they find it quicker than using a file explorer — and those two commands are a large part of that.

Congrats — now you’re ready to run the script. Next we need to get permission for the Google Search Console (GSC) API.

Turning on the API

In the same way you have to log in to see Search Console data, you need permission to use the API. Otherwise, anyone could get your data.

We also need to check whether the API is turned on — by default, it isn’t.

All the Google APIs live in the same place; Google Analytics is there, too. You can find them all at:

• https://console.cloud.google.com/apis/

You’ll need to sign in (making sure to use the Gmail account with access to your Search Console data). Then you can search for the Search Console API.

Once it’s selected, if it says “Enable here,” you’ll need to enable it.

Once that’s done we need to download an API key (which is equivalent to our password when signing into Search Console). A single API key gives you access to all of the Google services, in the same way that you use the same Gmail address to sign into Google Analytics and Search Console.

What is an API key? Different APIs have different types of keys. Sometimes it will just be a text string like “AHNSKDSJKS434SDJ”; other times it’s a file. The most common Google API key is a file.

So how do we get our Google API key? Once we’ve enabled the API, we select the “Credentials” tab and then create credentials. The three main kinds of API key are a basic text key, user OAuth credentials, and service account keys.

The first is quick and simple, the second is more secure and intended for users who will authenticate with a login, the third for automated data pulling.

There are some subtleties around permissions with these that we don’t really want to delve into here. The script is set up to use the second, so we’ll use that.

Go ahead and create an OAuth Client ID:

Ignore the pop-up and download the file from the credentials screen:

Move it to the same folder as your script. For ease of use, we’ll also rename it “credentials.json,” which is what the script is expecting the API key to be called. (A script will tell you what it’s expecting the API key to be called when you run it, or will have this in the documentation… assuming it’s well-written, of course).

Crucial note: By default, most versions of Windows will hide file extensions. Rather than naming the file “credentials.json,” you’ll accidentally name it “credentials.json.json.”

Because the file is already a JSON file, you can just name it “credentials” and check that the type is JSON. You can also turn on file extensions (instructions here) and then name it “credentials.json.”

In the screenshot below, I have file extensions visible. I’m afraid I don’t know if something equivalent exists in Mac — if you do, drop it in the comments!

Running our script

And we’re ready to go!

Hopefully now you’ve navigated to the folder with the script in using cd:

Now we try and run the script:

We get a module missing error. Normally you can solve this by running:

• pip install missing_module — or, in our case,
• pip install httplib2

And because we’ll get several of these errors, we need to install a couple modules.

• pip install oauth2client
• pip install –user –upgrade google-api-python-client

Interesting side point: It’s worth noting that the flag “–user” is the “pip” command line equivalent to the choice you often see when installing programs on a computer to install for all users or just you. (We saw this with Anaconda earlier.) If you do see permissions errors appearing in the command line with pip, try adding –user. And back to our script.

Now that we’ve installed all the things the script needs, we can try again (remember, you can just press up to see the previous command). Now we should get the script help, which will tell us how to run it. Any well-documented script should return something like this:

First, pay attention to the last line. Which arguments are required?

• property_uri
• start_date
• end_date

Our script needs to have these 3 arguments first in that order. Which looks like:

python search_console_query.py <a href="https://www.distilled.net/">https://www.distilled.net/</a> 2017-02-05 2017-02-06

Run that command and remember to change the URL to a property you have access to!

Your browser will open up and you’ll need to log in and authenticate the script (because it’s the first time we’re running the script):

You should be taken to a page that doesn’t load. If you look at the script, it’s now asking for an authentication code.

This is in the URL of the page, everything from the = up to the hash, which you’ll need to copy and paste back into the script and hit enter.

Check your folder where you saved the script and it should now look something like this:

The permission we gave the script is now saved in webmaster_credentials.dat. Each of our days of Search Console data we asked for sits in those CSV files. The script is designed to pull data for each day individually.

If we look back at our script options:

We can see some of the other options this script takes that we can use. This is where we can filter the results, change the country, device, etc.

• “Pages” takes a file of pages to individually query (example file)
  • By default, it pulls for the entire property.
• “Devices” takes a space-separated list
  • By default, it queries mobile, desktop, and tablet.
• Countries
  • By default, it does worldwide. Takes a space-separated list of country codes.
• By default the script will pull 100 rows of data per day. The API allows a limit of up to 5,000.

Here are some example queries using those options and what they do:

#get top queries for the search console property

python search_console_query.py https://www.distilled.net/ 2017-02-05 2017-02-06

#get top queries for multiple pages stored in file_of_pages and aggregate together

python search_console_query.py https://www.distilled.net/ 2017-02-05 2017-02-06 --pages file_of_pages

#get top queries for the property from desktop and mobile

python search_console_query.py https://www.distilled.net/ 2017-02-05 2017-02-06 --devices desktop mobile

#get the top queries for the property from the US & the UK

python search_console_query.py https://www.distilled.net/ 2017-02-05 2017-02-06 --countries USA GBR

#get the 5000 top queries for the property

python search_console_query.py https://www.distilled.net/ 2017-02-05 2017-02-06 --max-rows-per-day 5000

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

Posted by SimonPenson

Incredible, isn’t it? Despite all the fanfare and pageantry that has followed content marketing over the last few years, fewer than 6% of marketers confidently claim to be executing content marketing strategies properly.

It’s just one of a handful of eye-popping stats to come out of the State of Content Marketing Survey, a major new survey of senior UK marketers this month as part of a campaign to help create healthy debate around the misunderstood tactic.

With more budget than ever before pouring into the approach (60% of those surveyed said they were opening the purse strings further in 2017) 92% admitted to not knowing exactly how they should execute.

To check out all the results from the survey, click below (opens up in a new tab):

The biggest pain point of all to come out of the State of Content Marketing survey?

“Producing engaging content, consistently.”

I had been reading all the results with mild interest until those words stopped me dead in my tracks.

You may think the source of that concern stemmed from the fact that such a thing should be easy to manage, but it goes deeper than that.

Success with content is predicated entirely on your ability to consistently produce content that engages, resonates and adds value to your audience’s lives. And if producing that is the single biggest barrier then we have a problem!

You see, investment in content is a waste of money if you don’t have a well-designed plan to deliver constant content.

It doesn’t matter how brilliant your campaigns are if your audience has no other content to come back to and engage with.

And this is where the constant content plan comes in…

Constant content

The concept is a simple one: no content plan is complete unless it’s based around delivering content consistently.

To do this requires a focus on strategy, not just on a few blog posts and the odd bigger campaign.

The best way to explain this is to visualize it in a different way. Below, you’ll see a simple diagram to throw light on my point.

Here we can see how a campaign-led strategy exposes holes in your plan. While we have plenty of activity going on in both our owned and earned channels, the issue is what goes on between large content launches. Where do those people go during those periods of inactivity? How do we keep them engaged when there’s no central content hub to pull them into?

This kind of approach is something we see often, especially from larger brands where budgets allow for more creative content campaigns to be run regularly, and here’s why it doesn’t yield positive ROI.

As human beings, we like variety. To keep us hooked, content delivery needs to reflect this. Campaigns need to be designed as part of a whole, becoming a peak content moment rather than the only content moment, pulling new audiences back to the constant content activity going on at the center of brand activity.

You see it in the way magazines are organized, starting with an initial section of often short-form content before you then hit a four-plus-page feature. This is done to ensure we keep turning the pages, experiencing variation as we do so.

This is something I like to call content flow. It’s a great strategic “tool” to help ensure you design your overall strategy the right way.

The approach to strategy

The key is actually very simple. It focuses the mind on the creation of a content framework that enables you to produce lots of high-quality regular content and the ideas that flow from it. I call it the “Constant Content Plan.”

The right way to approach the content planning phase is to create a process that supports the building of layers of different content types, like we see below in our second diagram:

In this example, you can see how we intersperse the bigger campaigns with lots of owned content, creating a blog and resources section that gives the new visitor something to explore and come back to. Without it, they simply float back out into the content abyss and onto someone else’s radar.

That consistent delivery — and the audience retention it creates — comes from the smaller content pieces, the glue that binds it together; the strategy in its entirety.

“Smaller” doesn’t mean lower-quality, however, and investing lots of time through the ideation phase for these pieces is critical to success.

Creating smaller ideas

To do this well and create that constant content strategy, a great place to start is by looking at the ideas magazines use. For example, these are the regular content types you often find in the best-crafted titles:

1. What I’ve learned
   Advice piece from a heavy-hitter. Can sometimes be expanded to what I’ve learned in my 20s, 30s, 40s, etc.
2. The dual interview
   Get two people together for an interview. Write an intro as to why they’re there, and then transcribe their chat. Bingo: unique content.
3. Have you ever/What do you think of?
   Pose a question and ask ten people for their responses. Good reactive content to a particular event that might pertain to one of our clients.
4. Cash for questions
   Get an interviewee/expert and pose them a series of questions gathered from real-life members of the public.
5. A day in the life
   What it says on the tin — an in-depth look at someone of interest’s working day.
6. Person vs person debate
   Start with a question or subject matter, get two people, put it to them, and record the results.
7. Master xxxxxx in five minutes
   A short how-to — can be delivered in pictorial or video format.

This style of regular series content lends itself well to online strategy, too. By running these regularly, you create both variety and the critical stickiness required to keep the audience coming back.

Of course, with such variation it also then allows you to create better newsletters, social strategies, and even inbound marketing plans, maximizing that return on investment.

The strategy allows for informative content as well as entertaining pieces. In doing so, it gives your brand the opportunity to build subject trust and authority, as well as capturing key opportunities in the purchase funnel such as micro-moments and pain points.

This combination of informative and entertaining output ensures you’re front and center when your customer eventually falls into the purchase funnel.

Some examples

One way of bringing this to life is to look at brands already executing well.

One of the best blog strategies I have seen in some time is the one by Scotts Menswear. One of the key reasons for its quality is the fact it’s run by a very experienced print editor.

If we reverse-engineer what they’ve been doing on-page, we can clearly see that much thought has gone into creating variation, entertainment, and usefulness in a single well-rounded strategy.

Take the last ten posts, for instance. Here’s what we have and how it flows:

• Seven Films We’re Looking Forward to in 2017 – Video-based entertainment/lifestyle piece.
• Key Pieces for Your January Fitness Drive – Trending content with useful advice.
• Style Focus – A great regular piece that jumps on trending “news” to discuss the implications for fashion.
• Updated Classics from Puma – A news article on a new trainer release.
• Polo Shirts: A Wardrobe Staple – An in-depth guide to a key piece of clothing (part of a series).
• Our Guide to Valentine’s Day – Lifestyle guide that helps convey brand positioning, tonality, and opinion.
• Nail Your Valentine’s Day Outfit – Helpful guide to getting it right on a key seasonal event in the audience’s calendar. Clearly, they see Valentine’s as a sales peak.
• Get Your Overhead Jacket Kicks – Guide to a fashion staple.
• 5 Brands and Acts Tipped for Greatness – Lifestyle piece tapping into the music/fashion brand positioning.
• Our 5 Favorite Trainers Online Right Now – Great list feature to help the consumer buy smarter.

You can clearly see how they’re using structured thinking to create a blog of real variety and value. By combining this with a strong big-bang content plan that sucks in new visitors, you can build a hefty retained audience that improves critical metrics such as dwell time, returning visits, engagement, and sales.

Building our own plan

I know what you’re thinking. “Sounds great, for a brand in fashion. It’s cool and interesting. But I work in a ‘boring’ niche and this type of stuff just isn’t possible.”

While it could be a little more difficult that doesn’t mean it is impossible by any stretch of the imagination.

To prove the point, let’s look at a fictional example for a company in the medical products sector.

Here’s the deal: A2Z Medical is a company built up in the ’60s and ’70s. They have a huge B2B footprint but want to bring their marketing strategy into the current decade, in part because they are launching a consumer-facing brand for the first time. The new venture will offer medical kits for the general public and as such requires a proactive, content-led strategy to promote trust, awareness, and engagement alongside the obvious requirement for sales.

Audience research

The first step in building a content strategy is to understand your audience.

We could go into the detail of that all day long, but for the sake of this example we already have detailed data that tells us there are two main groups of people interested in coming to and buying from the site.

I’ve also written about the process I go through to define personas, and would always recommend this post too for background.

James is an obsessive ailment Googler, worrying over every little thing that he or his family suffers. He’s a detail man and wants to be prepared for all eventualities.

Chloe, on the other hand, has very different needs. She’s a mum, works part-time to help pay the bills, and then devotes herself to her family and children.

She’s time-poor and takes a practical view on life to make it work. Her purchase behavior is based on distress or urgent need.

Different need states

It is abundantly clear from this very quick overview that each have very different purchase journeys and needs from a content perspective.

We’ll look at what this means for our content strategy a little later. Before we dive into that, though, we must also look at our understanding of the market opportunity.

This data-dive helps us to understand what people are looking for now in the space, where they get it from currently, and where the gaps may be.

The data-dive

This work is carried out by one of our content strategists before any creative sessions take place. This ensures we can validate ideas back to what the data tells us.

So, what does that process involve? Let’s look at each stage briefly now:

1. Long-tail research
2. Quora/Reddit/forum research
3. Magazine research
4. Pub beers!

It’s a well-covered subject area, but also a very important one; it often yields ideas that convert fastest to traffic and revenue.

1. Long-tail research

Much has been written (including this piece I penned in 2015) on this subject area, and in much more detail than I aim to cover it here. Right now, let’s focus on some key tools and areas for opportunity.

It’s easy to get lost in this process, so the key is to keep it simple. To do this, I stick to a small handful of tools:

• SerpStat – Has a useful long-tail tool based on Google Suggest to give you lists of questions by keyword phrase.
• Keywordtool.io – A similar tool, but free to use. Slightly more clunky.
• Bloomberry – A new tool by the makers of Buzzsumo. Does a great job of finding opportunities from other sources, such as other sites and forums. It also has a nice data visualization view that gives you volume and key competitor info, the latter of which can be helpful for a later stage in the process. Here’s an example of a search for “first aid”:

• Storybase – A free tool that pulls long-tail phrases from a variety of sources for content ideas and also includes some demographic data. This can be helpful when it comes to matching ideas to personas.

For the sake of this process, we’re not looking to build a full long-tail strategy, of course. This is solely about finding content ideas with search volume attached to them.

By downloading from a bunch of sources (such as those above), it’s then relatively easy to de-dupe them in Excel and create a master list of ideas to pull into your overall plan.

It can make sense to segment or classify those ideas by persona, too. I do this via simple color coding, as you can see below. This allows you to create a shortlist of ideas that are on-brand and have the required level of opportunity attached.

Working this way makes sure you’re thinking hard about serving the needs and pain points of the personas.

To further reinforce this point, it can work very well to include a mini-brainstorm as part of this stage, gathering a few people to talk specifically about the pain points experienced by each persona.

In this session, it’s also useful to talk through the various micro-moment opportunities by asking what questions they ask in each of the following scenarios:

I want to go….
I want to do…
I want to know…
I want to buy…

You should end up with a list of content ideas per persona that covers pain points and interests.

2. Quora/Reddit/forum research

Another great source of information is the world of forums and aggregator sites. As you might expect, this starts with sub-Reddit research.

Within categories like those below lies a wealth of questions, the answers to which form brilliant article inspiration:

If we pop into the /AskDocs/ forum, we see a plethora of medical challenges from people looking for help — perfect real-world examples of everyday ailments that a site like ours could help to answer.

Q: I have a painful stomach when eating pork…?

Q: Will I need less sleep if I’m on a good diet and active?

Q: Swollen lymph nodes and nose bleeds. What could be going on?

The answers to these questions often require much research and professional advice, but by working through them for the less-serious everyday issues you could soon help Chloe out and become a useful ally.

The same is also true of Quora. You can play around with advanced search queries to drill into the juiciest boards by carrying out searches such as:

Another fantastic area worthy of research focus is forums. We use these to ask our peers and topic experts questions, so spending some time understanding what’s being asked within your market can be very helpful.

One of the best ways of doing this is to perform a simple advanced Google search as outlined below:

“keyword” + “forum”

For our example, we might type:

The search engine then delivers a list of super-relevant sites designed to answer medical questions and we can easily pick through them to extract ideas for popular content.

And as an extra tip search for your keyword and “vBulletin” – a popular software used for forum sites. This will often surface rarely found sites with some real insight into particularly the older demographic, who are more likely to use traditional forums.

3. Magazine research

Another very important area to explore is magazine research. They contain some of the most refined content strategies in existence; the level of expertise that goes into idea creation and headline writing is without equal.

It makes sense, therefore, to find titles relevant to your niche (in our case, health and medicine) and look for great content opportunities.

You can even do this online, to a degree. If you go to a site like magazines.com, greatmagazines.co.uk, other magazine subscription sites, or even perform a Google image search, you’ll find a myriad of headline ideas simply by looking at covers.

In the example below I’ve Googled “medical magazines” and found numerous cover lines that would form great digital content. Here’s an example from just one, 4Health Magazine:

4. Pub beers

And last but certainly not least, we have the tried and true “chat-in-a-pub” approach. It might sound like an excuse for a beer, but it’s actually very useful.

If you can find a handful of people aligned to your personas, offer to buy them a few drinks and chat through their experiences and challenges. You’ll be surprised what you find out!

Product range

Of course, it pays to add some level of alignment to the plan by understanding which products offer the best margin or are most important to the business.

This info should come out of your initial onboarding and overall strategy creation process, but it can also be found via Analytics (if set up correctly) by looking for the best-selling products and finding out their trade cost.

The creative process

By this point, you’ll be overflowing with data and ideas for content. The challenge, however, is ensuring that you can add variation to that ideas mix. I call this stage the “Magazine and Hero Process.”

Magazine ideas

To create that level of engagement and stickiness, we need ideas that are less practical and more entertaining. Any good content strategy should include a good mix of both informational and entertaining ideas; the first part of our creative brainstorm focuses on concepts that will achieve this balance.

We follow a structure that looks loosely like the below:

Stage one:

We start by asking “human” questions about each of our personas. While we may have completed all the keyword research in the world, it’s important to take a real-world view on pain points and so forth.

From here we discuss the purchase funnel stage, ensuring that we have ideas not just for the top of the funnel but all the way through it, backed by a mix of content types to support that variation aim.

That conversation will then be followed by a look at the brand’s wider marketing plan and seasonal events to ensure we plan key periods of activity thoroughly.

And the icing on the top is the quick look at our “swipe file,” a treasure trove of old ideas we’ve seen, to see if we can borrow a concept or two for our plan.

Stage two:

The second and final stage of our ideation is a forensic exploration into what magazines can offer. I am a voracious devourer of specialist magazines; it can really pay dividends to look for clever ideas or content series to bring into your plan before the massive validation process begins. This will sort the possible from the impossible.

By following a set way of discussing ideas, you’ll leave no stone unturned.

The discussion around the purchase funnel often turns out to be incredibly important: it ensures you look not just for ideas that help with awareness, but also further down the funnel. It’s also possible to tie content types in to this to ensure variation between the types of content you produce.

To do this we use the Content Matrix I created specifically for this purpose; you can see it below:

The idea here is that it makes it easier to decide what content types fit with which parts of the funnel best and also the relative size of that content in terms of the man hours required to create it.

Working in this logical fashion will help with overall content mix.

Hero ideas

Once you’ve worked through that process, it’s time to open up bigger ideas. These are important for one very simple reason: they help you find and reach new audiences to pull back into your sensational constant content plan.

We won’t go into detail here as to how to come up with consistently good big-bang ideas, as the point of this post is to look at the more regular content strategy, but if you want to read more about it click here.

For now, it’s enough to note that you should also include time to think about campaigns and how they fit into your overall plan.

Pulling it together – process + example

By now, you should be swimming in great ideas of every kind imaginable, every one of which ties back nicely to your personas.

In our example, we’ve been focusing on Chloe and James. The next job is to lay those ideas out based on what you know can be delivered.

This process is broken into two parts:

• The laying out of the content based on ensuring variation and content flow.
• Fitting that plan into an operational format that’s deliverable and based on available resources and/or budgets.

Getting that to work is little more than trial and error, but the result should be a content calendar that delivers on the promise of a great mix of regular content ideas, entertaining pieces, and helpful content that makes both James and Chloe want to come back to again and again.

Here’s an example of a two-week window to give you an idea of how just a portion of that regular content might play out:

Free downloads

Fancy giving it a go? You can use this free brand-as-publisher download to make the process easier. It contains all the tools and templates you need to ensure your output joins up the dots to maximize engagement and ‘stickiness’ from your regular content and to critically fix your issues with content marketing effectiveness.

And for those of you that want to see the Content Marketing Survey results in full click on the banner above to claim your free results ebook, complete with commentary, or scroll below for the highlights…

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

Posted by MarieHaynes

It has now been six months since the launch of Penguin 4.0. In my opinion, Penguin 4.0 was awesome. It took ages for Google to release this update, but when they did, it was much more fair than previous versions of Penguin. Previous versions of Penguin would cause entire sites to be suppressed if the algorithm thought that you’d engaged in manipulative link building. Even if a site did a thorough link cleanup, the suppression would remain present until Google re-ran the Penguin algorithm and recognized your cleanup efforts. It was not uncommon to see situations like this:

I saw many businesses that had looooooong periods of time of suppression — even years!

According to Google spokesperson Gary Illyes, the new version of Penguin that was released in September of 2016 no longer suppresses sites:

Now, instead of causing a sitewide demotion when Penguin detects spam, they’ll simply devalue that spam so that it can’t help improve a site’s rankings.

I’m guessing that it took a lot of brainpower to figure out how to do this. Google now has enough trust in their ability to find and devalue spam that they are comfortable removing the punitive aspect of Penguin. That’s impressive.

This change brings up a question that I am asked several times a week now:

If Penguin is able to devalue spam, is there any reason to disavow links any more?

I’ve been asked this enough times now that I figured it was a good idea to write an article on my answer to this question.

A brief refresher: What is the disavow tool?

The disavow tool was given to us in October of 2012.

You can use it by uploading a file to Google that contains a list of either URLs or domains. Then, as Google crawls the web, if they come across a URL or domain that is in your disavow file, they won’t use links from that page in their calculations of PageRank for your site. Those links also won’t be used by the Penguin algorithm when it decides whether your site has been involved in webspam.

For sites that were affected by Penguin in the past, the disavow tool was an integral part of getting the suppression lifted off the site. It was essentially a way of saying to Google, “Hey… in the past we made some bad links to our site. But we don’t want you to use those links in your calculations.” Ideally, it would be best to remove bad links from the web, but that’s not always possible. The disavow tool was, in my opinion, super important for any site that was hit by Penguin.

For more in-depth information on using the disavow tool, see this Moz post: https://moz.com/blog/guide-to-googles-disavow-tool

What does Google say about using the disavow tool now?

It wasn’t long after the release of Penguin 4.0 before people starting asking Google whether the disavow tool was still necessary. After all, if Google can just devalue spam links on their own, why should I have to disavow them?

Here are some replies from Google employees:

Now, the conspiracy theorists out there will say, “Of course Google wants you to disavow! They need that data to machine-learn for Penguin!”

Google has said that Penguin is not a machine learning algorithm:

And even if they ARE using disavow data for some kind of machine learning training set, really, does it matter? In my opinion, if Google is saying that we should be still using the disavow tool, I don’t think they’re trying to trick us. I think it still has a real purpose.

Three reasons why I still recommend using the disavow tool

There are three main reasons why I still recommend disavowing. However, I don’t recommend it in as many cases as I used to.

1) Manual actions still exist

You do NOT want to risk getting a manual unnatural links penalty. I have documented on Moz before about the many cases I’ve seen where a manual unnatural links penalty was devastating to the long-term health of a site.

Google employee Gary Illyes commented during a podcast that, when a Google webspam team member looks at your site’s links, they can often see labels next to the links. He said the following:

“If the manual actions team is reviewing a site for whatever reason, and they see that most of the links are labeled as Penguin Real-Time affected, then they might decide to take a much deeper look on the site… and then maybe apply a manual action on the site because of the links. ”

In other words, if you have an unnatural link profile and you leave it up to Penguin to devalue your links rather than disavowing, then you’re at risk for getting a manual action.

Of course, if you actually do have a manual action, then you’ll need to use the disavow tool as part of your cleanup efforts along with manual link removal.

2) There are other algorithms that use links

Link quality has always been important to Google. I believe that Penguin is just one way in which Google fights against unnatural links algorithmically. One example of another algorithm that likely uses links is the Payday Loans algorithm. This algorithm isn’t just for payday loans sites; it also affects sites in many high-competition verticals.

Bill Slawski recently posted this interesting article on his thoughts about a recent patent filed by Google. In one place, the patent talks about a situation where a resource may have a large number of links pointing to it but there is a disproportionate amount of traffic. In cases like that, the page being linked to might actually be demoted in rankings.

Now, that’s just a patent, so it doesn’t mean for sure that there’s actually an algorithm behind this… but there could be! Makes you think, right?

Google is always trying to fight against link spam and Penguin is just one of the ways in which they do this. If there are links that are potentially causing my link profile to look spammy to Google, then I don’t want them to count in any calculations that Google is making.

3) Can we trust that Penguin is able to devalue all spam pointing to our site?

The official announcement from Google on Penguin is here. Here’s what it says about devaluing as opposed to demoting:

“Penguin is now more granular. Penguin now devalues spam by adjusting ranking based on spam signals, rather than affecting ranking of the whole site.”

This statement is not clear to me. I have questions:

• When Google says they are “adjusting ranking,” could that also be negative adjustments?
• Can Penguin possibly demote rankings for certain pages rather than affecting the whole site?
• Can Penguin possibly demote rankings for certain keywords rather than affecting the whole site?

As posted above, we received some clarification on this from Google employees in a Facebook post (and again via tweets) to tell us that Penguin 4.0 doesn’t penalize, but rather devalues spam. However, these are not official statements from Google. These statements may mean that we never have to worry about any link pointing to our site ever again. Perhaps? Or they could mean that there’s less need to worry than there was previously.

Personally, if my business relies on Google organic rankings in order to succeed, I’m a little leery about putting all of my trust in this algorithm’s ability to ignore unnatural links and not let them hurt me.

Who should be disavowing?

While I do still recommend use of the disavow tool, I only recommend it in the following situations:

1. For sites that have made links for SEO purposes on a large scale – If you or an SEO company on your behalf made links in low-quality directories, low-quality article sites, bookmark sites, or as comment spam, then these need to be cleaned up. Here’s more information on what makes a link a low-quality link. You can also run links past my disavow blacklist if you’re not sure whether it’s a good one or not. Low-quality links like this are probably being devalued by Penguin, but they’re the type of link that could lead to a manual unnatural links penalty if you happen to get a manual review by the webspam team and they haven’t been disavowed.
2. For sites that previously had a manual action for unnatural links – I’ve found that if a site has enough of a spam problem to get an unnatural links penalty, then that site usually ends up collecting more spam links over the years. Sometimes this is because low-quality directories pop up and scrape info from other low-quality directories. Sometimes it’s because old automated link-generating processes keep on running. And sometimes I don’t have an explanation, but spammy links just keep appearing. In most cases, sites that have a history of collecting unnatural links tend to continue to collect them. If this is the case for you, then it’s best to disavow those on a regular basis (either monthly or quarterly) so that you can avoid getting another manual action.
3. For sites under obvious negative SEO attacks – The key here is the word “obvious.” I do believe that in most cases, Google is able to figure out that spam links pointed at a site are links to be ignored. However, at SMX West this year, Gary Illyes said that the algorithm can potentially make mistakes: If you have a bunch of pharma and porn links pointing at your site, it’s not a bad idea to disavow them, but actually in most cases I just ignore these. Where I do recommend disavowing for negative SEO attacks is when the links pointing at your site contain anchors for keywords for which you want to rank. If it’s possible that a webspam team member could look at your link profile and think that there are a lot of links there that exist just for SEO reasons, then you want to be sure that those are cleaned up.

Who does NOT need to disavow?

If you look at your links and notice some “weird” links that you can’t explain, don’t panic!

Every site gets strange links, and often quite a few of them. If you haven’t been involved in manipulative SEO, you probably do not need to be disavowing links.

When Google takes action either manually or algorithmically against a site for unnatural linking, it’s because the site has been actively trying to manipulate Google rankings on a large scale. If you made a couple of directory links in the past, you’re not going to get a penalty.

You also don’t need to disavow just because you notice sitewide links pointing to you. It can look scary to see in Google Search Console that one site is linking to you thousands of times, especially if that link is keyword-anchored. However, Google knows that this is a sitewide link and not thousands of individual links. If you made the link yourself in order to help your rankings, then sure, go ahead and disavow it. But if it just appeared, it’s probably nothing to worry about.

Borderline cases

There are some cases where it can be difficult to decide whether or not to disavow. I sometimes have trouble advising on cases where a company has hired a medium- to high-quality SEO firm that’s done a lot of link building — rather than link earning — for them.

Here’s an example of a case that would be difficult:

Let’s say you’ve been getting most of your links by guest posting. These guest posts are not on low-quality sites that exist just to post articles, but rather on sites that real humans read. Are those good links?

According to Google, if you’re guest posting primarily for the sake of getting links, then these are unnatural links. Here’s a quote from Google employee John Mueller:

“Think about whether or not this is a link that would be on your site if it weren’t for your actions…When it comes to guest blogging it’s a situation where you are placing links on other people’s sites together with this content, so that’s something I kind of shy away from purely from a link building point of view. It can make sense to guest blog on other people’s sites to drive some traffic to your site… but you should use a nofollow.”

If you have a small number of guest posts, Google is unlikely to go after you. But what if a webspam team member looks at your links and sees that you have a very large number of links built via guest posting efforts? That makes me uncomfortable.

You could consider disavowing those links to avoid getting a manual action. It’s quite possible, though, that those links are actually helping your site. Disavowing them could cause you to drop in rankings.

This article could easily turn into a discussion on the benefits and risks of guest posting if we had the space and time. My point in mentioning this is to say that some disavow decisions are tough.

In general, my rule of thumb is that you should use the disavow file if you have a good number of links that look like you made them with SEO as your primary goal.

Should you be auditing your disavow file?

I do believe that some sites could benefit from pruning their disavow file. However, I have yet to see any reports from anyone who has claimed to have done this and seen benefit that we can reasonably attribute to the recovery of PageRank that flows through those links.

If you have used your disavow file in the past in an effort to remove a manual action or recover from a Penguin hit, then there’s a good possibility that you were overly aggressive in your disavow efforts. I know I’ve had some manual penalties that were really difficult to remove and we likely disavowed more links than were necessary. In cases like those, we could go through our disavow files and remove the domains that were questionable disavow decisions.

It’s not always easy to do this, though, especially if you’ve done the correct thing and have disavowed on the domain level. If this is the case, you won’t have actual URLs in your disavow file to review. It’s hard to make reavowing decisions without seeing the actual link in question.

Here’s a process you can use to audit your disavow file. It gets a little technical, but if you want to give it a try, here it is:

(Note: Many of these steps are explained in greater detail and with pictures here.)

• Download your disavow file from Google: https://www.google.com/webmasters/tools/disavow-links-main
• Get a list of your links from Google Search Console. (It’s not a bad idea to also get links from other sources, as well.)
• On your CSV of links, make a column for domains. You can extract the domain by using this formula, assuming your URLs are in Column B:
  
  =LEFT(B1,FIND(“/”,B1,9)-1)
  
  You can then use Find and Replace to replace the http, https, and www. with blanks. Now you have a list of domains.
• On your disavow file, get a list of domains you’ve disavowed by replacing domain: with blanks. (This is assuming you have disavowed on the domain level and not the URL level.)
• Put your new list of disavowed domains on the second sheet of your links spreadsheet and fill Column B down with “disavowed”.
• Now, on the links list, we’re going to use a VLOOKUP to figure out which of our current live links are ones that we’ve previously disavowed. In this formula, your domains are in the first column of each spreadsheet and I’ve used 1000 as the total number of domains in my disavow list. Here goes:
  
  =VLOOKUP(A1,sheet2!$A$1:$B$1000,2,FALSE)
• Now you can take the domains that are in your disavow file and audit those URLs.

What we’re looking for here are URLs where we had disavowed them just to be safe, but in reality, they are probably OK links.

Note: Just as in regular link auditing work, do not make decisions based on blanket metrics. While some of these metrics can help us make decisions, you do not want to base your decision for reavowing solely on Domain Authority, spam score, or some other metric. Rather, you want to look at each domain and think, “If a webspam team member looked at this link, would they think it only exists for SEO reasons, or does it have a valid purpose outside of SEO?”

Let’s say we’ve gone through the links in our disavow file and have found 20 links that we’d like to reavow. We would then go back to the disavow file that we downloaded from Google and remove the lines that say “domain:example.com” for each of those domains which we want to reavow.

Upload your disavow file to Google again. This will overwrite your old file. At some point in the future Google should start counting the links that you’ve removed from the file again. However, there are a few things to note:

• Matt Cutts from Google mentioned in a video that reavowing a link takes “a lot longer” than disavowing. They built a lag function into the tool to try to stop spammers from reverse-engineering the algorithm.
• Matt Cutts also said in the same video that a reavowed link may not carry the same weight it once did.

If this whole process of reavowing sounds too complicated, you can hire me to do the work for you. I might be willing to do the work at a discount if you allow me to use your site (anonymously) as a case study to show whether reavowing had any discernible benefit.

Conclusions

Should we still be using the disavow tool? In some cases, the answer to this is yes. If you have links that are obviously there for mostly SEO reasons, then it’s best to disavow these so that they don’t cause you to get a manual action in the future. Also, we want to be sure that Google isn’t using these links in any sort of algorithmic calculations that take link quality into account. Remember, it’s not just Penguin that uses links.

I think that it is unlikely that filing a disavow will cause a site to see a big improvement in rankings, unless the site is using it to recover from a sitewide manual action. Others will disagree with me, however. In fact, a recent Moz blog post showed a possible recovery from an algorithmic suppression shortly after a site filed a disavow. I think that, in this case, the recovery may have been due to a big algorithm change that SEOs call Fred that happened at the same time, rather than the filing of a disavow file.

In reality, though, no one outside of Google knows for sure how effective the disavow tool is now. We know that Google says we should still use it if we find unnatural links pointing to our site. As such, my advice is that if you have unnatural links, you should still be disavowing.

I’d love to hear what you think. Please do leave a comment below!

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

Posted by Nick_Sayers

After five years at Moz, I’ve found that I work with some of the most creative and talented people I’ve ever met. When it came time to brainstorm ideas for an April Fools video this year, rather than tricking people like every other company does, we decided instead to showcase a few of the talented folks around the Mozplex. Alongside Kristina Keyser, we shot and edited this video so you can see what makes us Mozzers tick when we aren’t building sweet SEO software.

We know a lot of you, in the community, have random hobbies like us — we’d love to hear about them! Please share them with us in the comments.

Looking for more information on all the cool stuff you just saw? We have that for you, too!

Check out my movie, The Last Buck Hunt, on Amazon Prime.

If you want to sing Teenage Dirtbag like Chiaryn, start with this karaoke wiki.

David would like for you to check out his Seatlle-based band, Your Favorite Friend.

The Modern Quilt Guild is a great place to meet other quilters and find meet-ups to show off your work.

If you’re looking for a great community around knitting, Alex recommends Ravelry.

In the mood for some pop punk? Check out Kevin’s old band, Miracle Max. (But we also secretly think you should check out this article as well.)

Abe would like for you to know more about making old-school gaming beats by checking out Chip Music’s forum.

It’s really cool to see that Katie does martial arts at Seven Star Women’s Kung Fu, because it’s rare to see schools like this just for ladies.

Looking for beautiful pictures from around the world? Peep Kristina’s photography.

Looking to get super duper ripped like Lucas? Learn more about Bodypump.

Looking to adopt a dog in the Seattle area? Janisha recommends starting your adoption search now.

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!