About frans

Website:
frans has written 4625 articles so far, you can find them below.

How to Rank Well in Amazon, the US’s Largest Product Search Engine

Posted by n8ngrimm

The eCommerce SEO community is ignoring a huge opportunity by focusing almost exclusively on Google. But Google is the biggest search engine, right? Actually, if you are in eCommerce, Amazon should be far more important to you than Google, because it has roughly three times more search volume for products.

In 2012, The New York Times reported:

“In 2009, nearly a quarter of shoppers started research for an online purchase on a search engine like Google and 18 percent started on Amazon, according to a Forrester Research study. By last year, almost a third started on Amazon and just 13 percent on a search engine. Product searches on Amazon have grown 73 percent over the last year while searches on Google Shopping have been flat, according to comScore.”

When researching this post, I searched Moz.com for already-published material about ranking in Amazon. All I found was a single Q&A with five responses and little information. Conversely, there are many, many questions on Moz about how to rank your Amazon product pages in Google. It’s all very Google-focused.

I joined DNA Response and the eCommerce vertical from the world of education lead-gen where most of our traffic came from Google. I empathize with the Google myopia from which most SEOs suffer. My goal with this post is twofold:

  1. First, I want to convince all of the eCommerce search marketers to spend a lot more energy optimizing Amazon.
  2. Second, I want to provide marketers with a basic understanding of Amazon’s organic ranking algorithm.

Due to the lack of existing content on this topic, I felt the need to be somewhat comprehensive. I will address several conceptual problems I encountered when switching from a Google-focused niche to Amazon then share everything I have learned about Amazon’s ranking algorithms.

(Side note: I would like to apologize in advance that many of the links in my article require an Amazon Seller Central login to view. Amazon requires a login for most of their seller resources.)

Table of contents

Key Differences Between Amazon and Google

This section is mostly theoretical. Amazon is a fundamentally different search engine than Google so my thinking necessarily evolved when I made the switch from optimizing websites in Google to optimizing products in Amazon.

Conversion vs. user satisfaction

Google built a search engine so they could sell ads. Amazon built a search engine so they can sell products. That creates a basic difference in how each measures success. Google is successful when you find your answer quickly because you will return, perform more searches, and click on ads. Amazon is successful when you to find a great product at a great price and buy it because you will return and buy more products. Google’s search success metrics will revolve around dwell time, click-through-rate, search refinement rate, etc. Amazon can measure success by revenue or gross margin per search. If Amazon can sell more products by rearranging their search results, they will do that.

Because the two search engines measure success differently, the metrics you analyze to predict rankings success change. When optimizing for Google you focus on improving user engagement metrics and building external trust factors, because those factors tell Google that the users it sends to your website will be happy. Happy users equals more money for Google. When optimizing for Amazon, focus on improving conversion rates. More conversions equals more money for Amazon.

Structured vs. unstructured data

While Google has been encouraging site owners to add more structured data to their websites, the data in Amazon’s index is already completely structured. Here’s a screenshot of the page where a seller enters data about a product. Every field has a name, a definition, and sometimes a defined list of valid values:

Now compare that to the basic way to build a web page.

Site owners have a blank slate where they can express… anything. In Amazon, you need to give Amazon exactly what they want in the format they specify. Because Amazon has already determined the type of information you can give them about your product, spend time providing accurate and complete product data.

On-page vs. on-page + off-page

With Google you spend a lot of your time optimizing your off-page signals. You build links, manage a social media presence, and encourage brand mentions because Google is measuring those signals to calculate the popularity and trust of your website. While these activities may have secondary effects on a products ranking in Amazon (greater brand awareness creates more branded search leading to a higher sales rank and conversion rate leading Amazon to rank you higher), building a link to your blue widget page on Amazon will not directly improve its ranking for the search term “blue widget.”

On Amazon, that leaves you with optimizing for conversions, which can be frustrating due to the sparse user behavior data. Here’s all the data you get about user behavior on your listings.

Compared to an analytics package like Google Analytics, it’s nothing. You can’t even view a product’s page views or conversion rate over time without downloading one report per day, week, or month and combining it in Excel.

Compelling vs. unique content

When I first started working with online marketplaces I thought, “We need to write a unique description and bullet points for every marketplace we sell on or else Google won’t rank us well.” I didn’t realize that the bulk of our search traffic in Amazon comes from internal site search and Amazon doesn’t care if your listing has the exact same title, bullets, description, and images as another website. They just care if it converts their searchers into purchasers.

(By the way, I’m not saying that compelling and unique content are mutually exclusive.)

Results Page Mechanics

To properly interpret what a ranking means, you should understand the anatomy of a search results page. Like Google, Amazon’s search results pages can have several different looks depending on what type of search you entered.

Anatomy of the results page

Amazon has two formats for their results: a list view for searches in all departments and a gallery view when you search within a specific department or category. The list view contains 15 results per page (sometimes there are 16 results on the first page). The gallery results have 24 results per page.

List view

Gallery view

Some other important elements of the results page are the filter fields in the left sidebar. When a user clicks on a filter, they will see a subset of the original search results. This is one reason why it is so important to complete as many fields as possible when you create a product in Amazon. For instance, Amazon will not know that a blue widget is blue if you don’t fill out the color map field, which means it will be excluded when a user filters to only show blue products.

Finally, there are sponsored products. These are pay-per-click results that show up on the bottom of a search results page. In my experience, if I would like my ad to appear for a specific query, I must include all of the words in the query somewhere in my title or bullet points.

Query string parameters

Amazon builds the URL of a search results page with query parameters much like Google. There are many parameters that might be used but I will review the three most useful. To learn more about the parameters Amazon uses, play around with the filter fields available in the left sidebar and watch how the URL of the search page changes.

field-keywords: Your query in the search bar

node: A numeric string identifying a node in Amazon’s taxonomy (category tree). To determine which number corresponds to which category, navigate to the category on Amazon and find the number Amazon uses in the node parameter in the URL. For instance, the node ID for the Electronics category is 172282. The node IDs are also available in Amazon’s Browse Tree Guides (Seller Central login required).

field-brandtextbin: This represents the brand field. This field is very useful if you want to track how well your product ranks among other products from the same brand. It will not return results if it is the only parameter you include in the search URL. To ensure that you see all products from that brand, include the brand name in the field-keywords parameter.

Here’s how it looks when you use each of these three parameters in one search:

http://www.amazon.com/s?field-keywords=blue widgets&node=172282&field-brandtextbin=pioneer

That URL will search for blue widgets in the electronics category where the brand is pioneer.

Ranking Factors

First, let’s see what Amazon themselves say about how they rank products. This is an excerpt from a help file in Seller Central titled, Using Search and Browse (Seller Central login required).

“Search is the primary way that customers use to locate products on Amazon.com. Customers search by entering keywords, which are matched against the search terms you enter for a product. Well-chosen search terms increase a product’s visibility and sales. The number of views for a product detail page can increase significantly by adding just one additional search term – if it’s a relevant and compelling term.

“Factors such as price, availability, selection, and sales history help determine where your product appears in a customer’s search results. In general, better-selling products tend to be towards the beginning of the results list. As your sales of a product increase, so does your placement.”

Several statements in those paragraphs are very illuminating. First, search is the “primary” way customers find products. I interpret this to mean that most of the time, Amazon users will perform a search before purchasing. If you want your products to be found on Amazon you must think about search. Second, Amazon mentions some of the data they use to rank products. Specifically they mention the search terms, price, availability (meaning inventory levels), selection (not sure what that means), and sales history.

I will expand on each of these factors and include several more where I have observed an effect on rankings. To further clarify the type of effect each factor has on rankings I separated the factors into two categories: performance factors and relevance factors. A performance factor improves rankings by showing Amazon they will make more money by ranking the product, a relevance factor shows Amazon that a product is relevant to the search of a user.

Performance factors

Performance factors are pretty simple. Amazon wants to rank the product that will generate the most profit for them at the top of each search result. Each of these factors will indicate to Amazon that a product will sell well when ranked well.

Conversion rate

This is a pretty obvious factor to mention, albeit a difficult factor to improve with confidence. Amazon does share units and sessions but does not provide enough data to run A/B tests or even control for specific traffic sources. To find conversion data in Seller Central, navigate to Reports >> Business Reports >> Detail Page Sales and Traffic. Make sure the Unit Session Percentage column is visible. This is simply the number of units ordered divided by the number of sessions your listing received.

Amazon’s Definition of a session is: Sessions are visits to your Amazon.com pages by a user. All activity within a 24-hour period is considered a session.

If your offer is competing with other offers for the same product, be sure to weight your units ordered by your buy box percentage. Otherwise, you product will look like it converts more poorly than it actually does. Amazon will show you all of the sessions a listing received regardless of who was in the buy box but they only show you the number of units ordered from your seller account. If you had 50% of the buy box for a time period you probably received half of the total orders for the listing. Therefore the unit session percentage reported should be half of the unit session percentage observed across all sellers.

Images

Amazon strongly encourages sellers to follow their image guidelines. On their image requirements page they encourage sellers to upload images larger than 1000×1000 pixels (the size required to activate their zoom feature) by saying, “Zoom has proven to enhance sales.”

By including images that meet Amazon’s guidelines, you will ensure that your listings are not suppressed (which kills all sales) and possibly increase conversion rates. As Amazon stated on their Search and Browse page, more sales equals better rankings.

Price

Price often strongly influences conversion rates and units sales. If the price on Amazon compares well to the same product offered on other websites and retail stores, comparison shoppers will be more likely to buy from Amazon and vice versa.

Also consider how your product’s price compares to other products in the same category. My company used to sell a battery-operated vacuum pooper scooper that cost $150. It never ranked very well for searches like “pooper scooper.” I believe this was partly because every other pooper scooper costs between $10 and $20. If customers are used to paying $10 for a pooper scooper it takes a lot of convincing for them to shell out $150. Amazon either observed a low conversion rate and did not rank the product or predicted a low conversion rate and did not rank the product. Either way, the price probably kept our $150 pooper scooper from ranking well.

Relevance factors

Amazon will analyze the following fields to determine if a product is relevant to a search.

Title

The title of a product is one of the most important places to include keywords. Amazon suggests incorporating the following attributes in product titles.

  • Brand and description
  • Product line
  • Material or key ingredient
  • Color
  • Size
  • Quantity

What they do not mention, probably because they want to discourage keyword stuffing, is that you should include an important keyword in the product title. A title is also critical for earning a high click-through-rate and conversion rate by clearly stating what the product is. Since sales factor prominently in ranking, keyword-stuffed titles that discourage users from clicking will ultimately harm your rankings.

As an example of an optimized title, we sell a mineral sunscreen called Brush on Block. In addition to the brand/product name we want to make sure to include the keyword “mineral sunscreen” in the title. It helps users understand what the product is and it’s a valuable keyword. Our title is Brush On Block Broad Spectrum SPF 30 Mineral Powder Sunscreen.

Brand

The brand field in Amazon appears here on the product page. It will always link to a search result of more products from the same brand. When you list products, always include the proper brand name. It is very common for consumers to search for products based on their brand name, so be sure to include the correct one. If a product has multiple brand names you could use, use Google’s Keyword Planner to see which brand is searched most frequently.

Bullets and description

Anecdotally, the bullet points seem to be more influential on search rankings than the description. One of our clients has a line of products with a celebrity’s name attached to the product. After doing some keyword research in Google, we found that there were several popular ways to search for the celebrity’s name. There were many books written by the celebrity already ranking for these versions of their name. The day after including the celebrity’s name in the bullet points, our products began to appear on the second and third pages of results.

Search terms

If you are used to keywords for SEO and PPC, it’s easy to use the “search terms” fields on Amazon incorrectly. I’ve even seen articles written by industry experts that provide sub-optimal advice for using these fields. If you have a Seller Central account, the Search and Browse help page is worth reading. I’ll summarize what they say as well as give some examples share a few main points here. First, I’ll spell out the guidelines:

  • There are five fields that accept 50 characters each.
  • You do not need to repeat any words
  • Commas will be ignored
  • Quotation marks will unnecessarily limit your keyword
  • Including multiple variations of the same word is unnecessary
  • Including common misspellings is unnecessary
  • Order of the search terms may matter
  • Do include synonyms or spelling variations (e.g. include sun screen and sunscreen)

When I first started filling out search terms fields in Amazon, I would have done something like this:

Search Term 1 Sunblock
Search Term 2 Sunscreen
Search Term 3 Sun block
Search Term 4 Sun screen
Search Term 5 Mineral Sunscreen
Applying all the rules above, you actually want your fields to look like this.
Search Term 1 brush on block mineral powder sunscreen sunblock
Search Term 2 sun screen protection spf 30 suntan lotion tan kid
Search Term 3 baby spray face child family natural skin sport
Search Term 4 cream boat women men infant spf30 travel small
Search Term 5 solar defense uv facial sensitive babies

No word is repeated, there are no variations of the same word and I’ve used as many characters as possible for maximum exposure.

The search terms fields do influence ranking on Amazon. As an easy-to-prove example, one of our clients identified a common spelling of their brand name with only a single, outdated result. After adding the term to the Search Terms fields in all of their products, all of their products began to appear for that spelling of their brand name within an hour.

Seller name

I have not seen anything published about Amazon using seller names to build their search results, but I have seen a few situations that lead me to believe your seller name is used in Amazon’s organic search algorithm:

Situation 1: We sell a line of cell phone cases. There are over 17 million cell phone cases listed on Amazon. I have no idea where ours rank, but it’s nowhere near the top. However, if I search for “cell phone case” + our seller name, I can see all of our cell phone cases close to the top of the results.

Situation 2: We sell some workout DVDs. These do not rank very well amongst the 242,000 results for “workout,” but when you add part of our seller name (it’s made up of two words), you can see our workout DVDs on the first page of results.

Both situations demonstrate that Amazon is using your seller name as part of the content they index for search results. It may not be a good idea to change your seller name to optimize for your top keyword, but it will be used in search.

Track Your Progress

Now that you know what factors Amazon is using to rank products it’s time to get to work. Remember to track your progress to see if your changes help or hurt your rankings. There is a relative lack of testing between ranking factors on Amazon compared to Google, so act with care. To track rankings, I made a simple Google Spreadsheet (which can be improved upon, I’m sure).

UPDATE: The Google Spreadsheet was acting up, so I created this Excel version for you to download.

Here is the process I use to record rankings every day.

Setup

  1. Make a copy of the spreadsheet so you can edit it.
  2. Enter the ASINs you would like to track in column E (Don’t edit columns F through H).
  3. Keep a list of Keywords you want to track in column J.

Daily process

  1. Copy the first keyword from Column J and paste it into cell B1.
  2. Wait a few seconds for the spreadsheet to load the top 240 ASINs returned by Amazon.
  3. When Columns A through C are full of data, you should see your rankings in column G.
  4. In the example spreadsheet, I would highlight columns E3:H5 and copy
  5. To store the data, go to the Historical Data tab and paste values at the end of the table.
    • To paste values only, right click the first empty cell in column A, mouse over “Paste special,” and click on “Paste values only.”
    • If you do a normal paste, it will preserve the formulas and look wrong.

To analyze rankings changes over time, I download the file as an .xlsx, open it with Excel, highlight the table of data on the Historical Data tab and insert a pivot chart.

(Disclaimer: This spreadsheet uses scraped data from Amazon. Amazon may change the structure of their pages at any time, and if it does, this sheet may stop working. If this data is important to your business, make sure you understand the xpath used to scrape Amazon and are ready to fix it if Amazon changes its HTML.)

Other Visibility Systems

Here are a few other factors that may help increase your sales or ranking. I do not have any explicit statements, studies, or experience proving a relationship but they are worth trying out. I would encourage you to study the effect of these fields on rankings and sales.

Filter fields

Next to every search result is a list of attributes that allow users to filter their results. For your top keywords, make sure your product has a value filled out for each category of fields to ensure your product is still visible when users filter by color, size, or any other attribute. The Category, Eligible for Free Shipping, Brand, Avg. Customer Review, and Condition fields are visible on most search results.

Reviews

More reviews and better ratings might lead to better sales. Most products that rank well for broad searches have many reviews but it is difficult to tell if the good reviews lead to more sales or if high sales volume leads to more reviews. You can encourage more reviews by emailing your purchasers and asking them to leave a review.

Sales rank

Amazon maintains best sellers lists and reports a listings best sellers ranking for relevant categories on the listing page. This can be a quick way to see how your products’ sales histories compare to similar products.

Parent-child products

What if you could combine the sales histories of several similar products into one? Well, maybe you can. If you sell a product with several size or color options you can list them as a variation and combine them into one listing. While Amazon will combine the reviews from all listings onto the new listing, it’s unclear whether this leads to better rankings by combining the sales histories of the different options.

For instructions on configuring your products as parent/child listings, see the Creating Parent/Child Variation Relationships page in Seller Central (login required).

MPN

We have observed several products with significant search volume for the manufacturer part number (MPN). Check with the manufacturer to make sure you have the correct MPN on your listing.

In Closing

There are so many topics related to selling on Amazon that I cannot possibly cover them in one blog post. I would like to mention that even with the best organic optimization you can still have poor sales for a product if you don’t understand how to win the Amazon buy box, maintain a good seller rating, or keep inventory in stock. There are a lot of good resources that already exist on these topics.

I hope that, as a community, we can continue to study and educate ourselves on Amazon’s algorithm. It’s a more important search engine than Google in the world of eCommerce, and it continues to gain market share in the US. Ignore it at your own risk.

Did I miss anything? Do you have any questions about your company’s products? Ask away in the comments or get in touch with us at DNA Response.


Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

Continue reading →

Announcing MozBar 3.0: the Free, Completely Redesigned SEO Toolbar

Posted by jon.white

Today we are thrilled to announce version 3 of the MozBar browser extension. The SEO toolbar is now available for Chrome users. Expect the Firefox version to be available in a few weeks.

What is the MozBar?

The MozBar is a free browser extension that provides on-page access to Moz’s link metrics and site analysis tools. Over the years it has gained a very popular following and saved a ton of time for SEO’s and Inbound marketers alike. Whilst there are certain features that are only available to Pro subscribers, we try to keep as much as possible free. We think this is the TAGFEE thing to do, plus it really helps people as possible to get acquainted with our brand and our tools.

The MozBar, since its inception in 2008, solves three main problems for its users:

  1. SERP analysis
  2. Site/competitor research
  3. Link profile analysis

Here’s how those features work in version 3!

SERP analysis

As you search Google, Yahoo or Bing, the MozBar instantly shows you valuable statistics about each of the results you see. This new version of the MozBar makes deconstructing SERPs faster than ever.

Create search profiles for different engines and locations

If you are working in local search, the MozBar allows you to create search profiles for specific regions and cities. This allows you to easily switch between a search for “pizza” in Chicago and Seattle without changing your search query.


Export the SERP to a CSV

As you search, easily export into a CSV key data about each SERP including:

  • URL
  • Page Title
  • Description
  • Detailed link metrics

See Moz authority and search metrics next to each search result

You’ll get an overview of the most important statistics for each result on a SERP without even having to click through to those results.


Site/competitor research

This is another area where we’ve added a significant number of improvements, from on-page analysis to new structured data and markup detection.

See Moz authority and link metrics

For every URL you visit, the MozBar instantly shows you the link metrics at the top of the page, including MozRank, Domain Authority, subdomain metrics and more.


Highlight followed, nofollowed, external, and internal links

Easily spot which links are followed or nofollowed without having to dig through source code.


See important page attributes and elements on the page

The page analysis tools make up some of the strongest features of the MozBar. They allow you to perform an instant on-page audit of any URL you visit. With just a couple of clicks, instantly see important on page factors like title tags, meta description, canonical tags, page load time, HTTP status and more.


Link profile analysis

Detailed information about a page’s inbound links, including quick comparisons to the site’s domain and subdomain, are available at a glance.


What’s new in version 3?

Those of you familiar with the MozBar will notice that version 3 has a new look and design. The redesign is a result of a bunch of customer and design research and has been optimized around the tasks and use cases it is designed to solve. It is also much faster and more reliable. Some exciting new features for v3 include:

See social activity

No more hunting for that social sharing bar on pages you visit: MozBar now includes social statistics from Facebook, Twitter, and Google+ right on the page.


Validate and preview semantic markup

You’ll get an at-a-glance look at any semantic markup present on a page. Want to make sure a Twitter card is properly set up? No need to send a test tweet; just preview it in the MozBar.


View keyword difficulty on the SERP 

One of our most-requested features was the ability to make it easier to check keyword difficulty. Now you can get a keyword’s difficulty on the fly for any search query with a click of the button, right from the search page.

Note: This feature is only available to Pro subscribers.

Pro tip: MozBar obscuring your page? Hit Shift+Ctrl+Alt+M to show / hide the bar!

If you already have the MozBar installed, you don’t need to do anything. The Chrome Store will update to the new version automatically. If you don’t already have it, download it at the link above!


Or maybe you are feeling a bit nostalgic? Check out how the MozBar has evolved over the years! (And what better way to travel back in time than with a carousel and a cool gradient overlay?) 🙂

Looking for the Firefox version?

We are still ironing out some last-minute issues in the Firefox version, and will launch it as soon as it’s ready. For now, don’t worry; you can still use version 2.65.


Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

Continue reading →

A Content Marketer’s Guide to Data Scraping

Posted by MatthewBarby

As digital marketers, big data should be what we use to inform a lot of the decisions we make. Using intelligence to understand what works within your industry is absolutely crucial within content campaigns, but it blows my mind to know that so many businesses aren’t focusing on it.

One reason I often hear from businesses is that they don’t have the budget to invest in complex and expensive tools that can feed in reams of data to them. That said, you don’t always need to invest in expensive tools to gather valuable intelligence — this is where data scraping comes in.

Just so you understand, here’s a very brief overview of what data scraping is from Wikipedia:

Data scraping is a technique in which a computer program extracts data from human-readable output coming from another program.”

Essentially, it involves crawling through a web page and gathering nuggets of information that you can use for your analysis. For example, you could search through a site like Search Engine Land and scrape the author names of each of the posts that have been published, and then you could correlate this to social share data to find who the top performing authors are on that website.

Hopefully, you can start to see how this data can be valuable. What’s more, it doesn’t require any coding knowledge — if you’re able to follow my simple instructions, you can start gathering information that will inform your content campaigns. I’ve recently used this research to help me get a post published on the front page of BuzzFeed, getting viewed over 100,000 times and channeling a huge amount of traffic through to my blog.

Disclaimer: One thing that I really need to stress before you read on is the fact that scraping a website may breach its terms of service. You should ensure that this isn’t the case before carrying out any scraping activities. For example, Twitter completely prohibits the scraping of information on their site. This is from their Terms of Service:

“crawling the Services is permissible if done in accordance with the provisions of the robots.txt file, however, scraping the Services without the prior consent of Twitter is expressly prohibited

Google similarly forbids the scraping of content from their web properties:

Google’s Terms of Service do not allow the sending of automated queries of any sort to our system without express permission in advance from Google.

So be careful, kids.

Content analysis

Mastering the basics of data scraping will open up a whole new world of possibilities for content analysis. I’d advise any content marketer (or at least a member of their team) to get clued up on this.

Before I get started on the specific examples, you’ll need to ensure that you have Microsoft Excel on your computer (everyone should have Excel!) and also the SEO Tools plugin for Excel (free download here). I put together a full tutorial on using the SEO tools plugin that you may also be interested in.

Alongside this, you’ll want a web crawling tool like Screaming Frog’s SEO Spider or Xenu Link Sleuth (both have free options). Once you’ve got these set up, you’ll be able to do everything that I outline below.

So here are some ways in which you can use scraping to analyse content and how this can be applied into your content marketing campaigns:

1. Finding the different authors of a blog

Analysing big publications and blogs to find who the influential authors are can give you some really valuable data. Once you have a list of all the authors on a blog, you can find out which of those have created content that has performed well on social media, had a lot of engagement within the comments and also gather extra stats around their social following, etc.

I use this information on a daily basis to build relationships with influential writers and get my content placed on top tier websites. Here’s how you can do it:

Step 1: Gather a list of the URLs from the domain you’re analysing using Screaming Frog’s SEO Spider. Simply add the root domain into Screaming Frog’s interface and hit start (if you haven’t used this tool before, you can check out my tutorial here).

Once the tool has finished gathering all the URLs (this can take a little while for big websites), simply export them all to an Excel spreadsheet.

Step 2: Open up Google Chrome and navigate to one of the article pages of the domain you’re analysing and find where they mention the author’s name (this is usually within an author bio section or underneath the post title). Once you’ve found this, right-click their name and select inspect element (this will bring up the Chrome developer console).

Within the developer console, the line of code associated to the author’s name that you selected will be highlighted (see the below image). All you need to do now is right-click on the highlighted line of code and press Copy XPath.

For the Search Engine Land website, the following code would be copied:

//*[@id="leftCol"]/div[2]/p/span/a

This may not make any sense to you at this stage, but bear with me and you’ll see how it works.

Step 3: Go back to your spreadsheet of URLs and get rid of all the extra information that Screaming Frog gives you, leaving just the list of raw URLs – add these to the first column (column A) of your worksheet.

Step 4: In cell B2, add the following formula:

=XPathOnUrl(A2,"//*[@id='leftCol']/div[2]/p/span/a")

Just to break this formula down for you, the function XPathOnUrl allows you to use the XPath code directly within (this is with the SEO Tools plugin installed; it won’t work without this). The first element of the function specifies which URL we are going to scrape. In this instance I’ve selected cell A2, which contains a URL from the crawl I did within Screaming Frog (alternatively, you could just type the URL, making sure that you wrap it within quotation marks).

Finally, the last part of the function is our XPath code that we gathered. One thing to note is that you have to remove the quotation marks from the code and replace them with apostrophes. In this example, I’m referring to the “leftCol” section, which I’ve changed to ‘leftCol’ — if you don’t do this, Excel won’t read the formula correctly.

Once you press enter, there may be a couple of seconds delay whilst the SEO Tools plugin crawls the page, then it will return a result. It’s worth mentioning that within the example I’ve given above, we’re looking for author names on article pages, so if I try to run this on a URL that isn’t an article (e.g. the homepage) I will get an error.

For those interested, the XPath code itself works by starting at the top of the code of the URL specified and following the instructions outlined to find on-page elements and return results. So, for the following code:

//*[@id='leftCol']/div[2]/p/span/a

We’re telling it to look for any element (//*) that has an id of leftCol (@id=’leftCol’) and then go down to the second div tag after this (div[2]), followed by a p tag, a span tag and finally, an a tag (/p/span/a). The result returned should be the text within this a tag.

Don’t worry if you don’t understand this, but if you do, it will help you to create your own XPath. For example, if you wanted to grab the output of an a tag that has rel=author attached to it (another great way of finding page authors), then you could use some XPath that looked a little something like this:

//a[@rel='author']

As a full formula within Excel it would look something like this:

=XPathOnUrl(A2,"//a[@rel='author']")

Once you’ve created the formula, you can drag it down and apply it to a large number of URLs all at once. This is a huge time-saver as you’d have to manually go through each website and copy/paste each author to get the same results without scraping – I don’t need to explain how long this would take.

Now that I’ve explained the basics, I’ll show you some other ways in which scraping can be used…

2. Finding extra details around page authors

So, we’ve found a list of author names, which is great, but to really get some more insight into the authors we will need more data. Again, this can often be scraped from the website you’re analysing.

Most blogs/publications that list the names of the article author will actually have individual author pages. Again, using Search Engine Land as an example, if you click my name at the top of this post you will be taken to a page that has more details on me, including my Twitter profile, Google+ profile and LinkedIn profile. This is the kind of data that I’d want to gather because it gives me a point of contact for the author I’m looking to get in touch with.

Here’s how you can do it.

Step 1: First we need to get the author profile URLs so that we can scrape the extra details off of them. To do this, you can use the same approach to find the author’s name, with just a little addition to the formula:

=XPathOnUrl(A2,"//a[@rel='author']", <strong>"href"</strong>)

The addition of the “href” part of the formula will extract the output of the href attribute of the atag. In Lehman terms, it will find the hyperlink attached to the author name and return that URL as a result.

Step 2: Now that we have the author profile page URLs, you can go on and gather the social media profiles. Instead of scraping the article URLs, we’ll be using the profile URLs.

So, like last time, we need to find the XPath code to gather the Twitter, Google+ and LinkedIn links. To do this, open up Google Chrome and navigate to one of the author profile pages, right-click on the Twitter link and select Inspect Element.

Once you’ve done this, hover over the highlighted line of code within Chrome’s developer tools, right-click and select Copy XPath.

Step 3: Finally, open up your Excel spreadsheet and add in the following formula (using the XPath that you’ve copied over):

=XPathOnUrl(C2,"//*[@id='leftCol']/div[2]/p/a[2]", "href")

Remember that this is the code for scraping Search Engine Land, so if you’re doing this on a different website, it will almost certainly be different. One important thing to highlight here is that I’ve selected cell C2 here, which contains the URL of the author profile page and not just the article page. As well as this, you’ll notice that I’ve included “href” at the end because we want the actual Twitter profile URL and not just the words ‘Twitter’.

You can now repeat this same process to get the Google+ and LinkedIn profile URLs and add it to your spreadsheet. Hopefully you’re starting to see the value in this, and how it can be used to gather a lot of intelligence that can be used for all kinds of online activity, not least your SEO and social media campaigns.

3. Gathering the follower counts across social networks

Now that we have the author’s social media accounts, it makes sense to get their follower counts so that they can be ranked based on influence within the spreadsheet.

Here are the final XPath formulae that you can plug straight into Excel for each network to get their follower counts. All you’ll need to do is replace the text INSERT SOCIAL PROFILE URL with the cell reference to the Google+/LinkedIn URL:

Google+:

=XPathOnUrl(<strong>INSERTGOOGLEPROFILEURL</strong>,"//span[@class='BOfSxb']")

LinkedIn:

=XPathOnUrl(<strong>INSERTLINKEDINURL</strong>,"//dd[@class='overview-connections']/p/strong")

4. Scraping page titles

Once you’ve got a list of URLs, you’re going to want to get an idea of what the content is actually about. Using this quick bit of XPath against any URL will display the title of the page:

=XPathOnUrl(A2,"//title")

To be fair, if you’re using the SEO Tools plugin for Excel then you can just use the built-in feature to scrape page titles, but it’s always handy to know how to do it manually!

A nice extra touch for analysis is to look at the number of words used within the page titles. To do this, use the following formula:

=CountWords(A2)

From this you can get an understanding of what the optimum title length of a post within a website is. This is really handy if you’re pitching an article to a specific publication. If you make the post the best possible fit for the site and back up your decisions with historical data, you stand a much better chance of success.

Taking this a step further, you can gather the social shares for each URL using the following functions:

Twitter:

=TwitterCount(<strong>INSERTURLHERE</strong>)

Facebook:

=FacebookLikes(<strong>INSERTURLHERE</strong>)

Google+:

=GooglePlusCount(<strong>INSERTURLHERE</strong>)

Note: You can also use a tool like URL Profiler to pull in this data, which is much better for large data sets. The tool also helps you to gather large chunks of data from other social networks, link data sources like Ahrefs, Majestic SEO and Moz, which is awesome.

If you want to get even more social stats then you can use the SharedCount API, and this is how you go about doing it…

Firstly, create a new column in your Excel spreadsheet and add the following formula (where A2 is the URL of the webpage you want to gather social stats for):

=CONCATENATE("http://api.sharedcount.com/?url=",A2)

You should now have a cell that contains your webpage URL prefixed with the SharedCount API URL. This is what we will use to gather social stats. Now here’s the Excel formula to use for each network (where B2 is the cell that contaiins the formula above):

StumbleUpon:

=JsonPathOnUrl(B2,"StumbleUpon")

Reddit:

=JsonPathOnUrl(B2,"Reddit")

Delicious:

=JsonPathOnUrl(B2,"Delicious")

Digg:

=JsonPathOnUrl(B2,"Diggs")

Pinterest:

=JsonPathOnUrl(B2,"Pinterest")

LinkedIn:

=JsonPathOnUrl(B2,"Linkedin")

Facebook Shares:

=JsonPathOnUrl(B2,"Facebook.share_count")

Facebook Comments:

=JsonPathOnUrl(B2,"Facebook.comment_count")

Once you have this data, you can start looking much deeper into the elements of a successful post. Here’s an example of a chart that I created around a large sample of articles that I analysed within Upworthy.com.

The chart looks at the average number of social shares that an article on Upworthy receives vs the number of words within its title. This is invaluable data that can be used across a whole host of different on-page elements to get the perfect article template for the site you’re pitching to.

See, big data is useful!

5. Date/time the post was published

Along with analysing the details of headlines that are working within a site, you may want to look at the optimal posting times for best results. This is something that I regularly do within my blogs to ensure that I’m getting the best possible return from the time I spend writing.

Every site is different, which makes it very difficult for an automated, one-size-fits-all tool to gather this information. Some sites will have this data within the <head> section of their webpages, but others will display it directly under the article headline. Again, Search Engine Land is a perfect example of a website doing this…

So here’s how you can scrape this information from the articles on Search Engine Land:

=XPathOnUrl(<strong>INSERTARTICLEURL</strong>,"//*[@class='dateline']/text()")

Now you’ve got the date and time of the post. You may want to trim this down and reformat it for your data analysis, but you’ve got it all in Excel so that should be pretty easy.

Extra reading

Data scraping is seriously powerful, and once you’ve had a bit of a play around with it you’ll also realise that it’s not that complicated. The examples that I’ve given are just a starting point but once you get your creative head on, you’ll soon start to see the opportunities that arise from this intelligence.

Here’s some extra reading that you might find useful:

TL;DR

  • Start using actual data to inform your content campaigns instead of going on your gut feeling.
  • Gather intelligence around specific domains you want to target for content placement and create the perfect post for their audience.
  • Get clued up on XPath and JSON through using the SEO Tools plugin for Excel.
  • Spend more time analysing what content will get you results as opposed to what sites will give you links!
  • Check the website’s ToS before scraping.

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

Continue reading →

The Illustrated SEO Competitive Analysis Workflow

Posted by Aleyda

One of the most important activities for any SEO process is the initial competitive analysis. This process should correctly identify your SEO targets and provide fundamental input to establish your overall strategy.

Depending on the type, industry, and scope of the SEO process, this analysis can become quite complex, as there are many factors to take into consideration—more now than ever before.

In order to facilitate this process (and make it easy to replicate, control, and document), I’ve created a step-by-step workflow with the different activities and factors to take into consideration, including identifying SEO competitors, gathering the potential keywords to target, assessing their level of difficulty, and selecting them based on defined criteria:

If you prefer, you can also grab a higher resolution version of the workflow from here.

The four analysis phases

As you can see, the SEO analysis workflow is divided into four phases:

1. Identify your potential SEO competitors

This initial phase is especially helpful if you’re starting with an SEO process for a new client or industry that you don’t know anything about, and you need to start from scratch to identify all of the potentially relevant competitors.

It’s important to note that these are not necessarily limited to companies or websites that offer the same type of content, services, or products that you do, but can be any website that competes with you in the search results for your target keywords.

2. Validate your SEO competitors

Once you have the potential competitors that you have gathered from different relevant sources it’s time to validate them, by analyzing and filtering which of those are really already ranking, and to which degree, for the same keywords that you’re targeting.

Additionally, at this stage you’ll also expand your list of potential target keywords by performing keyword research. This should use sources beyond the ones that you had already identified coming from your competitors and your current organic search data—sources for which your competitors or yourself are still not ranking, that might represent new opportunities.

3. Compare with your SEO competitors

Now that you have your SEO competitors and potential target keywords, you can gather, list, and compare your site to your competitors, using all of the relevant data to select and prioritize those keywords. This will likely include keyword relevance, current rankings, search volume, ranked pages, as well as domains’ link popularity, content optimization, and page results characteristics, among others.

4. Select your target keywords

It’s finally time to analyze the previously gathered data for your own site and your competitors, using the specified criteria to select the best keyword to target for your own situation in the short-, mid-, and long-term during your SEO process: Those with the highest relevance, search volume, and profitability. The best starting point is in rankings where you are competitive from a popularity and content standpoint.

Tools & data sources

The data sources and tools—besides the traditional ones from search engines, like their keyword or webmaster tools—that can help you to implement the process (some of them mentioned in the workflow) are:

Hopefully with these resources you’ll be able to develop more and better SEO competitive analysis!


What other aspects do you take into consideration and which other tools do you? I look forward to hear about them in the comments.


Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

Continue reading →