Archives for 

seo

Why All SEOs Should Unblock JavaScript & CSS… And Why Google Cares

Posted by jenstar

If you’re a webmaster, you probably received one of those infamous “Googlebot cannot access CSS and JS files on example.com” warning letters that Google sent out to seemingly every SEO and webmaster. This was a brand new alert from Google, although we have been hearing from the search engine about the need to ensure all resources are unblocked—including both JavaScript and CSS.

There was definite confusion around these letters, supported by some of the reporting in Google Search Console. Here’s what you need to know about Google’s desire to see these resources unblocked and how you can easily unblock them to take advantage of the associated ranking boosts.

Why does Google care?

One of the biggest complaints about the warning emails lay in the fact that many felt there was no reason for Google to see these files. This was especially true because it was flagging files that, traditionally, webmasters blocked—such as files within the WordPress admin area and Wordpress plugin folders.

Here’s the letter in question that many received from Google. It definitely raised plenty of questions and concerns:

Of course, whenever Google does anything that could devalue rankings, the SEO industry tends to freak out. And the confusing message in the warning didn’t help the situation.

Why Google needs it

Google needs to render these files for a couple of key reasons. The most visible and well known is the mobile-friendly algorithm. Google needs to be able to render the page completely, including the JavaScript and CSS, to ensure that the page is mobile-friendly and to apply both the mobile-friendly tag in the search results and the associated ranking boost for mobile search results. Unblocking these resources was one of the things that Google was publicly recommending to webmasters to get the mobile-friendly boost for those pages.

However, there are other parts of the algorithm that rely on using it, as well. The page layout algorithm, the algorithm that looks at where content is placed on the page in relation to the advertisements, is one such example. If Google determines a webpage is mostly ads above the fold, with the actual content below the fold, it can devalue the rankings for those pages. But with the wizardry of CSS, webmasters can easily make it appear that the content is front and center, while the ads are the most visible part of the page above the fold.

And while it’s an old school trick and not very effective, people still use CSS and JavaScript in order to hide things like keyword stuffing and links—including, in the case of a hacked site, to hide it from the actual website owner. Googlebot crawling the CSS and JavaScript can determine if it is being used spammily.

Google also has hundreds of other signals in their search algo, and it is very likely that a few of those use data garnered from CSS and JavaScript in some fashion as well. And as Google changes things, there is always the possibility that Google will use it for future signals, as well.

Why now?

While many SEOs had their first introduction to the perils of blocking JavaScript and CSS when they received the email from Google, Matt Cutts was actually talking about it three-and-a-half years ago in a Google Webmaster Help video.


Then, last year, Google made a significant change to their webmaster guidelines by adding it to their technical guidelines:

Disallowing crawling of Javascript or CSS files in your site’s robots.txt directly harms how well our algorithms render and index your content and can result in suboptimal rankings.

It still got very little attention at the time, especially since most people believed they weren’t blocking anything.

However, one major issue was that some popular SEO Wordpress plugins were blocking some JavaScript and CSS. Since most Wordpress users weren’t aware this was happening, it came as a surprise to learn that they were, in fact, blocking resources.

It also began showing up in a new “Blocked Resources” section of Google Search Console in the month preceding the mobile-friendly algo launch.

How many sites were affected?

In usual Google fashion, they didn’t give specific numbers about how many webmasters received these blocked resources warnings. But Gary Illyes from Google did confirm that they were sent out to 18.7% of those that were sent out for the mobile-friendly warnings earlier this year:

@jenstar about 18.7% of that sent for mobile issues a few months back

— Gary Illyes (@methode) July 29, 2015

Finding blocked resources

The email that Google sent to webmasters alerting them to the issue of blocked CSS and JavaScript was confusing. It left many webmasters unsure of what exactly was being blocked and what was blocking it, particularly because they were receiving warnings for JavaScript and CSS hosted on other third-party sites.

If you received one of the warning letters, the suggestion for how to find blocked resources was to use the Fetch tool in Google Search Console. While this might be fine for checking the homepage, for sites with more than a handful of pages, this can get tedious quite quickly. Luckily, there’s an easier way than Google’s suggested method.

There’s a full walkthrough here, but for those familiar with Google Search Console, you’ll find a section called “Blocked Resources” under the “Google Index” which will tell you what JavaScript and CSS is blocked and what pages they’re found in.

You also should make sure that you check for blocked resources after any major redesign or when launching a new site, as it isn’t entirely clear if Google is still actively sending out these emails to alert webmasters of the problem.

Homepage

There’s been some concern about those who use specialized scripts on internal pages and don’t necessarily want to unblock them for security reasons. John Mueller from Google said that they are looking primarily at the homepage—both desktop and mobile—to see what JavaScript and CSS are blocked.

So at least for now, while it is certainly a best practice to unblock CSS and JavaScript from all pages, at the very least you want to make it a priority for the homepage, ensuring nothing on that page is blocked. After that, you can work your way through other pages, paying special attention to pages that have unique JavaScript or CSS.

Indexing of Javascript & CSS

Another reason many sites give for not wanting to unblock their CSS and JavaScript is because they don’t want them to be indexed by Google. But neither of those files are file types that Google will index, according to their long list of supported file types for indexation.

All variations

It is also worth remembering to check both the www and the non-www for blocked resources in Google Search Console. This is something that is often overlooked by those webmasters that only to tend to look at the version they prefer to use for the site.

Also, because the blocked resources data shown in Search Console is based on when Googlebot last crawled each page, you could find additional blocked resources when checking them both. This is especially true for for sites that may be older or not updated as frequently, and not crawled daily (like a more popular site is).

Likewise, if you have both a mobile version and a desktop version, you’ll want to ensure that both are not blocking any resources. It’s especially important for the mobile version, since it impacts whether each page gets the mobile-friendly tag and ranking boost in the mobile search results.

And if you serve different pages based on language and location, you’ll want to check each of those as well. Don’t just check the “main” version and assume it’s all good across the entire site. It’s not uncommon to discover surprises in other variations of the same site. At the very least, check the homepage for each language and location.

Wordpress and blocking Javascript & CSS

If you use one of the “SEO for Wordpress”-type plugins for a Wordpress-based site, chances are you’re blocking Javascript and CSS due to that plugin. It used to be one of the “out-of-the-box” default settings for some to block everything in the /wp-admin/ folder.

When the mobile-friendly algo came into play, because those admin pages were not being individually indexed, the majority of Wordpress users left that robots block intact. But this new Google warning does require all Wordpress-related JavaScript and CSS be unblocked, and Google will show it as an error if you block the JavaScript and CSS.

Yoast, creator of the popular Yoast SEO plugin (formerly Wordpress SEO), also recommends unblocking all the JavaScript and CSS in Wordpress, including the /wp-admin/ folder.

Third-party resources

One of the ironies of this was that Google was flagging third-party JavaScript, meaning JavaScript hosted on a third-party site that was called from each webpage. And yes, this includes Google’s own Google AdSense JavaScript.

Initially, Google suggested that website owners contact those third-party sites to ask them to unblock the JavaScript being used, so that Googlebot could crawl it. However, not many webmasters were doing this; they felt it wasn’t their job, especially when they had no control over what a third-party sites blocks from crawling.

Google later said that they were not concerned about third-party resources because of that lack of control webmasters have. So while it might come up on the blocked resources list, they are truly looking for URLs for both JavaScript and CSS that the website owner can control through their own robots.txt.

John Mueller revealed more recently that they were planning to reach out to some of the more frequently cited third-party sites in order to see if they could unblock the JavaScript. While we don’t know which sites they intend to contact, it was something they planned to do; I suspect they’ll successfully see some of them unblocked. Again, while this isn’t so much a webmaster problem, it’ll be nice to have some of those sites no longer flagged in the reports.

How to unblock your JavaScript and CSS

For most users, it’s just a case of checking the robots.txt and ensuring you’re allowing all JavaScript and CSS files to be crawled. For Yoast SEO users, you can edit your robots.txt file directly in the admin area of Wordpress.

Gary Illyes from Google also shared some detailed robots.txt changes on Stack Overflow. You can add these directives to your robots.txt file in order to allow Googlebot to crawl all Javascript and CSS.

To be doubly sure you’re unblocking all JavaScript and CSS, you can add the following to your robots.txt file, provided you don’t have any directories being blocked in it already:

User-Agent: Googlebot
Allow: .js
Allow: .css
If you have a more specialized robots.txt file, where you’re blocking entire directories, it can be a bit more complicated.

In these cases, you also need to allow the .js and.css for each of the directories you have blocked.

For example:

User-Agent: Googlebot
Disallow: /deep/
Allow: /deep/*.js
Allow: /deep/*.css

Repeat this for each directory you are blocking in robots.txt.

This allows Googlebot to crawl those files, while disallowing other crawlers (if you’ve blocked them). However, the chances are good that the kind of bots you’re most concerned about being allowed to crawl various JavaScript and CSS files aren’t the ones that honor robots.txt files.

You can change the User-Agent to *, which would allow all crawlers to crawl it. Bing does have its own version of the mobile-friendly algo, which requires crawling of JavaScript and CSS, although they haven’t sent out warnings about it.

Bottom line

If you want to rank as well as you possibly can, unblocking JavaScript and CSS is one of the easiest SEO changes you can make to your site. This is especially important for those with a significant amount of mobile traffic, since the mobile ranking algorithm does require they both be unblocked to get that mobile-friendly ranking boost.

Yes, you can continue blocking Google bot from crawling either of them, but your rankings will suffer if you do so. And in a world where every position gained counts, it doesn’t make sense to sacrifice rankings in order to keep those files private.


Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

Continue reading →

Are Your Analytics Telling the Right Story?

Posted by Bill.Sebald

A process can easily become a habit. A habit may not change without awareness or intervention.

Before it becomes a habit, a process should be adjusted to change along with new goals, constant learning, experimentation, and so on.

Considering your time in analytics, are you engaging in a process, or in an outdated habit?

That’s a real question that digital marketing practitioners should ask themselves. Inherently, marketers tend to be buried with work, reusing templates to speed up results. But many agencies lean on those templates a little too much, in my opinion.

Templates should never be written in stone.

If your company is pumping out canned reports, you’re not alone. I do the business development for our company and regularly ask prospects to explain or share the reports they’ve received in the past. Sometimes it’s truly discouraging, outdated, wasteful, and the reason businesses search for new SEO vendors.

Look—I’m all for scalability. It’s a huge help. But some things can’t be scaled and still be successful, especially in today’s SEO climate—or, frankly, marketing in general. Much of what was scalable in SEO prior to 2011 is now penalty-bait. Today’s analytics tools and platforms can slice and dice data faster than anything Ron Popeil ever sold, but the human element will always be necessary if you want your marketing to dominate.

Find the stories to tell

I like to tell stories. I’m real fun in the pub. What I’ve always loved about marketing is the challenge to not only find a story, but have that story change something for the better. I like adding my layer based on real data and experimenting.

Analytics work is all about finding the story. It’s detective work. It’s equal parts Sherlock Holmes, Batman, and Indiana Jones. If you’re lucky, the story jumps out with very little digging. However, it’s more likely you’ll be going on some expeditions. It’s common to start with a hunch or random click through reports, but you need to always be looking for the story.

A great place to start is through client conversations. We schedule at least one monthly call with our clients, where it’s truly a discussion session. We get conversations going to pull intel out of the key stakeholders. Case in point: Recently, we discovered through an open discussion that one of our clients had great success with an earlier email campaign targeted to business owners. There was specific information customers positively responded to, which was helpful in recent content development on their website. It’s amazing what you can learn by asking questions and simply listening to responses.

We should be true consultants, not report monkeys. Dive into the discussions started and enjoy the ride. I guarantee you’ll take note of a few ripe areas to review next time you log into your Google Analytics account.

An impromptu survey says it’s a time issue

Most SEO engagements are designed around a block of purchased hours. Hopefully the client understands they’re not only buying your time to complete SEO tasks, but also your expertise and analysis. If someone on your team were to say, “I don’t have time to do analysis because all my tasks used up their budget this month,” then you really need to question the value of the chosen tasks. Were they picked based on front-loaded analysis, or were they simply tasks pulled out of guesswork?

A few weeks ago I pushed a quick Survey Monkey survey out on Twitter and Linkedin. Thanks to a few retweets, 94 people responded (please consider the following results more directional than scientific—I’m well aware it’s a shallow survey pool). I asked two questions:

  1. If you work in-house or have clients, how often do you log into your clients’ analytics? (Multiple choices ranged from several times a day to a few times a month).
  2. Do you, or do you not, get enough time in Analytics to interpret the data?

The responses:

answers2

While some do make a habit of logging into analytics once or more times a day, more do not. Is it required to check under the hood every day? Personally, I believe it is—but your answer may vary on that one. If something went south overnight, I want to be aware before my client tells me. After all, that’s one of the things I’m paid for. I like the idea of being active—not reactive.

More notable is that most respondents didn’t feel they get enough time in analytics. That should absolutely change.

There was also a field for respondents to elaborate on their selections. There were several comments that jumped out at me:

“In house, day to day tasks and random projects prevent me from taking the deep dives in analytics that I feel are valuable.”

“It’s challenging to keep up with the changes and enhancements made in Google Analytics in particular, amongst other responsibilities and initiatives.”

“Too many things are on my plate for me to spend the time I know I should be spending in Google Analytics.”

“Finding the actionable info in Analytics always takes more time that expected—never enough time to crunch the numbers!”

“I log in to ‘spot check’ things but rarely do I get to delve into the data for long enough to suss out the issues and opportunities presented by the data.”

These results suggest that many marketers are not spending enough time with analytics. And possibly not because they don’t see the value, but simply because they don’t have time. “Either you run the day, or the day runs you (Jim Rohn)” is apropos here—you must make time. You need to get on top of all the people filling your plate. It’s not easy, but it needs to be done.

Get on top of those filling your plate. Kind of like professional crowd surfing.

Helpful resources

Dashboards are fantastic, but I rarely see them set up in analytics platforms. One of the best ways to get a quick glimpse of your key metrics are with dashboards. All good analytics platforms provide the ability to make custom dashboards. Get into work, grab a coffee, fire up the computer, click your dashboard bookmark. (I recommend that order!) Google Analytics, which most of us probably use, provides some decent options with their dashboards, though limited compared to enterprise analytics platforms.

However, this basic dashboard is the minimum you should review in analytics. We’ll get deeper soon.

Building these widgets are quite easy (I recently created a tutorial on my site). There are also websites that provide dashboards you can import into Google Analytics. Dashboard Junkie is a fun one. Here are some others from Econsultancy and Google themselves.

It’s not just analytics platforms that offer dashboards. There are several other vendors in the SEO space that port in analytics data and mesh with their own data—from Moz Analytics to SearchMetrics to Conductor to many, many others.

SEMrush has a unique data set that marketers should routinely review. While your traffic data in analytics will be truer, if you’re targeting pages you may be interested in monitoring keyword rank counts:

Are backlinks a target? Maybe you’d find Cognitive SEO’s dashboard valuable:

cognitive

RankRanger is another SaaS we use. It’s become way more than just our daily rank tracking software. The data you can port in creates excellent snapshots and graphs, and strong dashboards:

rankranger1

It also offers other graphing functionality to make pretty useful views:

While some of the bigger platforms, like SearchMetrics and Conductor, make it easier to get a lot of information within one login, I’m still finding myself logging into several programs to get the most useful data possible. C’est la vie.

Analytics is your vehicle to identifying problems and opportunity

Remember, dashboards are simply the “quick and dirty” window into your site. They help spotlight drastic changes, and make your website’s general traction more visible. Certainly valuable for when your CMO corners you by the Keurig machine. It’s a state of the union, but doesn’t focus on subsections that may need attention.

Agencies and consultants tend to create SEO reports for their clients as a standard practice, though sometimes these reports become extremely boilerplate. Boilerplate reports essentially force you to look under the same rocks month after month. How can you get a bigger view of the world if you never leave your comfortable neighborhood? A new routine needs to be created by generating new reports and correlations, finding trends that were hidden, and using all the tools at your disposal (from Analytics to link tools to competitive tools).

Your analytics app is not a toy—it’s the lifeblood of your website.

Deeper dives with Google Analytics

Grouped pages lookup

A quick way to look at chunks of the site is by identifying a footprint in the URL and searching with that. For example, go to Behavior > Site Content > All Pages or Landing Pages. Then, in the search bar right below the graph, search for the footprint. For example, take www.mystoreisdabomb.com/blog/2015/ as a real URL. if you want to see everything in the blog, enter */blog/ into the search bar. This is especially useful in getting the temperature of an eCommerce category.

Segment sessions with conversions/transactions

So often in SEO we spend our time analyzing what’s not working or posing as a barrier. This report helps us take a look at what is performing (by leads or sales generated) and the customer behavior, channels, and demographic information that goes along with that. Then we can identify opportunities to make use of our success and improve our overall inbound strategy.

Below is a deeper dive into the conversions “Lead Generation” segment, although these same reports can just as aptly be applied to transactions. Ultimately, there are a lot of ways to slice and dice the analysis, so you’ll have to know what makes sense for your client, but here are three different reports from this segment that provided useful insights that will enhance our strategy.

  • Conversions
    One of the easy and most valuable ones! Directions: Under any report, go to Add a Segment > Sessions with Conversions > Apply.
  • Demographics – age, gender, location
    For example, our client is based in Pennsylvania, but is receiving almost as many request form submissions from Texas and New York, and has a high ratio of request form submissions to visitors for both of these other states. Given our client’s industry, this gives us ideas on how to market to these individuals and additional information the Texans may need given the long distance.
  • Mobile – overview, device type, landing pages
    For this client, we see more confirmation of what has been called the “micro-moment” in that our mobile users spend less time on the site, view less pages per visit, have a higher bounce rate, and are more likely to be new users (less brand affinity). This would indicate that the site is mobile optimized and performing as expected. From here, I would next go into mobile traffic segments to find pages that aren’t receiving a lot of mobile traffic, but are similar to those that are, and find ways to drive traffic to those pages as well.
  • Acquisition
    Here we’re looking at how the inbound channels stack up for driving conversions. Organic and Paid channels are neck and neck, although referral and social are unexpected wins (and social, glad we’ve proven your viability to make money!). We’ll now dig deeper into the referring sites and social channels to see where the opportunities are here.

Assisted conversions

There’s more to the story than last click. In Analytics, go to Conversions > Multi-Channel Funnels > Assisted conversions. Many clients have difficulty understanding the concept of attribution. This report seems to provide the best introduction to the world of attribution. Last click isn’t going to be replaced anytime soon, but we can start to educate and optimize for other parts of the funnel.

True stories from analytics detective work

Granted, this is not a post about favorite reports. But this is a post about why digging through analytics can open up huge opportunities. So, it’s real-life example time from Greenlane’s own experience!

Story 1: The Forgotten Links

The client is a big fashion brand. They’ve been a popular brick-and-mortar retail destination since the early 80s, but only went online in 1996. This is the type of company that builds links based on their brand ambassadors and trendy styles. SEO wasn’t the mainstream channel it is today, so it’s likely they had some serious architecture changes since the 90s, right?

For this company, analytics data can only be traced back about seven years. We thought, “Let’s take a look at what drove traffic in their early years. Let’s see if there were any trends that drove volume and sales where they may be slipping today. If they had authority then, and are slipping now, it might be easier to recoup that authority versus building from scratch.”

The good news—this brand had been able to essentially maintain the authority they launched with, as there were not any real noticeable gaps between search data then and search data today. But, in the digging, we uncovered a gem. We found a lot of URLs that used to draw traffic that are not on their tree today. After digging furthur, we found a redesign occurred in the late 90s. SEO wasn’t factored in, creating a ton of 404s. These 404s were not even being charted in Google Webmaster Tools, yet they are still being linked to today from external sites (remember, GWT is still quite directional in terms of the data they provide). Better yet, we pulled links from OSE and Majestic, and saw that thousands of forgotten links existed.

This is an easy campaign—create a 301 redirect matrix for those dead pages and bring those old backlinks to life.

But we kept wondering what pages were out there before the days where analytics was implemented. Using the Wayback Machine, we found that even more redesigns had occurred in the first few years of the site’s life. We didn’t have data for these pages, so we had to get creative. Using Screaming Frog, we crawled the Wayback Machine to pull out URLs we didn’t know existed. We fed them into the link tools, and sure enough, there were links there, too.

Story 2: To “View All” or Not To “View All”

Most eCommerce sites have pagination issues. It’s a given. A seasoned SEO knows immediately to look for these issues. SEOs use rel=”next” and “prev” to help Google understand the relationships. But does Google always behave the way we think they should? Golly, no!

Example 2 is a company that sells barware online. They have a lot of products, and tend to show only “page 1” of a given category. Yet, the analytics showed instances where Google preferred to show the view all page. These were long “view all” pages, which, after comparing to the “page 1” pages, showed a much lower bounce rate and higher conversions. Google seemed to prefer them in several cases anyway, so a quick change to default to “view all” started showing very positive returns in three months.

Story 3: Selling What Analytics Says to Sell

I have to change some details of this story because of NDAs, but once upon a time there was a jewelry company that sold artisan products. They were fond of creating certain kinds of keepsakes based on what sold well in their retail stores. Online, though, they weren’t performing very well selling these same products. The website was fairly new and hadn’t quite earned the footing they thought their brand should have, but that wasn’t the terminal answer we wanted to give them. Instead, we wanted to focus on areas they could compete with, while building up the entire site and turning their offline brand into an online brand.

Conversion rates, search metrics, and even PPC data showed a small but consistent win on a niche product that didn’t perform nearly as well in the brick-and-mortar stores. It wasn’t a target for us or the CEO. Yet online, there was obvious interest. Not only that, with low effort, this series of products was poised to score big in natural search due to low competition. The estimated search volume (per Google Keyword Planner) wasn’t extraordinary by any stretch, but it led to traffic that spent considerable dollars on these products. So much so, in fact, that this product became a focus point of the website. Sometimes, mining through rocks can uncover gold (jewelry pun intended).

Conclusion

My biggest hope is that your takeaway after reading this piece is a candid look at your role as an SEO or digital marketer. You’re a person with a “unique set of skills,” being called upon to perform works of brilliance. Being busy does create pressure; that pressure can sometimes force you to look for shortcuts or “phone it in.” If you really want to find the purest joy in what you’ve chosen as a career, I believe it’s from the stories embedded within the data. Go get ’em, Sherlock!


Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

Continue reading →

How to Get Your App Content Indexed by Google

Posted by bridget.randolph

As mobile technology becomes an increasingly common way for users to access the internet, you need to ensure that your mobile content (whether on a mobile website or in a mobile app) is as accessible to users as possible. In the past this process has been relatively siloed, with separate URLs for desktop and mobile content and apps tucked away in app stores.

But as app and mobile web usage continues to rise, the ways in which people access this content is beginning to converge, which means it’s becoming more important to keep all of these different content locations linked up. This means that the way we think about managing our web and mobile content is evolving:

So how do we improve the interaction between these different types of content and different platforms, getting to the point of being able to have a single URL which takes the user to the most appropriate version of the content based on their personal context?

The first step is to ensure that we are correctly implementing deep linking (e.g., linking to a particular screen within an app) for apps which have comparable webpage content, to allow for our app content to rank in mobile search.

Image credit: Google Developers

Google indexation provides benefits for both Android and iOS apps. The benefits for Android apps are twofold:

  • users searching on an Android device who have not yet installed your app will see the app show up in mobile search results; and
  • Android users who do have your app installed will get query autocompletions when they use browser search which can include results from your app, as well as seeing enhanced display elements in the SERP (such as the app icon). It’s basically like rich snippets for apps.

Image credit: Google Developers

On iOS, app ranking is currently only supported for apps already installed on the device. Apple users should see search results which include links to installed apps and also include the enhanced display elements mentioned above.

In addition, Google recently announced that mobile apps which use the new App Indexing API for deep linking may receive a rankings boost in mobile web search. They are releasing a new and improved version of Google Now, “Now on Tap,” in their latest OS update (Android M), which allows you to search content across your phone without navigating out of whatever app (or website) you are currently using. The catch is, that app content has to be in their index in order to be included in a “Now on Tap” search.

It’s not just Google, either; Apple is implementing their own version of a search index to allow iOS9 users to search and discover web and app content without using a third-party search engine, Bing has its own approach to app indexation and ranking, and other services aren’t far behind.

This post, however, will focus on how to setup your Android and iOS apps to appear in Google search results. While the idea of app indexation isn’t new, it is an area of rapid innovation and the process for getting your apps indexed by Google has recently been simplified. This post is therefore intended to provide a brief overview of that process and to serve as an update to the information which is currently available.

The implementation

The good news is that it’s getting simpler to add the relevant markup to your web content and get your app content indexed and ranking in mobile search results.

The basic process is only three steps:

  1. Support HTTP deep links in your mobile app. For iOS you will need to do this by setting up support for “Universal Links.” “Universal Links” are what Apple calls HTTP links that have a single URL which can open both a specific page on a website and the corresponding view in an app.
    Note: At this point, you can register your app with Google, associate it with your website and stop there—as long as you are using the same URLs for your web content and your app content, they should be able to automatically crawl, index, and attempt to rank your app content based on your website’s structure. However, implementing App Indexing and explicitly mapping your web content to your app content using on-page markup can provide additional benefits and allow for a bit more control. Therefore, I recommend following the full process, if possible.
  2. Implement Google App Indexing using the App Indexing API for Android, or by integrating the App Indexing SDK for iOS 9.
  3. Explicitly map your web pages to their corresponding app screens using either a rel=alternate link element on the individual page, by referencing the app URLs in your XML sitemaps, or by using schema.org markup.

You can find a more step-by-step explanation of this process (looking at Android and iOS separately) below.


The app indexation process used to be a bit more complex, because HTTP links aren’t supported by older iOS versions. Instead, developers had to use something called “Custom URL Schemes” to link to iOS app content. This meant that you essentially had to create a unique scheme for your app URLs and then add support for these in the app code.

Custom URL schemes have a couple other downsides besides adding complexity, namely:

  • different app developers can claim the same custom URL scheme, whereas with HTTP links you can associate the app to a particular domain or set of domains; and
  • with custom URL schemes, tapping the URL when the app isn’t installed results in a broken link (because it only links to content within the app), whereas HTTP links are web links as well and can take the user to a webpage if the app isn’t installed (as long as the URL is the same for both the app view and the corresponding webpage).

While you can still use the custom URL scheme approach, the good news is that Google’s App Indexing is now compatible with HTTP deep link standards for iOS 9, which Apple calls “Universal Links.”

You should still add markup to any webpages which have content corresponding to a particular app screen. Think of it like like rel=canonical or like mobile switchboard tags, but for apps. Be aware that when Google finds a link between a webpage and an app page which they think are equivalent, they will compare the two pages and you will receive a ‘Content Mismatch’ error in the Search Console if they don’t believe the content is similar enough.

Getting Android apps indexed in Google

Step 1: Support HTTP deep links in your app by adding intent filters to your manifest.

An intent filter is a way of specifying how an app responds to a particular action. Intent filters for deep links have three required elements: <action>, <category>, and <data>. You can find more guidance on this from Google Developers. Here is their example of an intent filter which enables support for HTTP deep links:

<intent-filter android:label="@string/filter_title_viewrecipes">

<action android:name="android.intent.action.VIEW" />
<category android:name="android.intent.category.DEFAULT" />
<category android:name="android.intent.category.BROWSABLE" />
<data android:scheme="http"
android:host="recipe-app.com"
android:pathPrefix="/recipes" />
</intent-filter>
</activity>

Noindex option:
Just like for websites, you can add noindex directives for app content as well. Include a noindex.xml file in your app to indicate which deep links should not be indexed, and then reference that file in the app’s manifest (AndroidManifest.xml) file. You can find more detail on how to create and reference the noindex.xml file here.

Step 2: Associate your app to your site in Google Search Console.

This is done in Google Search Console (you can also do it from the Developer Console). As long as your app is set up to support deep links, this step is technically all you have to do to allow Google to start indexing your app. It will allow Google to index and crawl your app automatically by attempting to figure out the app structure from your website structure.

However, if you do stop here, you will not have as much control over how Google understands your content, which is why the explicit mapping of pages to app versions is recommended. Also, if you can’t use the API for some reason, you need to make sure that Googlebot can access your content. You can check that this is configured correctly in your site’s robots.txt file by testing some of your deep links using the robots.txt tester tool in the Search Console.

Step 3: Implement app indexing using the App Indexing API.

Using the App Indexing API is definitely worthwhile; apart from anything else, apps which use the API should receive a rankings boost in mobile search results, and you don’t need to worry about Googlebot struggling to access your content.

The App Indexing API allows you to annotate information about the activities within your app that support deep links (as laid out in your intent filters). For details on how to set this up, see the Google Developers guidance.

Step 4: Test your implementation.

You can test your implementation (always on a fresh installation of your app!) with the following tools. (Find more info about how to use each of these tools here.)

Android Debug Bridge – to test deep links from the command line

Fetch as Google (Search Console) – to test what Google sees when it crawls your app deep links

You can also track search traffic to these deep links in the Search Console’s Search Analytics report.

Getting iOS apps indexed in Google

Step 1: Support HTTP deep links in your app by setting up support for “Universal Links.”

To support universal links in your iOS app, you need to first ensure that your app handles these links correctly by adopting the UIApplicationDelegate methods (if it doesn’t already use this protocol). Once this is in place, you can associate your app with your domain.

You’ll do this by:

  • adding an “associated domains” entitlement file to your app’s project in XCode that lists each domain associated with your app; and
  • uploading an apple-app-site-association file to each of these domains with the content your app supports—note that the file must be hosted at the root level and on a domain that supports HTTPS.

To learn more about supporting Universal Links, view the Apple Developer guidance.

Step 2: Register your app with Google (using the GoogleAppIndexing SDK for iOS 9).

You’ll need to add the App Indexing SDK to your app using the CocoaPods dependency manager. For step by step instructions, check the Google Developers’ guide. Basically what this does is allows you to register your app with Google, just like Android apps are registered via the Search Console. This also means that Google can now read the apple-app-site-association file to understand what URLs your app can open.

Step 3: Test your implementation.

You can test whether this is set up correctly by tapping a universal link in Safari on an iOS 9 device and checking that it opens the right location in your app.

Mapping your webpages to your app with on-page markup or sitemaps

Once you’ve set up the deep linking support for your Android and/or iOS app(s), the final step is to explicitly identify the corresponding webpages to the correct app screens using one of the supported markup options. This step allows you to indicate more clearly to Google what the relationship is between a given page and its corresponding app link (both of which should already share the same URL if you are using HTTP links). Following this step also allows you to indicate the relationship to Bing crawlers, which otherwise wouldn’t see the app content, and to allow Apple to index your iOS app.

You can do this mapping either in the head of the individual page using a link element, using schema.org markup (for Android only), or in an XML sitemap.

A note on formats for app links

The format for an Android HTTP link uses the format of:

android-app://{package_name}/http/{host_path}

The {package_name} is the app’s “Application ID,” which is how it is referenced in the Google Play Store. So a link to the (example) Gizmos app might look like this:

android-app://com.gizmos.android/http/gizmos.com/example

For iOS links, you use the app’s iTunes ID instead of the Package Name. So an iOS app URL uses this format:

ios-app://{itunes_id}/{scheme}/{host_path}

For HTTP links the {scheme} is “http,” which would mean your URL would look like this:

ios-app://{itunes_id}/http/{host_path}

How to reference your app links

Note: Google provides guidance on the three currently supported deep link methods here.

Option 1: Link rel=alternate element

To add an app link reference to an individual page, you can use an HTML <link> element in the <head> of the page.

Here is an example of how this might look if you have both an iOS and Android app:

<html>
<head>
...
<link rel="alternate" href="android-app://com.gizmos.android/http/gizmos.com/example" />
<link rel="alternate" href="ios-app://123456/http/gizmos/example" /></head>
<body> … </body>

Option 2: Schema.org markup (currently supported on Android only)

Alternatively, if you have an Android app, you can use schema.org markup for the ViewAction potential action on an individual page to reference the corresponding app link.

Here is an example of how this might look:

script type="application/ld+json">
{
"@context": "http://schema.org",
"@type": "WebPage",
"@id": "http://gizmos.com/example",
"potentialAction": {
"@type": "ViewAction",
"target": "android-app://com.gizmos.android/http/gizmos.com/example"
}
}
</script>

Option 3: Add your app deep links to your XML sitemap

Instead of marking up individual pages, you can use an <xhtml:link> element in your XML sitemap, inside the <url> element specifying the relevant webpage.

Here is an example of how this would look if you have both an iOS and an Android app:

<?xml version="1.0" encoding="UTF-8" ?>
http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:xhtml="http://www.w3.org/1999/xhtml">
<url>
http://gizmos.com/example
<xhtml:link rel="alternate" href="ios-app://123456/http/gizmos/example" /></url>
<xhtml:link rel="alternate" href="android-app://com.gizmos.android/http/gizmos.com/example" />
...
</urlset>

Additional information

What about apps which don’t have corresponding web pages?

Unfortunately, as of this writing, Google does not officially offer app indexation for apps which don’t have corresponding web content. However, they are trying to move in this direction, and as such are beginning to try this out with a handful of apps with “app-only” content. If you have an app with app-only content, and would like to get this content indexed, you can express interest using this form.

What about getting my app indexed in Bing?

Bing supports two open standard options for linking webpages to app links:

To learn more about how to implement these types of markup, see the guidance on the Bing blog.

Quick reference checklists

Will Critchlow recently spoke about app indexation in his presentation at Searchlove London. He provided two useful checklists for Android and iOS app indexing:

Image source: http://www.slideshare.net/DistilledSEO/searchlove-…

To learn more about app indexing by Google, check out Emily Grossman and Cindy Krum’s excellent post over on SearchEngineLand.


Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

Continue reading →

Will Google Bring Back Google Authorship?

Posted by MarkTraphagen

Recently, Google Webmaster Trends analyst Gary Illyes surprised many of us with a remark he made during his keynote Q&A with Danny Sullivan at SMX East in New York City. Illyes said that he recommended webmasters not remove the rel=author tag from their site content.

Google had used rel=author as part of its Google Authorship feature that (potentially) displayed a special author rich snippet in search results for content using the tag. Google ended support of this feature in August 2014.

The phrase that made everyone sit up and say, “Did he just say that?” was this: “…because it is possible Google might make use of [rel=author] again in the future.”

Even though Google’s John Mueller made the same recommendation after he announced that Google was no longer making use of Google Authorship in search (to be precise, Mueller said leaving the tag in place “did no harm”), Illyes’s statement seemed to shock many because Google has said nothing about Google Authorship or the rel=author tag since they said they stopped supporting it.

In a subsequent Twitter exchange I had with Gary Illyes, he explained that if enough users are implementing something, Google might consider using it. I asked him if that meant specifically if more people started using rel=author again, that Google might make use of it again. Illyes replied, “That would be safe to say.”

Before I provide my commentary on what all this means, and whether we should expect to see a resumption of Google Authorship in Google Search, let me provide a brief overview of Authorship for anyone who may not be familiar with it. If you already understand Google Authorship, feel free to skip down to the Will Google Bring Back Authorship? section.

A brief history of Google Authorship

Google Authorship was a feature that showed in Google Search results for about three years (from July 2011 until August 2014). It allowed authors and publishers to tag their content, linking it to an author’s Google+ profile, in order to provide a more-certain identification of the content author for Google.

In return, Google said they might display an authorship rich snippet for content so tagged in search results. The authorship rich snippet varied in form over the three years Authorship was in use, but generally it consisted of the author’s profile photo next to the result and his or her byline name under the title. For part of the run of Authorship, one could click on an author byline in search to see results showing related content from that author.

Google Authorship began with an official blog post in June of 2011 where Othar Hansson announced that Google would begin supporting the rel=author tag, but with no specifics on how they might use it.

Then in a July 2011 video, Hansson and Matt Cutts explained that Google+ would be the hub for author identification, and that Google might start showing a special Authorship rich snippet result for properly tagged content.

Those rich snippets slowly began appearing for more and more authors using rel=author over the next several months. During the three years of the program, Google experimented with many different configurations of the rich snippet, and also which authors and content would get it in response to various search queries.

Interest in Google Authorship from the SEO and online marketing communities was spurred even more by its possible connection to Google’s Agent Rank patent, first revealed by Bill Slawski. In this patent, Google described a system by which particular “agents” or “entities” could be identified, scored by their level of authority, and that score then be used as a search ranking factor.

Since one of the types of agents identified in the patent was a content author, the patent rapidly became known as “author rank” in the SEO community. The connection with Authorship in particular, though, came from Cutts and Hansson stating in the above-mentioned Authorship video that Google might someday use Authorship as a search ranking factor.

Speculation about so-called Author Rank, and whether or not it was “on” as a ranking factor, continued throughout the life of the Authorship program. Throughout that period, however, Cutts continued to refer to it as something Google might do in the future. (You can find my own take on why I believed Authorship was never used as a direct ranking factor here.)

The first hint that Google might be drawing back from Authorship came at Pubcon Las Vegas in October 2013 when Matt Cutts, in his keynote “State of Search” address, revealed that at some point in the near future Google would be cutting back on the amount of Authorship rich snippets shown by “around 15%.” Cutts said that in experiments, Google found that reducing Authorship rich snippets by that much “improved the quality of those results.”

Sure enough, in early December of that year, Moz’s Peter Meyers detected a rapid decline over several days in the number of Authorship rich snippets in search results, as measured by his Mozcast Features tool.

Around that same time Google implemented what I called “two-class Authorship,” a first class of authors who continued to get the full rich snippet, and a second class who now got only a byline (no author photo).

Finally, in August 2014, this author was contacted directly by John Mueller, offering to share some information under an NDA embargo until the information was made public. In my call with Mueller, he told me that he was letting me know 24 hours in advance that Google Authorship was going to be discontinued. He added that he was making this call as a courtesy to me since I had become the primary non-Google source of information about Authorship.

With that information, Eric Enge and I were able to compose an in-depth article on Authorship and its demise for Search Engine Land that went live within two minutes of John Mueller’s own public announcement on Google+. In our article linked above, Eric and I give our takes on the reasons behind the death of Authorship and the possible future of author authority on Google.

Will Google bring back Authorship?

From the day Authorship was “killed” in August 2013, we heard no more about it from Google—until Gary Illyes’s remarks at SMX East. So do Gary’s remarks mean we should expect to see a return of Google Authorship to search results?

I don’t think so, at least not in any form similar to what we saw before.

Let me explain why.

1. Illyes made no promise. Far too often people take statements about what Google “could” or “might” do from spokespersons like Gary Illyes, Matt Cutts, and John Mueller and translate “could/might” to “will.” That is unfair to those spokespeople, and an abuse of what they are saying. Just because something is spoken of as a possibility, it does not follow that a promise is being made.

2. It ain’t broke so…. So if there are no actual plans by Google to restore Google Authorship, why would Illyes make a point of stating publicly that authors and publishers should continue to use the rel=author tag? I think a primary reason may be that once Google gets any set of people to begin using any kind of schema, they’d rather have it remain in place. Anything that helps better organize the information on web pages is good for a search engine, whether or not that particular information is “in play” at present.

In the case of rel=author, I think it still may be useful to Google to be able to have confidence about content connected with certain authors. When Authorship ended, many people asked me if I were going to remove the tags from my content. I responded why would I? Having them there doesn’t hurt anything. But more important, as an author trying to build my personal brand reputation online, why wouldn’t I want to give Google every possible hint about the content with which I should be identified?

3. The reasons why Authorship was killed still remain. As with any change in Google search, we’ll probably never know all the reasons behind it, but the public reasons stated by John Mueller centered around Google’s commitment to a “mobile first” user experience strategy. Mobile first is a recognition that search is more and more a mobile experience. Recently, Google announced that more of all searches are now done on mobile than desktop. That trend will likely never reverse.

In response, we’ve seen Google continually moving toward simpler, cleaner, less-cluttered design in all its products, including search. Even their recent logo redesign was motivated by the requirements of the small screen. According to Mueller, Authorship snippets were too much clutter for a mobile world, with not enough user benefit to justify their continuation.

In our Search Engine Land article, Eric Enge and I speculated that another reason Google may have ended the Authorship experiment was relatively poor adoption of the tagging, low participation in Google+ (which was being used as the “anchor” on Google’s side for author identification), and incorrect implementation of the tags by many who did try to use them.

On the latter point, Enge conducted a study of major publishers, which showed that even among those who bothered to implement the authorship tagging, the majority was doing it wrong. That was true even among high-tech and SEO publications!

Alt that points to a messy and lopsided signal, not the kind of signal a search engine wants. At the end of the day, Google couldn’t guarantee that a result showing an Authorship rich snippet was really any better than the surrounding results, so why give it such a prominent highlight?

Despite Gary Illyes saying that if more sites used rel=author Google might begin using it again, I don’t see that doing so would change any of the conditions stated above. Therefore, I believe that any future use of rel=author by Google, if it ever occurs, will look nothing like the Authorship program we knew and loved.

So is there any future for author authority in search?

To this question, I answer a resounding “Yes!”

Every indication I’ve had from Googlers, both publicly and privately, is that author authority continues to be of interest to them, even if they have no sound way to implement it yet.

So how would Google go about assessing author identity and authority in a world where authors and publishers will never mass-tag everything accurately?

The answer: the Knowledge Graph, entity search, and machine learning.

The very first attempts at search engines were mostly human-curated. For example, the original Yahoo search was fed by a group of editors who attempted to classify every web page they came across. But as the World Wide Web took off and started growing exponentially, it was quickly obvious that such attempts couldn’t scale. Hyperlinks between web pages as a means of assessing both the subject matter and relative authority of web pages proved to be a better solution. Search at the scale of the web was born.

Remember that Google’s actual mission statement is to “organize the world’s information.” Over time, Google realized that just knowing about web pages was not enough. The real world is organized by relationships between entities—persons, places, things, concepts—and Google needed a way to learn the relationships between those things, also at scale.

The Knowledge Graph is the repository of what Google is learning, and machine learning is the engine that helps them do that learning at scale. At a simple level, search engine machine learning is the development of an algorithm that learns on its own as a result of feedback mechanisms. Google is applying this technology to the acquisition of and linking together of entities and their relationships at scale.

It’s my contention that this process will be the next evolutionary step that will eventually enable Google to identify authors who matter on a given topic with their actual content, evaluate the relative authority of that content in the perceptions of readers, and use that as a search ranking factor.

In fact, Matt Cutts seemed to hint at a Knowledge Graph-based approach in a June 2013 video about the future of authorship where he talked about how Google was moving away from dependence on keywords, from “strings to things,” figuring out how to discover the “real-world people” behind web content and “their relationships” to improve search results.

Notice that nothing in a machine learning process is dependent upon humans doing anything other than what they already do on the web.

The project is already underway. Take a moment right now and ask Google, “Who is Mark Traphagen?” If you are in the US or most English-speaking countries, you’ll probably see this at the top of the results:

That’s a Knowledge Panel result from Google’s Knowledge Graph. It reveals a couple of things:

1. Google has a high confidence that I’m likely the droids, er, the “Mark Traphagen” you’re looking for. There are a few other Mark Traphagens in the world who potentially show up in Google Search, but Google sees that the vast majority of searchers who search for “Mark Traphagen” are looking for a result about me. Thanks, everybody!

2. Google has high confidence that the Mark Traphagen you’re looking for is the guy who writes for Search Engine Land, so that site’s bio for me is likely a good instant answer to your lifelong quest to find the Real Mark Traphagen (a quest some compare to the search for the Holy Grail).

If Google can continue to do that at scale, then they can lick a problem like assessing author authority for search rankings without any help from us, thank you very much.

How does all this fit with Gary Illyes’s recommendation? I think that while Google knows it ultimately has to depend on machine learning to carry off such projects at scale, any help we can give the machine along the way is appreciated. Back in the Google Authorship I days, some of us (myself included) believed that one of the real purposes for the Authorship project was to enlist our help in training the machine learning algorithm. It may be that rel=author is still useful for that.

What might Authorship look like in the future?

Allow me to speculate a bit.

I don’t expect we’ll ever again see the mass implementation of author rich snippets we saw before, where almost anyone could get highlighted just for having used the tagging on their content and having a Google+ profile. As I stated above, I think Google saw that doing that was a non-useful skewing of the results, as more people were probably clicking on those rich snippets without necessarily getting a better piece of content on the other end.

Instead, I would expect that Google would see the most value in identifying the top few authors for any given topic, and boosting them. This would be similar to their behavior with major brands in search. We often see major, well-known brands dominating the top results for commercial queries because user behavior data tells Google that’s what people want to see. In a similar way, people might be happy to be led directly to authors they already know and trust. They really don’t care about anyone else, no matter how dashing their profile image might be.

Furthermore, for reasons also stated above, I don’t expect that we’ll see a return to the full rich snippets of the glory days of Authorship I. Instead, the boost to top authors might simply be algorithmic; that is, other factors being equal, their content would get a little ranking boost for queries where they are relevant to the topic and the searcher.

It’s also possible that such author’s content could be featured in a highlighted box, similar to how we see local search results or Google News results now.

But notice what I said above: “…when [the authors] are relevant to the topic and the searcher.” That latter part is important, because I believe it is likely that personalization will come into play here as well. It makes sense that boosting or highlighting a particular author has the most value when my search behavior shows that author already has value to me.

We already see this at work with Google+ posts in personalized (logged in) search. When I search for something that AJ Kohn has posted on Google+ while I’m logged in to my Google account, Google will elevate that result to my first page of results and even give it a good old-fashioned Authorship rich snippet! Google has high confidence that’s a result I might want to see because AJ is in my circles, and my interactions with him and his content show that he is probably very relevant and useful to me. Good guess, Google, you’re right!

It is now obvious that Google knows they have to expand beyond Google+ in entity identification and assessment. If Google+ had taken off and become a real rival to Facebook, Google’s job might have been a lot easier. But in the end, building machine learning algorithms that sniff out our “who’s who” and “who matters to whom” may be an even better, if vastly more difficult, solution.

So to sum up, I do expect that at some point in the future, author authority will become a factor in how Google assesses and ranks search results. However, I think that boost will be a “rich get richer” benefit for only the top, most reputable, most trusted authors in each topic. Finally, I think the output will be more subtle and personalized than we saw during the first attempt at Authorship in search.

How to prepare for Authorship II

Since it is unlikely that Authorship II, the future implementation of author identity and authority in search, will be anything like Authorship I, is there anything you can be doing to increase the odds that Authorship II will benefit you and your content? I think there are several things.

1. Set a goal of being the 10X content creator in your niche. Part of the Gospel According to Rand Fishkin these days is that “good, unique” content is not good enough anymore. In order to stand out and get the real benefits of content, you’ve got to be producing and publishing content that is ten times better than anything currently on page one of Google for your topic. That means it’s time to sacrifice quantity (churning out posts like a blogging machine) for quality (publishing only that which kicks butt and makes readers stand up, take notice, and share, recommend and link).

2. Publishers need to become 10X publishers. If you run a publishing site that accepts user-generated content, you’ve got to raise your standards. Accepting any article from any writer just to fill space on your pages won’t cut it.

3. Build and encourage your tribe. If you are authoring truly great, useful stuff, sooner or later you will start to attract some fans. Work hard to identify those fans, to draw them into a community around your work, and to reward and encourage them any way you can. Become insanely accessible to those people. They are the ones who will begin to transmit the signals that will say to Google, “This person matters!”

4. Work as hard offline as you do online. Maybe harder. More and more as I talk with other authors who have been working hard at building their personal brands and tribes, I’m hearing that their offline activities seem to be driving tremendous benefit that flows over into online. I’m talking about speaking at conferences and events, being available for interviews, being prominent in your participation in the organizations and communities around your topic, and dozens of other such opportunities.

BONUS: Doing all four of those recommendations will reap rewards for you in the here and now, whether or not Google ever implements any kind of “author rank.”

The natural power of the fact that people trust other people long before they will trust faceless brands continues, in my opinion, to be one of the least understood and underutilized methodologies in online marketing. Those who work hard to build real author authority in their topic areas will reap the rewards as Google begins to seek them out in the days to come.


Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

Continue reading →

How to Integrate Traditional &amp; Digital Marketing

Posted by SamuelScott

It’s October 26, 1985. The top song in the United States is Whitney Houston’s “Saving All My Love for You.” The top weekend movie at the US box office is “Jagged Edge,” starring Glenn Close and Jeff Bridges. And Marty McFly was about to travel back in time in “Back to the Future.”

(photo: Back to the Future Day)

Yes, for any Mozzers who do not know, Back to the Future Day—the exact date when Marty went to the future in the sequel to the original film—is today!

Also in the mid-1980s, the Internet was largely confined to the US military and large higher-education institutions. Most marketing at the time, of course, occurred via print, magazine, TV, and radio mediums and channels in what is now often called “outbound marketing.”

Why, then, is this relevant in 2015? One problem in digital marketing is that many digital marketers do not have much education or experience in traditional marketing and communications. If you mention the 4 Ps or ask about the promotion mix at most SEO conferences, you’ll probably receive blank stares in response.

It’s important to know what marketers did before the Internet because many of the strategies that had been developed and honed since the early twentieth century are still applicable today. So, to help the community, I wanted to give a high-level overview of traditional marketing and communications and then provide discussion topics for the comments below, as well as actionable tasks for readers to start integrating traditional marketing principles into their digital strategies.

By the end of this post, you’ll have a solid sense of the following:

  • What is the difference between marketing and communications?
  • What is the integrated discipline “marcom?”
  • What are the 4 Ps?
  • What parts comprise the promotion mix (within the 4 Ps)?
  • When should I do outbound and inbound marketing?
  • How do the Internet and digital marketing fit into all of this?
  • What about SEO, social media marketing, content marketing, growth hacking, and linkbuilding?
  • What actionable things can I do?
  • Why did Marty McFly’s mom and dad not recognize him from the 1950s when he had grown into a teenager in the 1980s?

This post is related to an earlier Moz post of mine on the marketing department of the future—I will address the connection between the two essays below. I hope you’re excited! Where we’re going, we don’t need roads—just a little bit of time.

All “Back to the Future” photos are from the official site

The basic points to remember

Here are two of my favorite quotes in the sidebar of the blog of advertising industry veteran Bob Hoffman:

Marketing does not really change that much because people do not change—and anyone who says differently is selling something. Countless gurus, writers, and keynote speakers have proclaimed that “inbound marketing is the future,” “social media has changed everything (a parody of TED talks),” and “advertising is dead.”

But some examples from Hoffman prove otherwise: Traditional live TV viewing is not “dying” at all, and most people are not using DVRs to skip TV ads. Traditional “outbound” TV advertising is often still important enough that in one example, Pepsi lost a lost of money and dropped to third place in terms of market share when the company moved its entire ad spend from television ads to social media.

The point is that no marketing strategy or tactic is always best for every purpose, product, brand, or industry. Sometimes TV advertising should be a part of the promotion mix; sometimes not. Sometimes content marketing is the best way to go; sometimes not. Sometimes modern “inbound” methods deliver the greatest value; sometimes it’s traditional “outbound” ones. More on that below.

Here, I wanted to take us back—back to the past to show the future where we have been, why it matters, and how we should incorporate it into digital marketing.

A full list of resources is provided at the end for those who wish to learn more.


The marcom workflow

This flowchart is a high-level overview of the step-by-step process that I will describe in detail below. For those who want a quick summary, here is the workflow that marcom executives typically use:

  1. List all of the external audiences with whom your company communicates
  2. Remember that current and potential customers are just one of your audiences
  3. Determine the specific messages that you want to communicate to each “public” based on your audience research and buyer personas

    For customers:
  4. Establish product-market fit
  5. Decide on a pricing strategy
  6. Choose how you will prioritize direct marketing, personal selling, advertising, sales promotion, and publicity in your promotion mix in general and then allocate your resources accordingly
  7. Determine which online and offline communications channels you will prioritize within your promotion mix and then allocate your resources accordingly
  8. Create the needed creatives, sales collateral, and marketing content
  9. Transmit the marketing materials over your selected channels to your audience
  10. Measure and audit the results
  11. Return to steps 3–8 as necessary and adjust to attempt to maximize revenue, sales, ROI, or any other metric based on your business and marketing goals

Marketing vs. communications

Traditionally, marketing and communications had been entirely different functions that each had their own departments. Marketing focused on issues such as customers, sales, and brand awareness. Communications (often called public relations or external relations) dealt with everyone else in the outside world with whom the company interacted, such as the government, community, media, and financial analysts.

In other words, communications (another word for PR) as a whole is the act of communicating with any relevant external group of people. Obviously, companies would usually not want to say the exact same thing to customers, influencers, the media, the government, and the local community.

Today, however, more and more companies are combining Marketing and Communications under a single department (often called “marketing communications” or “marcom”) to become more efficient and ensure that all messages are consistent among all audiences and across all channels.

The key to understand: Publish and transmit unified, integrated messaging on and across all online and offline marcom channels including websites, social networks, advertising campaigns, online content, news releases, product brochures, and sales catalogues.

SEO pro tip: When relevant, include your desired keywords—using natural language rather than keyword stuffing, of course—in your messaging everywhere (for the co-occurrence benefits; websites seem to rank more highly for search terms when links to those sites appear on pages that also mention those terms!).

Remember:

Communication is one person speaking with another. Marketing is one type of communication.


The 4 Ps in traditional marketing

In a unified marcom strategy, all of a company’s audiences need to be included strategically. However, because Moz’s readers focus mainly on marketing rather than PR, I will focus the rest of this essay on “customer relations” (or “marketing”) specifically.

After creating the overall marketing messages that a company will communicate to current and potential customers, the next step in traditional marketing theory is to focus on the 4 Ps: product, price, place, and promotion. (Note: This process is not set in stone. Sometimes the 4 Ps will be determined before the overall marketing messaging is decided.)

Product

Not a bad product!

Product strategy, according to Study.com, is essentially maximizing product-market fit—the degree to which a product or service satisfies the demand for it. In the modern, inbound world (and most specifically in terms of SaaS products), product strategy has been rebranded as “growth hacking.” For those who are interested, Ryan Holiday goes into more detail in “Growth Hacker Marketing.”

The process generally follows these steps:

1. Create a Minimum Viable Product (MVP), traditionally called a prototype.

2. Get a core set of early testers/users to establish product-market fit. Show them the MVP and quiz them on what they like, what they don’t like, what would be more useful, and what is not necessary. Revise the product and get more feedback, revise the product and get more feedback to have a product that is constantly improving.

3. Incorporate sharing and growth naturally in the product. Push new users to refer friends for a discount. Put social sharing buttons inside the user dashboard.

4. Experiment with the different ways to get “traction” to see what gets the most users quickly at the lowest cost. It could be advertising, organic traffic, media coverage, or any other potential marketing strategy.

Once you have established product-market fit, how will you communicate the value that the product provides on all online and offline marcom channels, including websites, social outlets, advertising campaigns, online content, news releases, product brochures, and sales catalogues?

Price

Thankfully, this ended up not happening.

Pricing strategy is not only a purview of the finance department—it is also a part of marketing. If, for example, a company wants to earn $100 in revenue, it can sell one “widget” for $100 or 100 “widgets” for $1 each. Each tactic requires a different marketing strategy.

A high price communicates high value and rarity (think about the high prices of diamonds, which are essentially shiny, useless rocks marketed to rich people) while a low price shows affordability and availability (think about Walmart and its value-based marketing to the less-rich Everyman). In a marketing context, the pricing strategy will also need to take into account items such as the size of the market, the cost to produce the product or service, the level of competition, the economics of the target market, and whether the company wants to market itself based on quality or value (or perhaps both).

Once you have decided on your pricing strategy, how will you incorporate that element into online and offline channels, including websites, social outlets, advertising campaigns, online content, news releases, product brochures, and sales catalogues?

Place

Distribution (how Place is now known) is the decision on how to transmit a product to customers. It might be setting up a lemonade stand on the corner, using franchisees, or mass producing widgets and then selling them to intermediaries, who then resell them.

In the Internet Age, when products and services can distributed, purchased, and consumed or used almost immediately from anywhere, this traditional part of the 4 Ps is becoming less important.

Promotion

One way to promote yourself is to get in the mass media.

Promotion is “raising customer awareness of a product or brand, generating sales, and creating brand loyalty.”

Promotion is also the most complex part of the 4 Ps because of the decisions that marketers make when deciding which methods are the best ways to achieve those three goals. The promotion mix has always consisted of five elements: direct marketing, sales promotion, personal selling, advertising, and publicity. Based on the product, industry, goal, and target audience, each element is given a different weight and priority both online and offline.


The promotion mix

For a second, forget about “SEO,” “content marketing,” “social media marketing,” and the other terms that are frequently used among digital marketers. Here are the five elements of the traditional promotion mix described in specific detail.

I will use specific definitions from my old MBA marketing textbook (“Principles of Marketing” by Philip T. Kotler and Gary Armstrong) because, as I will explain below, how we define our terms greatly affects our marketing success.

Direct marketing

Direct marketing is “direct connections with carefully targeted individual consumers to both obtain an immediate response and cultivate lasting customer relationships.” Direct marketing “includes catalogs, telephone marketing, kiosks, the Internet, mobile marketing, and more. ”

Advertising

Advertising is “any paid form of non-personal presentation and promotion of ideas, goods, or services by an identified sponsor.” Advertising “includes broadcast, print, Internet, outdoor, and other forms.”

Personal selling

Personal selling is “personal presentation by the firm’s salesforce for the purpose of making sales and building customer relationships.” Personal selling “includes sales presentations, trade shows, and incentive programs.”

Sales Promotion

Sales promotion is “short-term incentives to encourage the purchase or sale of a product or service.” Sales promotion “includes discounts, coupons, displays, and demonstrations.”

Publicity

Publicity is “gaining public visibility or awareness for a product, service, or your company via the media.”

To learn more about publicity, read “The Father of Spin,” a biography on the person who essentially invented the practice in the early-twentieth century. As described in Amazon’s description, he did “publicity campaigns for American Tobacco, Ivory Soap, United Fruit, book publishers, manufacturers of eggs and bacon, and the platforms of presidents from Coolidge to Eisenhower.” (Marketing can be done for good or evil ends—I will leave the choice up to the readers.)

For a modern example, see this Kickstarter campaign to build a miniature version of the hoverboard that was shown in “Back to the Future II”:

Perhaps inspired by the Kickstarter, Lexus is also releasing its own hoverboard and getting a lot of publicity as a result. How much brand awareness and how many links and social shares and followings do you think they have gained as by-products?

Creating the best promotion mix

The chart above presents an overview of the uses, benefits, and drawbacks of each element of the promotion mix. Answering some of the questions below can help to determine which of the following are relevant to one’s marketing goals and show companies to allocate resources to each element appropriately.
  • I need to introduce a new product to a new market (advertising)
  • I have a product that’s under attack by competitor’s products, and I need to retain my current customer base (sales promotion)
  • I have a product that is highly specialized, technical, or expensive (personal selling)
  • I need to correct false impressions or counter false claims made about my product (publicity)
  • I need to create greater brand awareness of my product (advertising)
  • I need to communicate new features to increase consumption by present customers (direct marketing)
  • I have a product with a long sales cycle (personal selling)
  • I need to generate more “buzz” or word-of-mouth business (sales promotion or publicity)
  • I need to build a new image and reposition my product (advertising)

Companies will generally allocate different weights to each part of the promotion mix based on how they answer these types of questions. Here is an example from the Edward Lowe Foundation, a US non-profit organization that helps to encourage local entrepreneurship:

Whichever elements you choose, how will you incorporate them into your desired online and offline channels, including websites, social outlets, advertising campaigns, online content, news releases, product brochures, and sales catalogues?


Don’t get distracted by buzzwords

Now, I have been discussing external communications, the 4 Ps, and the promotion mix—all of which have been used by traditional marketers since before the Internet had even existed. But why is this important to digital marketers today?

In my earlier Moz essay on the integration of PR and SEO (from when I was working for an agency), I explained the traditional communications process in this manner:

Here’s what I had written at that time:

A sender decides upon a message; the message is packaged into a piece of content; the content is transmitted via a desired channel; and the channel delivers the content to the receiver. Marketing is essentially sending a message that is packaged into a piece of content to a receiver via a channel. The rest is just details.

The Internet is just a new set of communications channels over which marketing promotional mixes are executed. As Kotler and Armstrong note in my old textbook (emphasis added):

As noted earlier, online marketing is the fastest-growing form of direct marketing. Widespread use of the Internet is having a dramatic impact on both buyers and the marketers who serve them. In this section, we examine how marketing strategy and practice are changing to take advantage of today’s Internet technologies.

“Digital marketing” is really just doing direct marketing, sales promotion, personal selling, advertising, or publicity via a specific collection of communications channels that we call the Internet.

The myth of “social media marketing”

I think I see Friendster and Orkut in there.

There is no such thing as “social media marketing” as a thing unto itself. (Please notice that I put that phrase as a whole in quotes.)

  • Take the Lexus Back to the Future hoverboard’s Facebook page. All of the posts that are gaining likes, comments, and shares are not examples of “social media marketing”; by definition, it is “publicity” (via an online channel) because it is “gaining public visibility or awareness for a product, service, or your company via the media.”
  • If I export a list of people who mention “widgets” on Twitter and then tweet to each person to sell them widgets, that is not “social media marketing.” By definition, it is doing direct marketing (via an online channel) because I am establishing “direct connections with carefully-targeted individual consumers to both obtain an immediate response and cultivate lasting customer relationships.”
  • If I respond to customers who are asking questions on Facebook or Twitter about how my company’s product works, then I am not doing “social media marketing.” Obviously, I’m doing customer support.

“Social media” is not a marketing strategy. Social media, just like the telephone, is a communications channel over which marketing, PR, customer support, and more can all be performed. After all, there’s no such thing as a “telephone strategy.”

Why words matter

As a former journalist, I take pride in being very precise with language because it is always important to communicate ideas 100% accurately, fairly, and objectively. Within the articles that I contribute to the digital marketing community (and sometimes in the comments on others), I often discuss the definitions of terms because being precise with our language is the best way to help all of us do our jobs better.

If you want to integrate traditional and digital marketing—or, to be more accurate, if you want to market over digital channels—here are some examples of what to do:

  • Stop studying “social media marketing.” Start studying the best practices in “direct marketing,” “customer support,” or any other desired activity and then apply those ideas whenever you use social media channels. In the coming years, publicists, customer support representatives, and others will naturally incorporate social media into their existing functions. “Social media” is not going to be a job unto itself.
  • Stop studying “linkbuilding” and “doing content marketing to earn links.” Start studying the best practices in “publicity” because 99% of natural, quality, and authoritative links come as natural by-products of getting the media, bloggers, and people in general to talk about you online. The same principles of publicity apply regardless of whether I am using the channel of the telephone, e-mail, or social media when communicating with the media.
  • Start studying the best practices in personal selling and sales promotion, and apply those principles whenever you do personal selling or sales promotion over digital channels.

Selecting the channels for your promotion mix

What channels would the makers of “Back to the Future” use to market the film in 1985 compared to 2015?

After you have decided upon your communications messages, determined the 4 Ps, and selected your promotion mix, only then is it time to select your channels by answering these questions:

  • Can our audience be best reached online or offline (or some degree of both)?
  • Based on the answer to the first question, which channels within each category should we use? (For example, the offline channels of TV, radio, newspapers, or magazines, or the online channels of advertising networks, social media, or online communities?)

The rule of thumb is to “go where the target audience lies”—whether it’s online or offline or both—but it’s more complicated than that. Channels themselves have their positives and negatives. Here are some examples based on executing the promotion mix online or offline:

  • Advertising: The results and ROI of traditional offline advertisements are difficult to track precisely, but people generally pay more attention to them. Online advertising is easier to track, but the industry is rife with alleged fraud (a Moz essay of mine), and more and more people are using ad blockers.
  • Direct marketing: Is it best to grow an e-mail list, to run searches on Twitter to isolate groups of people who are interested in what you offer, or to send a sales catalogue and track who purchases products? The answer will be different for everyone based on the audience.
  • Personal selling: If you’re selling, say, diamonds or expensive enterprise software, fewer people might buy following a Pinterest campaign compared to using the telephone or even meeting someone in person.
  • Sales promotion: Offering quick discounts needs to convey a sense of urgency, so it’s important to use channels that will reach audiences immediately. Therefore, it makes more sense today to use mobile marketing over snail-mail, for example.
  • Publicity: Traditional and digital PR are increasingly two separate entities. As I explained in a Moz post on PR 101 for digital marketers, publicity in the past has focused on writing pitches, creating media lists, and pitching reporters and bloggers on a story. However, gaining attention for a company today may require using Facebook, Twitter, and other social media networks to ensure mass exposure to a creative campaign.

Now, I understand that Moz’s audience is focused almost entirely on digital. But digital is not always the answer. Take this question from Hoffman, the retired ad agency CEO:

First, I want you to think about your refrigerator.

Think about all the stuff that’s in there: The cheese, the eggs, the juice, the jelly, the butter, the beer, the mayonnaise, the bacon, the mustard, the frozen chicken strips…

Now think about your pantry. The cereals, the beans, the napkins, the flour, the detergent, the sugar, the rice, the bleach…

Now answer these questions:

Do you “share branded content” about any of this stuff?
Do you feel “personally engaged” with these brands?
Do you “join conversations” online about this crap?
Do you ever “co-create” with any of these brands?
Do you feel like you are part of these brands’ “communities?”

Now answer me this: If you don’t, why in the f—— world do you believe anyone else does?

The specific product and industry is important to keep in mind while selecting channels. Big consumer brands usually benefit the most from traditional channels. After all, Pepsi lost a lot of money when it moved all its ad spend from TV to social media. SaaS products, in one example, might be completely different. Again, the key is to test to see what works.

The rest of the marketing process

The 4 Ps, promotion mix, buyer personas, and channel research are only the strategic first-half of the marketing process. As I explained in my earlier essay on the marketing department of the future, the rest of the process consists of creating online and offline marketing collateral and content, transmitting it to the audience, and then auditing the results.

Examples of creatives

  • Direct marketing: producing sales collateral to give to prospects directly
  • Advertising: creating online and offline ads
  • Personal selling: designing presentations and webinars
  • Sales promotion: creating coupons, landing pages, and more
  • Publicity: writing online and offline by-lined articles or capturing a publicity stunt on video

Transmission & audit

The next step is to transmit the creatives to the audience over the selected online or offline channels.

Once a full marketing campaign has been executed, it’s time to audit the results. A company might find, for example, that a combination of offline advertising, online direct marketing, and a combination of online and offline publicity works the best. Or it could be something else entirely. The only way to know is to test.

A hypothetical example

Here are four examples—one hypothetical and three real ones—of the different ways that traditional and digital marketing strategies can be integrated.

One comment I left a few months ago on another Moz essay on creating demand for products was this:

Moz sells, in part, “SEO software.” Say SEOmoz (as it was called in the beginning) had been launched in 1995. There would be little keyword volume for “SEO software” for this reason: No one knows that “SEO” exists in the first place, so there would obviously be no demand for “SEO software” specifically. Moz would probably have had only one customer—Danny Sullivan. 🙂

So, in such a scenario, if I were to sell SEO software in 1995, I would first do a PR and advertising campaign to generate awareness of SEO in general and SEO software specifically. I’d bet that search volumes would increase in due time. Then, once people know that both things exist, then you can start to capture prospects and move them down the funnel via inbound marketing.

“Outbound marketing” and “inbound marketing” will always be needed because outbound marketing creates demand while inbound marketing fulfills demand. In my opinion, studies that purport to show that inbound marketing is always better than outbound marketing only tell part of the story. If traditional, offline advertising is so ineffective, they why do I still see thousands of ads everywhere I go every day?

Three real-world examples

An American pizza restaurant

As I once explained in a BrightonSEO talk (see a summary with slides on my website), a small pizza parlor in Philadelphia got the best results—a lot of brand awareness, hundreds of high-quality links, and thousands of Facebook “likes”—by thinking not about “SEO” or “social media marketing” or “content marketing,” but rather good, old-fashioned “publicity” within the promotion mix.

The local business—whether intentionally or not—got a lot of local news coverage by allowing people to “pay it forward” by purchasing slices to give to people in need. The news coverage snowballed into national coverage in the United States and the business owner appearing on the talk show “Ellen.”

The pizzeria did not produce one piece of “content,” or even have a blog at all. For most small, local businesses (especially restaurants), I’d invest in a good, creative publicist over “content marketing” any day. Still, it’s crucial that one’s marketing toolkit contain every potential strategy and tactic because different promotion mixes work best for different companies and industries.

Pizza Hut Israel

Yes, I’ve got a soft spot for pizza.

While I was writing this post here in Tel Aviv, I received this e-mail and saw this Facebook post:

(That’s Pizza Hut Israel selling a pizza with a crust made of bite-sized pieces filled with cream cheese.)

Now, what is this? It’s not “e-mail marketing” or “social media marketing.” Pizza Hut here had decided to do a sales promotion via the channels of e-mail and Facebook. It was the company deciding upon a certain promotion mix (likely for the reasons described above) and then choosing to execute that promotion over the channels of e-mail and social media.

Logz.io

Logz.io cofounders CEO Tomer Levy (left) and VP Product Asaf Yigal (right) at AWS re:Invent 2015

In another example, we at Logz.io offer predictive error detection in our ELK-as-a-service cloud platform for DevOps engineers, and we have found that the best results for us come from a combination of personal selling and publicity.

The personal selling is when we sponsor and speak at conferences for DevOps engineers and system administrators; the publicity is when we publish and then publicize informational articles and guides in major publications and on our website. (For example, our CEO, Tomer Levy, wrote about different open-source DevOps tools on DevOps.com, we’ve published a guide on how to deploy the ELK Stack, and I discussed how to use server log analysis for technical SEO here on Moz.)

Again, every company and industry is different. It’s important to test every possible promotion mix to see what delivers the greatest ROI for you. Within the marketing industry, there are many self-interested parties that advise one promotion mix or another. The best thing to do is to test everything and see for yourself. I hope these three examples will help to get you thinking.

Now, where’s SEO?

I’ve discussed my opinions at length in the other essays of mine to which I’ve linked here, so I will just summarize here so I will not always repeat myself: “SEO” is now technical and on-page optimization. The more that one understands traditional marketing, the more that one sees that (good) “off-page SEO” tactics are really just doing good traditional marketing. Almost any “off-page SEO” tactic that anyone can name is simply direct marketing, sales promotion, personal selling, advertising, or publicity by another name.

The reason it is important to understand this concept is that, as Google’s algorithm becomes smarter and thinks more and more like a human being, it’s becoming imperative to think more and more about building a brand among people over the long term rather than chasing an algorithm and directly trying to get high rankings in the short term. To help the community, my goal here is for SEOs to stop thinking so much about SEO specifically and to think more about marketing. After doing all of the needed technical and on-page SEO, the best results—higher rankings, greater backlinks, and more engagement—will come simply as by-products of building real brands that have a lot of authority and engagement.

Today, this is what will happen if you try to manipulate, outsmart, or otherwise chase Google’s algorithm:

Summary

If you’ve read this far, I’m glad that you found my thoughts to be interesting! I’ll leave you with this final idea:

How would you market yourself if the Internet didn’t exist? Answer that, and it’ll help your online marketing too.
— Samuel Scott (@samueljscott) August 25, 2015

Complete List of Resources:


Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

Continue reading →