As a direct result of Google switching to a secure connection (https) and encrypting user searches, we have been witnessing how an increasing amount of keyword data in Google Analytics fell under the (not provided) category.

While in November 2011 about 12% of search queries were ‘not provided’, the percentage rose to 50% by August 2013, 80% by December 2013 and is closing in on 100%. (source: http://www.notprovidedcount.com/)

Slowly, we reached the moment we were so terrified of, that moment when search data no longer made it back to us, creating a huge void in our analysis.

The tragic moment of 100% (not provided) Keywords

When (not provided) takes up the majority of your keyword list, you’re faced with a whole new set of challenges. You need to find a workaround for every task in your routine that was relying on organic keyword data.

The greatest impact this change had on SEO was the fact that it crippled your ability to measure, assess and report the effectiveness of your work to your clients or employers.

Beside this, without keyword data, you are no longer allowed to segment users by keywords directly within your web analytics software, branded vs. non-branded keyword trends can no longer be monitored. You are now faced with the impossibility to pinpoint new or trending search terms, that could potentially generate traffic for your website.

Can we find our way back to Not Provided search queries?

Luckily, these search queries are not completely hidden from us. There are other ways in which you can access organic search data for your websites:

• Google Webmaster Tools has a significant amount of keyword-specific data available, including keywords from encrypted search queries. By linking GWT with Google Analytics (GA) you can also access this data directly from GA.

• Google AdWords displays specific keyword data for paid campaigns (PPC), but you can easily rely on this information when setting up your organic campaigns also.

• Non-Google search engines still pass some keyword data back to webmasters. However, by March 31, 2014 all Yahoo properties will also switch to encrypted search, making search data unavailable. As for Bing, although it has already switched to secure search, it is not the default option for users, therefore only a small amount of searchers use it.

• Site searches also can give you a pretty good idea about the keywords your visitors are using when searching for your products or services. If you need help setting up site search for your website, this article gives great instructions for this.

For a more complex view over how different tools can provide you with a piece of the missing information, I recommend you to also check out these great articles:

• Google ‘(Not Provided)’ Keywords: 10 Ways to Get Organic Search Data

Can we regain the ability to report?

Although as SEO professionals we are used to mixing together the data we gather from multiple sources, these new keyword data sources that we are now forced to use bring something new to the table.

Thanks to (not provided) search queries, in order to still be able to prove to your clients that your optimization efforts paid off and translated into traffic and conversions, you’ll need to reconfigure your entire reporting flow.

Let’s break things down:

Step one – Gather and secure the data

Being the next best thing compared to what we used to have before the (not provided) era, Google Webmaster Tools will probably become your primary keyword data source now.

To efficiently use this data, you should first of all know that this information is only available in your GWT account for 90 days, so constant, timely, backup is required.

For this, GWT offers a direct ‘Download’ option, for both the raw data and the charts, either as CSV file or to your Google Drive. However, it lacks the ability to automate this task by schedule, therefore you can either perform it manually every three months or link your GWT account to an additional monitoring system to pull the information for you.

Keep in mind though that linking GWT with GA does not secure the data for more than the 90 day standard period.

Step two – Compiling reports

Since keyword data served multiple purposes in your reporting flow, there are several types of reports that have been affected by the (not provided) change. We’ll further discuss the three most common types of such reports:

• The evolution of organic keyword traffic

The metrics GWT provides for organic keywords are different than those found in GA, changing the way keyword traffic is estimated:

With only Impressions, Clicks, CTR and Average Position metrics available for each search term, the Clicks metric remains a sole indicator of keyword performance.

Although at first sight this might seem similar to the old traffic metric, you should note that the Clicks value is different from the actual Visits value. This has to do with the way Google counts them. For example, it is counted as a click but not a visit in cases like: when an user clicks your website in search results but closes the page before it’s fully loaded (counts as one click and no visits), when an user clicks your website twice in search results in a 30 minutes time frame (counts as two clicks but only one visit) or when users who don’t have JavaScript enabled in their browser visit your site from search results (GA tracks only visits from users with JavaScript enabled).

• Brand awareness monitoring (branded vs non-branded keywords)

Segmenting the branded from the non-branded traffic driven to your website has also changed, being restricted now only to the data provided by GWT. Even if you import the data from GWT to GA, you can’t create advanced segments for determining the branded traffic like you used to do before (not provided). Therefore, the only solutions remaining are that you either process the CSV exports manually or use a dedicated software to manage your keyword list.

Just about any type of spreadsheet should allow you to create advanced filters for segmenting the daily traffic (clicks) into branded and non-branded and get something somewhat similar to this:

• Identify poorly optimized terms and pages

Basically, by comparing the keywords that are actually bringing you traffic with their performance in SERPs you are able to identify profitable, but poorly optimized for, terms to further focus your efforts on. Although GWT’s reports provide you with relevant information for this task, you need to pay close attention to a few aspects.

First of all, note that the information you see in the Top Queries report is only relevant at website level:

“Impressions—the number of times any URL from your site appeared in search results viewed by a user, not including paid AdWords search impressions. Clicks—the number of clicks on your website URLs from a Google Search results page, not including clicks on paid AdWords search results. […] In the case where a query returns more than one URL from your site, the average position is based on the most prominent URL in the search results when only a low number of URLs from your site are displayed.”

(source: https://support.google.com/analytics/answer/1308626?hl=en)

What it means is that if you are struggling to promote a landing page on your site for a specific keyword, but your website index page for example also ranks for that term on a higher position, the information provided by GWT in the Top Queries report will not be relevant for evaluating the performance of your landing page. You will see the Average position of your Index page and the Clicks and Impressions of all your website pages ranking for that term.

If you wish to see this information tailored to each ranking URL, you need to click the desired query and access the detailed report. For a complete view, you need to download all detailed reports and merge them into a separate spreadsheet.

Secondly, the Average position is not just a general metric, calculated for all URLs of a website, but it is also the average of top positions:

“To calculate average position, we take into account the top ranking URL from your site for a particular query. For example, if Jane’s query returns your site as the #1 and #2 result, and David’s query returns your site in positions #2 and #7, your average top position would be 1.5.”

(source: https://support.google.com/webmasters/answer/35252?hl=en)

Therefore, the average position is only relevant for the utopian case when your website ranks for each keyword only with the landing page you are specifically optimizing and tracking. We all know that this rarely happens, and you will often find the Average position metric to be insufficient in your analyses.

Ideally you would want to use updated keyword rankings and search volumes added to the data retrieved from Google Website Tools all together in a report, to be able to identify, for example, a lower-ranking keyword that however amasses a large number of monthly searches but only a few actual clicks.

Conclusion

Although (not provided) seemed to leave us in the dark, in the end things are not that bad. Google Webmaster Tools is quite a formidable tool, that leveraged properly and paired with additional sources of data, can provide you with relevant information to answer all your needs. For further reading:

• SEMRUSH review : SEM Tool to research competitors keyword
• Fastest way to find long tail high paying Keyword with great AdSense CPC

But what about you? Have you developed any workarounds for the (not provided) apocalypse? I look forward to your stories in the comments section below.

This masterpiece is by Dana Loiz. If you would like to submit an original article, check our submission guidelines.