In the past, websites wanting to show content to international users would have separate URLs for each locale. Now, more sites are following the trend of dynamically changing page content. The page changes its content based on a web visitor’s country (IP address) or language (Accept-Language HTTP Header).
Google recently admitted it has issues when crawling, finding and indexing these locale-adaptive pages. Because Googlebot crawler’s default IP address is in the US, Google may not crawl, index or rank this international content.
To resolve this issue, Google announced improvements in January 2015 to the way Googlebot crawls locale-adaptive websites. Googlebot now uses geo-distributed crawling and language-dependent crawling. This means Googlebot can now use IP addresses appearing to come from other countries. Googlebot can also crawl with an Accept-Language HTTP header in the request. The update ensures the version of a page intended for a particular country or language speaker is presented in organic search results.
If you’ve worked hard to create a locale-adaptive site, be sure you correctly configure your site for the Googlebot crawler. Merkle provides a great free Locale-Adaptive Pages Testing Tool. This tool allows you to specify the languages you would like to check and the browser user-agent (Google or Bing). Once you enter your information and click run, you get immediate results. The results determine whether you have the correction configuration of your site. This allows you to make the necessary changes to rank and index your site. Google still recommends using separate URLs for each locale. For locale-adaptive sites that cannot use separate URLs, as long as you properly configure your site, you can now rank in relevant countries.
For more information on Locale-aware crawling by Googlebot, visit Google’s Help Forum.
