Google Gives Update On Crawling JavaScript Sites & Progressive Web Apps

Mar 7, 2016

by Barry Schwartz

John Mueller of Google posted a detailed update on how Google currently handles JavaScript sites (including AJAX) and Progressive Web Apps in their index, crawling, and ranking. He posted this on Google+ and titled it "An update (March 2016) on the current state & recommendations for JavaScript sites / Progressive Web Apps in Google Search.

I am just bringing it more daylight by referencing it over here:

• Don't cloak to Googlebot. Use "feature detection" & "progressive enhancement" [2] techniques to make your content available to all users. Avoid redirecting to an "unsupported browser" page. Consider using a polyfill or other safe fallback where needed. The features Googlebot currently doesn't support include Service Workers, the Fetch API, Promises, and requestAnimationFrame.
• Use rel=canonical [3] when serving content from multiple URLs is required.
• Avoid the AJAX-Crawling scheme on new sites. Consider migrating old sites that use this scheme soon. Remember to remove "meta fragment" tags when migrating. Don't use a "meta fragment" tag if the "escaped fragment" URL doesn't serve fully rendered content. [4]
• Avoid using "#" in URLs (outside of "#!"). Googlebot rarely indexes URLs with "#" in them. Use "normal" URLs with path/filename/query-parameters instead, consider using the History API for navigation.
• Use Search Console's Fetch and Render tool [5] to test how Googlebot sees your pages. Note that this tool doesn't support "#!" or "#" URLs.
• Ensure that all required resources (including JavaScript files / frameworks, server responses, 3rd-party APIs, etc) aren't blocked by robots.txt. The Fetch and Render tool will list blocked resources discovered. If resources are uncontrollably blocked by robots.txt (e.g., 3rd-party APIs) or otherwise temporarily unavailable, ensure that your client-side code fails gracefully.
• Limit the number of embedded resources, in particular the number of JavaScript files and server responses required to render your page. A high number of required URLs can result in timeouts & rendering without these resources being available (e.g., some JavaScript files might not be loaded). Use reasonable HTTP caching directives.
• Google supports the use of JavaScript to provide titles, description & robots meta tags, structured data, and other meta-data. When using AMP, the AMP HTML page must be static as required by the spec, but the associated web page can be built using JS/PWA techniques. Remember to use a sitemap file with correct "lastmod" dates for signaling changes on your website.
• Finally, keep in mind that other search engines and web services accessing your content might not support JavaScript at all, or might support a different subset.

John ended by saying:

Looking at this list, none of these recommendations are completely new & limited to today -- and they'll continue to be valid for foreseeable future. Working with modern JavaScript frameworks for search can be a bit intimidating at first, but they open up some really neat possibilities to make fast & awesome sites!

Forum discussion at Google+.

Barry Schwartz is the CEO of RustyBrick, a New York Web service firm specializing in customized online technology that helps companies decrease costs and increase sales. Barry is also the founder of the Search Engine Roundtable and the News Editor of Search Engine Land. He is well known & respected for his expertise in the search marketing industry. Barry graduated from the City University of New York and lives with his family in the NYC region. You can follow Barry on Twitter at @rustybrick or on Google + and read his full bio over here.

Disclaimer: Digital Propeller does Not own this content please give all credit to the original author (Barry Schwartz) and site owner (https://www.seroundtable.com/google-on-crawling-javascript-sites-progressive-web-apps-21737.html).