Integrate with Crowdin localization project

[bvaughn]

Migrated from facebook/react/issues/8063

Coordinate with docs translators via @ericnakagawa

This issue involves multiple steps:

• Integrate with Crowdin to download localized content
• Generate localized URLs using said content
• Automatically sync English content changes from GitHub to Crowdin (see this comment and this repo)
• Maintain backwards compatibility for all current (non-localized, English) links
• Add localization toggle to header
• Update the "Edit this page" link to point to GitHub for English pages and Crowdin for other languages.

Integrate with Crowdin

This could be done via a custom Gatsby plug-in. We've already written a couple. (It would be nice to work with @KyleAMathews to share this plug-in back with Gatsby so that other users could benefit from it as well.)

If we run into any performance problems, @ericnakagawa has volunteered to work with Netlify and Crowdin to simplify or speed up this syncing process. Ideally this sync process will be fast enough to allow Netlify's PR previews to also be used to test localized builds of the site.

Misc thoughts and questions

• In the case of non-localized content, Crowdin serves English as fallback.
• How can site consume list of active locales? Is there a Crowdin API?
• How does this impact urls (eg /docs/<file>, /docs/<lang>/<file>)?

[KyleAMathews]

Happy to help create a Gatsby source plugin for Crowdin! That'd be really powerful. And a lot of people would use it including sooner than later gatsbyjs.org

From a bit of research, it looks like it'd be pretty easy to pull data using their API:

• https://support.crowdin.com/api/download/
• Guide for creating a source plugin https://www.gatsbyjs.org/docs/create-source-plugin/ — this guide is a bit wordy right now. When someone wants to tackle this, I could take a pass at improving the guide + adding a mini-tutorial on creating source plugins.

I spent a bunch of time adding localization support to https://www.gatsbyjs.org/packages/gatsby-source-contentful/ as well so have some opinions on this. See the example site for Contentful + its source

[bvaughn]

Fantastic 😄 There's a good chance we'll end up collaborating on this soon then.

[Daniel15]

Netlify also has a way of doing redirects based on Accept-Language header and GeoIP. For example, if someone from Italy (or someone with their browser or OS set to request Italian content) visits the home page and they don't have a language cookie already set, we can automatically redirect to the Italian version. We use this on the Yarn site - You can look at how Yarn does it for inspiration

How does this impact urls (eg /docs/, /docs//)?

I'd suggest /<lang>/ (eg. /<lang>/docs/<file>), this is consistent with other Facebook projects that use localization:
https://yarnpkg.com/en/docs/getting-started
https://flow.org/en/docs/getting-started/

If we run into any performance problems, @ericnakagawa has volunteered to work with Netlify and Crowdin to simplify or speed up this syncing process. Ideally this sync process will be fast enough to allow Netlify's PR previews to also be used to test localized builds of the site.

We have perf issues with the Yarn site, but a lot of it is because we're using Jekyll. Most of the build time is actually the Jekyll build time rather than CrowdIn. We had a prototype using Hugo that was much faster. I don't think CrowdIn takes a significant amount of time there.

From a bit of research, it looks like it'd be pretty easy to pull data using their API:

Would the React site actually need a custom API integration? It could be useful, but CrowdIn has a CLI tool (https://support.crowdin.com/cli-tool/) which we might be able to use with less effort. For Yarn we just run crowdin-cli download -b master to download all the localized files as part of the build (in that case, there's a bunch of Markdown files as well as some yml files for language strings). Then you just build the site like normal and everything just works, as long as the files are in the correct locations.

[bvaughn]

How does this impact urls (eg /docs/, /docs//)?

I'd suggest // (eg. //docs/), this is consistent with other Facebook projects that use localization:

Don't have time to fully digest this response now- but wanted to mention that we should ensure whatever change we make does not break pre-existing links (like how links broke for Jest).

[KyleAMathews]

Would the React site actually need a custom API integration

It doesn't need it but it seems simpler perhaps to just query the data through graphql instead of writing it out first then pulling it in.

But yeah, definitely more straightforward to just use the cli.

Would you checkin the translated content?

[Daniel15]

but wanted to mention that we should ensure whatever change we make does not break pre-existing links

Jest is limited in what it can do, because it's using GitHub Pages. With Netlify, you can generate some redirects as part of the build. Just need to redirect "raw" URLs (/docs/foo) to the localized URL (/en/docs/foo), either with /en/ hard-coded or using Netlify's language redirects that I mentioned earlier (so it'd redirect /docs/foo to /<user's preferred language>/docs/foo).

Check it out with Yarn: https://yarnpkg.com/docs/getting-started redirects to the right place. :)

Would you checkin the translated content?

We don't for Yarn. It's pulled in as part of the build process for the site. Since CrowdIn tracks the history for the translations, we didn't really need to also have Git track it.

Some sites do commit the translated files though. Up to you - Either way is fine.

[bvaughn]

Would you checkin the translated content?

No

[bvaughn]

or using Netlify's language redirects that I mentioned earlier (so it'd redirect /docs/foo to /<user's preferred language>/docs/foo).

Cool! 👍

[Daniel15]

Here's how the Yarn site builds all the redirects from non-localized URLs to localized URLs: https://github.com/yarnpkg/website/blob/master/_redirects#L48-L70 (this would also be how you handle your redirects from 'old' URLs to new localized URLs). Unfortunately the Yarn one is super difficult to understand because it's using the Jekyll templating syntax. Somehow @thejameskyle got it working though. You could likely make a nicer JS script to do the same thing for the React site.

The resulting redirects file looks something like this:

/docs/cli/add/  /en/docs/cli/add/  302  Language=en
/docs/cli/add/  /fr/docs/cli/add/  302  Language=fr
/docs/cli/add/  /pt-BR/docs/cli/add/  302  Language=pt-br
/docs/cli/add/  /zh-Hans/docs/cli/add/  302  Language=zh-cn

Here's the entire language redirects section: https://gist.github.com/Daniel15/d84b39e8a260ee2a7d627751d015181b

Docs: https://www.netlify.com/docs/redirects/#geoip-and-language-based-redirects

Might be able to use a regex or placeholders or something rather than having to generate all the redirects like that.

[KyleAMathews]

@bvaughn wrote a nice redirect API for Gatsby + added it to gatsby-plugin-netlify so this setting up redirects would be pretty easy :-)