Add multilingual support

[azerupi]

Add support for multiple languages.

[FuGangqiang]

multiple languages for document?

[azerupi]

Yes, I think Gitbook does support something like that.

Instead of having the markdown files directly in the source folder you would have some sub folders like this:

src/
├── de
├── en
└── fr

And their would be an easy way to change the language in the rendered book.

It's definitely something I would like to add, but it's not the highest priority at the moment

[azerupi]

Multiple designs possible:

• One SUMMARY.md to rule them all

  pros:

  • Changes in structure are reflected in all languages immediately
  • 1 to 1 mapping from pages from one language to another, would allow changing the language of the page directly from a menu button

  cons:

  • If one language is lagging behind it's going to get ugly

• One SUMMARY.md for every language

  pros:

  • Every language can have it's own pace

  cons:

  • Does not push all languages to be up to date / coherent
  • No 1 to 1 mapping guarantee and thus not possible to toggle the language from a page without having the risk that the page does not exist in the other language

[mkpankov]

I don't think one SUMMARY.md for everything is a good idea. I consider consistency within translated version more important than consistency with original. Otherwise, we can easily start having broken links because upstream renamed some chapter and translation didn't, yet. I believe a book that has no broken links is the minimum standard.

Also, I don't support the idea of "pushing" to be up-to-date. AFAIK, translations (not only ours) are done by enthusiasts and it's not always possible to keep up at all times.

Moreover, 1 to 1 mapping of pages doesn't look straightforward to me, even in case there's single SUMMARY. Words have different length in different languages, and in Russian translation we consistently have sentences that are noticeably longer than original. But I'd love to have it so that one click can show the same point in text in original language.

I think this can be handled by tracking 1-to-1 mapping of paragraphs - sections aka markdown files are too big. Paragraphs also seem a good candidate because sentences get paraphrased and reordered sometimes, but the paragraphs stay in same order and have same gist.

[azerupi]

Thanks for the input! I really appreciate the feedback :)

Otherwise, we can easily start having broken links because upstream renamed some chapter and translation didn't, yet. I believe a book that has no broken links is the minimum standard.

Moreover, 1 to 1 mapping of pages doesn't look straightforward to me

When I am talking about 1 to 1 mapping I am talking about page to page mapping, not sentence to sentence (that would be insane 😉).

Let's take a hypothetical situation with the Rust book. Let's say I am reading a blog post and it references some chapter in the Rust book, for example the chapter about ownership. But English is not my main language and it would be a lot easier to understand the chapter in my native language. If we have 1 to 1 mapping on page / chapter level the user could then select his language (if it is supported) from a dropdown menu and he would land on the exact same page in his chosen language.

However for this to work correctly we need a guarantee that every page in one language has an equivalent page in the other language. If you allow a different SUMMARY.md per language there is no way to know what pages are equivalent if any equivalent page even exists at all.

Also, I don't support the idea of "pushing" to be up-to-date. AFAIK, translations (not only ours) are done by enthusiasts and it's not always possible to keep up at all times.

Of course, I totally agree with you. But the SUMMARY.md is only about structure, so what order the chapters come in, not the content.

If there is one SUMMARY.md for all languages I think it will only cause trouble if:

1. New chapters get added, as equivalent chapter in other languages will just be blank until they are translated
2. The markdown files get renamed, this should not happen often when it does it is not difficult to rename the files accordingly for every language
3. A reorderering of the chapters where the continuity of the content is broken. This too should not happen often, but it's more challenging to fix as it requires the translators to translate the text that changed

To be honest, once a book has it's definitive structure the SUMMARY.md is not likely to change often unless there is a major rewrite being done.

I think both designs have advantages and drawbacks, we need to figure out which one we want / need the most.

---

Idea for Rust book workflow when translations are in tree

When / if translations are moved into the official repository we could create a more elaborate pull request process. This is only an idea, it may be flawed 😉

When a pull request is made that contain changes that need translation (e.g. not typos) we could wait to merge the pull request until translations have been made for all officially supported languages.

The pull request could track what translations have been made using a check list like this:

• Russian
• French
• German

Once all the translations are ready the pull request is merged in.
Officially supported languages could be languages with a minimum number of "official" maintainers.

This would add a little / lot of overhead for the english version but it would solve the two big issues with translations.

1. Translations would always be up to date!
2. This is probably the easiest way to track changes

There may be organizational problems I haven't considered though. @steveklabnik

[steveklabnik]

The biggest problem with blocking English changes to non-English changes is that I am paid for my work, but others are not. This places a big burden on them; I'm gonna want to land changes ASAP, and that's not fair to people who can't do this as a day job.

[azerupi]

That's true, didn't think of that.
It could still be applied without blocking the English changes? Just for tracking. Not sure if it's worth the overhead though.

Anyways, do you have a preference for any of the two design choices (one vs. multiple SUMMARY.md)?

[steveklabnik]

I think I prefer a single for the reasons you've stated, but since I'm not doing the translations themselves, I don't think my opinions matter much :)

And yeah, tracking might be different/better than actually blocking on them landing.

[mkpankov]

When I am talking about 1 to 1 mapping I am talking about page to page mapping, not sentence to sentence (that would be insane 😉).

Ok, I think what I was trying to say but couldn't get across is this: page-to-page mapping isn't enough for printed versions, as same pages will have different content. And if by page you meant a web page, that is not enough either. Some sections (pages) are tens of screens long, and to provide smooth transition from one version to another we should track smaller units than entire files (web pages).

I originally thought you were talking about printed pages and written the following, but I'm not sure now. For printed versions, depending on length of the section and sentence-length difference with the original, this can very from "I see not the beginning of the paragraph that talks about Foo feature, but the end" to "I don't see the paragraph that talks about Foo feature on screen at all", when linked to "page 83 of PDF".

So let's clarify the terms before continuing as apparently I misunderstood something 😄

[azerupi]

Ok yes, I will try to do my best to explain what I envision:

So in this issue I am not at all talking about tracking any changes for translations, only about how to support multiple languages in the same folder / book.

Before I continue, let's explain what the SUMMARy.md does exactly.

When you render the book (mdbook build) it is going to search for the SUMMARY.md and parse it. The SUMMARY dictates

• The Order of the chapters.
• The names of the chapters.
• The markdown file corresponding to each chapter.

That is the "only" information we get from the SUMMARY.md

If we want to support multiple languages for one book, there are two possible designs (that I thought off):

• One SUMMARY.md at the root of the source directory that will be used for all languages.
• SUMMARY.md for every language

Let's see both in more details.

One SUMMARY.md for all languages

Consider this SUMMARY.md for a book:

# Summary

- [hello world](hello-world.md)
- [second chapter](second-chapter.md)

and this directory structure:

├── book
└── src
    ├── en
    │   ├── hello-world.md
    │   └── second-chapter.md
    ├── fr
    │   ├── hello-world.md
    │   └── second-chapter.md
    ├── ru
    │   ├── hello-world.md
    │   └── second-chapter.md
    └── SUMMARY.md

As you can see here, every language has the same markdown files defined in the global SUMMARY.md. This means that the "hello world" chapter has a corresponding page in every language! (1 to 1 mapping)

Advantages

Having a guarantee that every chapter in one language has a corresponding chapter in another language gives us the possibility to change the language from any chapter and land on that same chapter in the other language.

Example: I am reading the "borrowing" chapter of the Rust book. I want to see that same chapter in French. I just select "French" from the dropdown button in the menu-bar and I will land on the French version of the chapter.

Drawbacks

When the SUMMARY.md is modified it can cause some consistency problems in the translations because changes in the SUMMARY.md
will be reflected immediately in all languages. However, changes in the SUMMARY.md should be relatively rare once the book has found it's "final" structure.

Problems that could occur:

• Chapter is moved: When a chapter is moved (the order of the chapters is rearenged) it could cause problems with text flow.
• markdown file is renamed: When a markdown file is renamed it should be renamed in all languages and in all the references to it. This should not be too big of a problem.
• New chapter is added: When a new chapter is added it will appear blank in the other languages until it's translated.

Content is not modified by the SUMMARY.md so any of the designs here is not going to cause any trouble with the content if the SUMMARY.md is modified.

Another drawback is that I am not sure yet how translations will give a translation for the chapter titles in the sidebar (SUMMARY.md). Maybe just take the first heading from the corresponding markdown file?

One SUMMARY.md for EVERY language

Let's consider this directory structure:

├── book
└── src
    ├── en
    │   ├── hello-world.md
    │   ├── second-chapter.md
    │   └── SUMMARY.md
    ├── fr
    │   ├── hello-world.md
    │   └── SUMMARY.md
    └── ru
        ├── hello-world.md
        ├── second-chapter.md
        └── SUMMARY.md

As you can see here, every language has it's own SUMMARY.md and thus can define the order of their chapters and the markdown files as they wish.

There is absolutely no more guarantee that the French version contains the same chapters as the English version. No 1 to 1 mapping. Essentially every language is its own separate book, they could have exactly the same structure or they could have totally different chapters. There is no way for the program to know that.

It is thus impossible to change the language from a chapter. You would have navigate to the French version manually and search for the chapter you were reading if it exists in the French version at all!

Advantages

Translations have a lot more freedom, but this can also be seen as a drawback. Translations do not need to have the same structure, so when the SUMMARY.md is changed in the English version, absolutely nothing is going to change in the other languages. Every change in the translations has to be done manually.

Drawbacks

There is no guarantee that a chapter in one language as an equivalent in another language.(No 1 to 1 mapping) The program can not know what chapters are equivalent in the different languages and it would thus be impossible to change the language from a chapter to land on the same chapter in the other language.

---

I hope this made it more clear, if there is still something you don't understand I can elaborate more on some specific area. 😉

EDIT: A little quote from a response I made on Rust's internals forum:

And to be honest, if you have different TOCs you essentially have different books. There is little gain to support that, other than being able to group all the translations in one directory and build them in one go.

You can already group the multiple translations in one directory as different books each with it's own SUMMARY.md and book.json and if you configure the source and destination directories correctly there should be minimum trouble to integrate with automatic deployment scripts etc.