Blender: Going Locale #68588

Issue

Blender is used by artist around the world - [x] however, our efforts do not meet the demand for providing localization for the software and documentation.
The current workflow for both Blender itself and the user manaul are a bit convoluted. For both projects on boarding is difficult, there are no clear steps for how a new translator should get commit writes.
The only translators that we get are the ones that are either brave enough to make the changes and submit patches are create some sort of task/email stating they would like to help.
Communication among translators for a specific language are very limited and it is difficult for multiple translators to work together.

Proposal

Weblate

Blender has grown quite actively over the years and our current method of dealing with translations are not enough to keep up with such a large project.
Most large projects offload this work to an online localization platform. The issue with this for the Blender Foundation is that most of these make use of 3rd party websites.
However, I did find one service, called Weblate - [x], that is developed on open source principles can support custom installs.

Weblate is an online localization service with tools to manage and translation projects using an online editor.
The benefits of such a platform are, online editing of translations, communication features to individual language groups,
a translation reviewal system and the ability to easily be able to request access to translations without having to know advanced tools like SVN/Git and sphinx.
A full list of features are explained here - [x]. Weblate is used by over a thousand projects including, OpenSuse and Godot Game Engine [4].

For us, the way I see this running is to have weblate installed at https://translate.blender.org which would make it super easy and convenient for translators to get involved in translating.
Weblate integrates directly with version control software, both Git and SVN. For the main translations the use of the SVN repository would be eliminated as Weblate would work directly with the main Git repository.
Changes will be made directly through weblate itself and these changes are then be periodically (or automatically via a webhook) pulled into the repository.
In essence the only thing that should be interacting with the Git repository is Weblate itself. This raises the issue of a workflow change for translators as translations should be translated in the web interface.
However, Weblate has a feature where translators can download translation from Weblate and use a gettext editor workflow [5].

For the user manual things would work in a similar way, except for using Git we can stick to using our SVN repository (although switching to Git would be extremely useful).
There is an issue with the user manual where Weblate uses a single translation file for each language while Sphinx uses multiple translation files per language.
While we can still make this work it introduces a middle step to merge files to a format Weblate can use. This is simple by using msgmerge - [x] and godot has worked around this problem [7].

All in all, migrating to a more centralized system like this should improve both the quantity and quality of translations by making translation onboarding easier along with providing a simple online based workflow.

{F7712287, width=50%}

Improved Communication

Another issue that we have now is poor communication our current efforts include the old bf-translations-dev mailing list which only sees a few stray emails a year.
There are a couple of things that I would like to do here:

- Create a Translations category on devtalk with sub language topics
- Create a #translations channel on blender.chat

Implementation

Timeline

We want to get this system working sometime around 2.82 which is set to release in 3-4 months.
Implementation details of this proposal will happen immediately has we need time to make this changes and iron out the new workflow.

Tasks

Some initial tasks to get done that I can think of off the top of my head to get this working:

- Migrate repositories to Git (not required but helpful)
- - Archive old repos and restrict commit access
- Update the user manual locale structure to use single translation files (#83212)
- Get weblate installed on the server
- Import the repositories setting up projects and components
- Update existing translation guides on wiki and user manual)

Questions

There are some things we might want to consider related to how we configure weblate / the workflow we choose.

Weblate allows new languages to be added on the fly. Do we want to do this or make users request new languages and languages can be added manually.
Weblate can automatically commit changes to the repositories. Do we wan to do this or manually merge changes made in weblate into the repositories.

# Issue Blender is used by artist around the world - [x] however, our efforts do not meet the demand for providing localization for the software and documentation. The current workflow for both Blender itself and the user manaul are a bit convoluted. For both projects on boarding is difficult, there are no clear steps for how a new translator should get commit writes. The only translators that we get are the ones that are either brave enough to make the changes and submit patches are create some sort of task/email stating they would like to help. Communication among translators for a specific language are very limited and it is difficult for multiple translators to work together. # Proposal ## Weblate Blender has grown quite actively over the years and our current method of dealing with translations are not enough to keep up with such a large project. Most large projects offload this work to an online localization platform. The issue with this for the Blender Foundation is that most of these make use of 3rd party websites. However, I did find one service, called Weblate - [x], that is developed on open source principles can support custom installs. Weblate is an online localization service with tools to manage and translation projects using an online editor. The benefits of such a platform are, online editing of translations, communication features to individual language groups, a translation reviewal system and the ability to easily be able to request access to translations without having to know advanced tools like SVN/Git and sphinx. A full list of features are explained here - [x]. Weblate is used by over a thousand projects including, OpenSuse and Godot Game Engine [4]. For us, the way I see this running is to have weblate installed at https://translate.blender.org which would make it super easy and convenient for translators to get involved in translating. Weblate integrates directly with version control software, both Git and SVN. For the main translations the use of the SVN repository would be eliminated as Weblate would work directly with the main Git repository. Changes will be made directly through weblate itself and these changes are then be periodically (or automatically via a webhook) pulled into the repository. In essence the only thing that should be interacting with the Git repository is Weblate itself. This raises the issue of a workflow change for translators as translations should be translated in the web interface. However, Weblate has a feature where translators can download translation from Weblate and use a gettext editor workflow [5]. For the user manual things would work in a similar way, except for using Git we can stick to using our SVN repository (although switching to Git would be extremely useful). There is an issue with the user manual where Weblate uses a single translation file for each language while Sphinx uses multiple translation files per language. While we can still make this work it introduces a middle step to merge files to a format Weblate can use. This is simple by using msgmerge - [x] and godot has worked around this problem [7]. All in all, migrating to a more centralized system like this should improve both the quantity and quality of translations by making translation onboarding easier along with providing a simple online based workflow. {[F7712287](https://archive.blender.org/developer/F7712287/Screenshot_from_2019-08-31_16-30-48.png), width=50%} ## Improved Communication Another issue that we have now is poor communication our current efforts include the old `bf-translations-dev` mailing list which only sees a few stray emails a year. There are a couple of things that I would like to do here: - - [ ] Create a Translations category on devtalk with sub language topics - - [x] Create a #translations channel on blender.chat # Implementation ## Timeline We want to get this system working sometime around 2.82 which is set to release in 3-4 months. Implementation details of this proposal will happen immediately has we need time to make this changes and iron out the new workflow. ## Tasks Some initial tasks to get done that I can think of off the top of my head to get this working: - - [ ] Migrate repositories to Git (not required but helpful) - - [ ] Archive old repos and restrict commit access - - [x] Update the user manual locale structure to use single translation files (#83212) - - [ ] Get weblate installed on the server - - [ ] Import the repositories setting up projects and components - - [ ] Update existing translation guides on wiki and user manual) ## Questions There are some things we might want to consider related to how we configure weblate / the workflow we choose. - Weblate allows new languages to be added on the fly. Do we want to do this or make users request new languages and languages can be added manually. - Weblate can automatically commit changes to the repositories. Do we wan to do this or manually merge changes made in weblate into the repositories. ----------------------- 1. https://www.blender.org/wp-content/uploads/2013/08/Screen-Shot-2017-08-28-at-14.37.14.png 2. https://weblate.org/ 3. https://weblate.org/en/features/ 4. https://hosted.weblate.org/projects/godot-engine/godot-docs/ 5. https://docs.weblate.org/en/latest/user/files.html 6. https://www.gnu.org/software/gettext/manual/html_node/msgmerge-Invocation.html 7. https://github.com/godotengine/godot-docs-l10n

Added subscribers: @Blendify, @jesterking, @brecht, @mont29

Added subscriber: @GabrielGazzan

Hi @Blendify ,
I'm the translator of the UI into Spanish (one of the few languages that has been consistently maintained throughout these years), I've handed it alone since 2011, and a couple of times people tried to jump onboard to help, but unfortunately they ultimately never did any significant contribution, for different reasons I think.
I understand what you are looking for with these changes, and it could be a good thing, if done properly.

What I wouldn't like is to lose in the process the simplicity of the actual system, I mean, if I can keep doing it without having to deal with a tedious bureaucracy or endless discussions about "what is the right term for X feature name", then thumbs up! and we'll keep on rolling as always. But if, on the other hand, the fact of having lowered the entry barrier attracts lots of people with little experience either in translating things, with technology in general, or even with Blender itself, etc... then I think it'd be inevitable that I will lose the guts to continue doing it... one way or another.

I stress, this is not to say that I don't like the idea, and for sure I understand the purpose behind it, but the task of translating Blender itself (for free) is worth it, if I can focus myself just on doing it for the sake of the community and don't having to lose energy on silly and useless political discussions in the process.
I really like team work, but there should be a smart way of implementing it for these tasks.
In this regard, I think some measures should be in place to assure that the people jumping in will have a certain minimum level of expertise to really be able to "sit around that table" and prove to be a positive force in helping to improve what we currently have.
For example, I also translate the UI for some other free projects like OpenToonz and Kdenlive, and in KDE (to whom Kdenlive belongs to) when someone comes in with the intention to join the translation team, the first thing they do is assigning him/her a small project, like a way of starting to deal with the mechanism of translation itself, and with all other things around translating software, and I think it works pretty good indeed.
...that kind of things, mainly.

I hope this new initiative works great!
Cheers

Hi @Blendify , I'm the translator of the UI into Spanish (one of the few languages that has been consistently maintained throughout these years), I've handed it alone since 2011, and a couple of times people tried to jump onboard to help, but unfortunately they ultimately never did any significant contribution, for different reasons I think. I understand what you are looking for with these changes, and it could be a good thing, if done properly. What I wouldn't like is to lose in the process the simplicity of the actual system, I mean, if I can keep doing it without having to deal with a tedious bureaucracy or endless discussions about "what is the right term for X feature name", then thumbs up! and we'll keep on rolling as always. But if, on the other hand, the fact of having lowered the entry barrier attracts lots of people with little experience either in translating things, with technology in general, or even with Blender itself, etc... then I think it'd be inevitable that I will lose the guts to continue doing it... one way or another. I stress, this is not to say that I don't like the idea, and for sure I understand the purpose behind it, but the task of translating Blender itself (for free) is worth it, if I can focus myself just on doing it for the sake of the community and don't having to lose energy on silly and useless political discussions in the process. I really like team work, but there should be a smart way of implementing it for these tasks. In this regard, I think some measures should be in place to assure that the people jumping in will have a certain minimum level of expertise to really be able to "sit around that table" and prove to be a positive force in helping to improve what we currently have. For example, I also translate the UI for some other free projects like OpenToonz and Kdenlive, and in KDE (to whom Kdenlive belongs to) when someone comes in with the intention to join the translation team, the first thing they do is assigning him/her a small project, like a way of starting to deal with the mechanism of translation itself, and with all other things around translating software, and I think it works pretty good indeed. ...that kind of things, mainly. I hope this new initiative works great! Cheers

Added subscriber: @ChristopherAnderssarian

What's the scope of this? Everything on the blender.org domain (store, cloud, dev wiki, ect...), something in between or just the Blender UI and Manual*(for now)//?

Or another way of asking: Is anything off the table?

What's the scope of this? Everything on the [blender.org ](https:*blender.org) domain (store, cloud, dev wiki, ect...), something in between or just the Blender UI and Manual*(for now)//? Or another way of asking: *Is anything off the table?*

Note that you cannot really get rid of the svn repo for Blender UI translations, as those stored in GIT are fully stripped to be as small as possible, they are essentially useless as working translation material...

@GabrielGazzan Thank you for the response, part of my goal here is to get feedback from current translators.

If I can keep doing it without having to deal with a tedious bureaucracy or endless discussions about "what is the right term for X feature name", then thumbs up!

One of the benefits of using a localization platform such as this is that we can adopt a glossary - [x] for each project language, which should help solve this very issue.

But if, on the other hand, the fact of having lowered the entry barrier attracts lots of people with little experience either in translating things, with technology in general, or even with Blender itself, etc... then I think it'd be inevitable that I will lose the guts to continue doing it... one way or another.

Another benefit of such a system is to add a better mechanism for translation review - [x]. Currently reviews are very hard as trying to do patch reviews for translations is extremely tedious and time consuming, not to mention that we do not have enough active contributes to see multiple people working together.

For example, I also translate the UI for some other free projects like OpenToonz and Kdenlive, and in KDE (to whom Kdenlive belongs to) when someone comes in with the intention to join the translation team, the first thing they do is assigning him/her a small project, like a way of starting to deal with the mechanism of translation itself, and with all other things around translating software, and I think it works pretty good indeed.

This sounds like a good idea for something we can improve on for all areas of contribution in Blender. For Blender itself we have the #quick_hacks project and we can incorporate something similar for translations and even the manual itself.
Some of these quick hacks could also just be routine tasks. Other projects use some sort of "Help wanted" tag for projects that can easily be assigned to new contributors.
I think this would be something nice to look into and that @jesterking in particular could help guide the foundation towards.

In #68588#752032, @ChristopherAnderssarian wrote:
What's the scope of this? Everything on the blender.org domain (store, cloud, dev wiki, ect...), something in between or just the Blender UI and Manual*(for now)//?

Or another way of asking: Is anything off the table?

For now I would like to limit this to only the Blender UI and the User Manual. In the future it would be nice to localize other services. But I would rather us focus and refine the workflow for what we already have.

I would say somethings should be off the table though, mainly, API docs and Developer Wiki as anyone who is involved with coding should be an English speakers.

In #68588#752075, @mont29 wrote:
Note that you cannot really get rid of the svn repo for Blender UI translations, as those stored in GIT are fully stripped to be as small as possible, they are essentially useless as working translation material...

What is the reason for this? Is this to reduce the size of the generated Machine Object files? If so cant this be consider a part of the build process?

@GabrielGazzan Thank you for the response, part of my goal here is to get feedback from current translators. > If I can keep doing it without having to deal with a tedious bureaucracy or endless discussions about "what is the right term for X feature name", then thumbs up! One of the benefits of using a localization platform such as this is that we can adopt a glossary - [x] for each project language, which should help solve this very issue. > But if, on the other hand, the fact of having lowered the entry barrier attracts lots of people with little experience either in translating things, with technology in general, or even with Blender itself, etc... then I think it'd be inevitable that I will lose the guts to continue doing it... one way or another. Another benefit of such a system is to add a better mechanism for translation review - [x]. Currently reviews are very hard as trying to do patch reviews for translations is extremely tedious and time consuming, not to mention that we do not have enough active contributes to see multiple people working together. > For example, I also translate the UI for some other free projects like OpenToonz and Kdenlive, and in KDE (to whom Kdenlive belongs to) when someone comes in with the intention to join the translation team, the first thing they do is assigning him/her a small project, like a way of starting to deal with the mechanism of translation itself, and with all other things around translating software, and I think it works pretty good indeed. This sounds like a good idea for something we can improve on for all areas of contribution in Blender. For Blender itself we have the #quick_hacks project and we can incorporate something similar for translations and even the manual itself. Some of these quick hacks could also just be routine tasks. Other projects use some sort of "Help wanted" tag for projects that can easily be assigned to new contributors. I think this would be something nice to look into and that @jesterking in particular could help guide the foundation towards. 1. https://docs.weblate.org/en/latest/user/translating.html#glossary 2. https://docs.weblate.org/en/latest/workflows.html#peer-review ------------- > In #68588#752032, @ChristopherAnderssarian wrote: > What's the scope of this? Everything on the [blender.org ](https:*blender.org) domain (store, cloud, dev wiki, ect...), something in between or just the Blender UI and Manual*(for now)//? > > Or another way of asking: *Is anything off the table?* For now I would like to limit this to only the Blender UI and the User Manual. In the future it would be nice to localize other services. But I would rather us focus and refine the workflow for what we already have. I would say somethings should be off the table though, mainly, API docs and Developer Wiki as anyone who is involved with coding should be an English speakers. > In #68588#752075, @mont29 wrote: > Note that you cannot really get rid of the svn repo for Blender UI translations, as those stored in GIT are fully stripped to be as small as possible, they are essentially useless as working translation material... What is the reason for this? Is this to reduce the size of the generated Machine Object files? If so cant this be consider a part of the build process?

Added subscriber: @nge

In a discussion with @nge in #docs on blender.chat one issue with the manual translations is the lack of quality of translations due to translators need to know proper RST formatting to translate po files.
Because moving to such a system that is online, translators will not be building docs locally and there is concern that the amount of formatting errors would increase.

One way to combat this is to write custom check within Weblate for RST syntax errors. https://docs.weblate.org/en/latest/user/checks.html
We could also improve are documentation stating that care needs to be taken to keep the syntax intact. This has caused issues like #68108

In a discussion with @nge in #docs on blender.chat one issue with the manual translations is the lack of quality of translations due to translators need to know proper RST formatting to translate po files. Because moving to such a system that is online, translators will not be building docs locally and there is concern that the amount of formatting errors would increase. One way to combat this is to write custom check within Weblate for RST syntax errors. https://docs.weblate.org/en/latest/user/checks.html We could also improve are documentation stating that care needs to be taken to keep the syntax intact. This has caused issues like #68108

In my opinion, there are three requirements for new translators:

Acquaintance with one or more parts of Blender

We gather our translators from BlenderCN community to assure this.

A knowledge of both English and target language
Actually, we don't have a way to qualify this(especially English), since a certificate of English is some kind of privacy.
A knowledge of basic rst syntax and svn.
Since most of Blender users are artists, and not familiar with rst, we summarized a step-by-step tutorial for building local manual, and make it a compulsory task for all Chinese translators. Every contributor should build locally and check the result page before submitting translated files to make sure there is no syntax error in them.

The new system should try to lower the barrier of the requirements above.

I'd like to share some of my suggestions:

rst syntax auto-check
As mentioned above, rst sets the barrier itself, we need to building locally to check and find out the errors in translated texts. So rst syntax auto-check ,suggestion and/or correction will help a lot.

Without this feature, this will be merely a online version of poedit, with some feature like glossary and peer review, of course.

But if this is not to be solved, we still can't assure the quality of translations, or it will be still be hard to attract more qualified volunteer.

Fast entry for translation.
Since the po files will be merged into one huge file, it will be neccesarry to add a fast entry in every manual page to redirect to corresponding strings in weblate.

And we can also add a entry of UI translation for every menu item, like the manual entry for every menu in RMB menu.

This will make contribution more covenient.

In my opinion, there are three requirements for new translators: 1. Acquaintance with one or more parts of Blender We gather our translators from BlenderCN community to assure this. 2. A knowledge of both English and target language Actually, we don't have a way to qualify this(especially English), since a certificate of English is some kind of privacy. 3. A knowledge of basic **rst** syntax and **svn**. Since most of Blender users are artists, and not familiar with **rst**, we summarized a step-by-step [tutorial](https://www.jianshu.com/p/b1a48105e05c) for building local manual, and make it a compulsory task for all Chinese translators. Every contributor should build locally and check the result page before submitting translated files to make sure there is no syntax error in them. The new system should try to lower the barrier of the requirements above. --- I'd like to share some of my suggestions: 1. rst syntax auto-check As mentioned above, **rst** sets the barrier itself, we need to building locally to check and find out the errors in translated texts. So rst syntax auto-check ,suggestion and/or correction will help a lot. Without this feature, this will be merely a online version of poedit, with some feature like glossary and peer review, of course. But if this is not to be solved, we still can't assure the quality of translations, or it will be still be hard to attract more qualified volunteer. 2. Fast entry for translation. Since the `po` files will be merged into one huge file, it will be neccesarry to add a fast entry in every manual page to redirect to corresponding strings in weblate. And we can also add a entry of UI translation for every menu item, like the manual entry for every menu in `RMB` menu. ![图片.png](https://archive.blender.org/developer/F7663206/图片.png) This will make contribution more covenient.

Added subscriber: @dmcgrath

In #68588#752509, @Blendify wrote:
[..] translation review [..]

I think this would be something nice to look into and that @jesterking in particular could help guide the foundation towards.

The proposed tool weblate looks like a good way forward to bring some sense into the translation process.

There should indeed be a good way to vet new translators to ensure we can get the best documentation and translations.

What I'd like to see is a better connection between code changes and the documentation and translation work it may or may not cause.

In #68588#752032, @ChristopherAnderssarian wrote:
What's the scope of this? Everything on the blender.org domain (store, cloud, dev wiki, ect...), something in between or just the Blender UI and Manual*(for now)//?

Or another way of asking: Is anything off the table?

For now I would like to limit this to only the Blender UI and the User Manual. In the future it would be nice to localize other services. But I would rather us focus and refine the workflow for what we already have.

Lets keep it for now with Blender and user manual. If we find that the tool works well in enabling our translators to do efficient work then we can see if we can tie other services into this system.

In #68588#752075, @mont29 wrote:
Note that you cannot really get rid of the svn repo for Blender UI translations, as those stored in GIT are fully stripped to be as small as possible, they are essentially useless as working translation material...

What is the reason for this? Is this to reduce the size of the generated Machine Object files? If so cant this be consider a part of the build process?

I'd also like to know the reason for this.

Ideally we move localisation files into a Git repository that would become a submodule of the main blender.git repository. Doing any post-processing of files should indeed be done by the build process. Maybe not by default, but there could be a target that is specifically for handling localisation files and preparing releases.

With the localisation on Git we have a much better connection with Blender, and can look into setting up some workflows where commits trigger tasks for translators, and ideally even documentors. That way we get an automatic list of issues that need to be handled by translators prior to release, and can be handled as changes to Blender happen in a much more intuitive way.

For any concerns about binary files in our repositories: we should enabled git-lfs and put binary files there. It has worked great for many years already, very stable.

I suppose next up would be asking @dmcgrath if we could get weblate hosted soonish.

@Blendify do you have a good grasp of the configuration of weblate and the integrations it provides?

Perhaps figure out these things as next step in preparation for a move to weblate for translation work.

So the following with in mind we use weblate at some blender.org address (translate.blender.org sounds good to me)

user registration - possible to use phabricator user database? Or some other way to keep these in sync?
propose review workflow
propose onboarding/testing of prospect translators for quality
find instructions to set up the git integration
propose how to do move to git from svn
propose a time schedule. Could we do it for 2.81 already? Or would it be something that we start now, but target to finalize before 2.82?

Anyway, this looks like a useful move to make. Thanks for writing up!

> In #68588#752509, @Blendify wrote: > [..] translation review [..] > I think this would be something nice to look into and that @jesterking in particular could help guide the foundation towards. The proposed tool weblate looks like a good way forward to bring some sense into the translation process. There should indeed be a good way to vet new translators to ensure we can get the best documentation and translations. What I'd like to see is a better connection between code changes and the documentation and translation work it may or may not cause. >> In #68588#752032, @ChristopherAnderssarian wrote: >> What's the scope of this? Everything on the [blender.org ](https:*blender.org) domain (store, cloud, dev wiki, ect...), something in between or just the Blender UI and Manual*(for now)//? >> >> Or another way of asking: *Is anything off the table?* > > For now I would like to limit this to only the Blender UI and the User Manual. In the future it would be nice to localize other services. But I would rather us focus and refine the workflow for what we already have. Lets keep it for now with Blender and user manual. If we find that the tool works well in enabling our translators to do efficient work then we can see if we can tie other services into this system. >> In #68588#752075, @mont29 wrote: >> Note that you cannot really get rid of the svn repo for Blender UI translations, as those stored in GIT are fully stripped to be as small as possible, they are essentially useless as working translation material... > > What is the reason for this? Is this to reduce the size of the generated Machine Object files? If so cant this be consider a part of the build process? I'd also like to know the reason for this. Ideally we move localisation files into a Git repository that would become a submodule of the main blender.git repository. Doing any post-processing of files should indeed be done by the build process. Maybe not by default, but there could be a target that is specifically for handling localisation files and preparing releases. With the localisation on Git we have a much better connection with Blender, and can look into setting up some workflows where commits trigger tasks for translators, and ideally even documentors. That way we get an automatic list of issues that need to be handled by translators prior to release, and can be handled as changes to Blender happen in a much more intuitive way. For any concerns about binary files in our repositories: we should enabled git-lfs and put binary files there. It has worked great for many years already, very stable. I suppose next up would be asking @dmcgrath if we could get weblate hosted soonish. @Blendify do you have a good grasp of the configuration of weblate and the integrations it provides? Perhaps figure out these things as next step in preparation for a move to weblate for translation work. So the following with in mind we use weblate at some blender.org address (translate.blender.org sounds good to me) * user registration - possible to use phabricator user database? Or some other way to keep these in sync? * propose review workflow * propose onboarding/testing of prospect translators for quality * find instructions to set up the git integration * propose how to do move to git from svn * propose a time schedule. Could we do it for 2.81 already? Or would it be something that we start now, but target to finalize before 2.82? Anyway, this looks like a useful move to make. Thanks for writing up!

Added subscriber: @DuarteRamos

user registration - possible to use phabricator user database? Or some other way to keep these in sync?

This is a topic I want to bring up for discussion separately ideally shouldnt we be using Blender ID for this and phabricator?

> user registration - possible to use phabricator user database? Or some other way to keep these in sync? This is a topic I want to bring up for discussion separately ideally shouldnt we be using Blender ID for this and phabricator?

In #68588#752075, @mont29 wrote:
Note that you cannot really get rid of the svn repo for Blender UI translations, as those stored in GIT are fully stripped to be as small as possible, they are essentially useless as working translation material...

What is the reason for this? Is this to reduce the size of the generated Machine Object files? If so cant this be consider a part of the build process?

Because we want the git repo (which has to be checked out by everybody that wants to build Blender) to be as small as possible. And a tremendous amount of changes in PO files, from week to week, are actually in the 'source' comments (line number changes). those info are mandatory for translation work, but useless if you only want to generate the MOs binaries...

Although the current SVN model has some historical heritage that we could probably get rid of now (e.g. I don't think generating MO's and storing them there is needed anymore...). I would still keep the 'translator' side of things outside of the 'blender' side of thing, but I guess the website could then run the validation/cleanup scripts I am running currently every week when I update the repos... And just use two git repos, one for full raw PO's, and one for proper trimmed out, checked and safe PO's?

>> In #68588#752075, @mont29 wrote: >> Note that you cannot really get rid of the svn repo for Blender UI translations, as those stored in GIT are fully stripped to be as small as possible, they are essentially useless as working translation material... > > What is the reason for this? Is this to reduce the size of the generated Machine Object files? If so cant this be consider a part of the build process? Because we want the git repo (which has to be checked out by everybody that wants to build Blender) to be as small as possible. And a tremendous amount of changes in PO files, from week to week, are actually in the 'source' comments (line number changes). those info are mandatory for translation work, but useless if you only want to generate the MOs binaries... Although the current SVN model has some historical heritage that we could probably get rid of now (e.g. I don't think generating MO's and storing them there is needed anymore...). I would still keep the 'translator' side of things outside of the 'blender' side of thing, but I guess the website could then run the validation/cleanup scripts I am running currently every week when I update the repos... And just use two git repos, one for full raw PO's, and one for proper trimmed out, checked and safe PO's?

Because we want the git repo (which has to be checked out by everybody that wants to build Blender) to be as small as possible. And a tremendous amount of changes in PO files, from week to week, are actually in the 'source' comments (line number changes). those info are mandatory for translation work, but useless if you only want to generate the MOs binaries...

Maybe we can improve out git flow for this. For example if we are worried about the the checkout size of the history then we can use a shallow checkout:
https://git-scm.com/docs/git-clone#Documentation/git-clone.txt---depthltdepthgt

This can be set up to only grab the files for the latest revision. Translators will be interfacing with weblate so the repository is basically only needed to be read only.
A few admins for example can checkout the full repository elsewhere on disk for purposes such as updating files and such.

We can also just handle this normally as I dont think the repository will be that large with only ~5000 commits so far.

@jesterking should probably orient us in the best option as the git flow.

Although the current SVN model has some historical heritage that we could probably get rid of now (e.g. I don't think generating MO's and storing them there is needed anymore...).

I agree, these should only be built as part of the build process.

> Because we want the git repo (which has to be checked out by everybody that wants to build Blender) to be as small as possible. And a tremendous amount of changes in PO files, from week to week, are actually in the 'source' comments (line number changes). those info are mandatory for translation work, but useless if you only want to generate the MOs binaries... Maybe we can improve out git flow for this. For example if we are worried about the the checkout size of the history then we can use a shallow checkout: https://git-scm.com/docs/git-clone#Documentation/git-clone.txt---depthltdepthgt This can be set up to only grab the files for the latest revision. Translators will be interfacing with weblate so the repository is basically only needed to be read only. A few admins for example can checkout the full repository elsewhere on disk for purposes such as updating files and such. We can also just handle this normally as I dont think the repository will be that large with only ~5000 commits so far. @jesterking should probably orient us in the best option as the git flow. > Although the current SVN model has some historical heritage that we could probably get rid of now (e.g. I don't think generating MO's and storing them there is needed anymore...). I agree, these should only be built as part of the build process.

I created two patches; one for the blender source and the other for the add-on to remove mo files from the translations repository.

D5648
D5649

After these changes, assuming I did not miss anything, the locale directory can be removed from svn.

Note that these will require a workflow change which needs to be updated here:

https://wiki.blender.org/wiki/Process/Translate_Blender

Specifically "How to test your translations - translators" and any reference to blender.mo

I created two patches; one for the blender source and the other for the add-on to remove mo files from the translations repository. - [D5648](https://archive.blender.org/developer/D5648) - [D5649](https://archive.blender.org/developer/D5649) After these changes, assuming I did not miss anything, the `locale` directory can be removed from svn. Note that these will require a workflow change which needs to be updated here: https://wiki.blender.org/wiki/Process/Translate_Blender Specifically "How to test your translations - translators" and any reference to `blender.mo`

@mont29 Can I get added to #translations ?

Somethings I noticed while testing UI translations:

es and es_ES are identical, I propose we remove es_ES
We currently have several po-files with no translations, I propose that we remove these also:
- Amharic (am)
- Estonian (et)
- Kazakh (kk)
- Serbian Latin Chars (sr@latin)
- Uzbek (uz)
- Uzbek Cryllic Chars (uz@cryllic)

Doing this will just help the migration process. In the future these can be easily re-added when we actually have translators for them.

Somethings I noticed while testing UI translations: - `es` and `es_ES` are identical, I propose we remove `es_ES` - We currently have several po-files with no translations, I propose that we remove these also: - Amharic (`am`) - Estonian (`et`) - Kazakh (`kk`) - Serbian Latin Chars (`sr@latin`) - Uzbek (`uz`) - Uzbek Cryllic Chars (`uz@cryllic`) Doing this will just help the migration process. In the future these can be easily re-added when we actually have translators for them.

@Blendify

es and es_ES are identical, I propose we remove es_ES

yes, please do it.
nobody has ever been behind it, since I started with es from scratch back in the day (and es_ES was already at 3%). Since then, I've been simply copying es over es_ES in the hopes that somebody could ever pick it up and develop it, but that never really happened.

@Blendify > es and es_ES are identical, I propose we remove es_ES yes, please do it. nobody has ever been behind it, since I started with **es** from scratch back in the day (and **es_ES** was already at 3%). Since then, I've been simply copying **es** over **es_ES** in the hopes that somebody could ever pick it up and develop it, but that never really happened.

I’d rather stay in charge of translation project, including which languages we keep or not… If es and es_ES are identical, then indeed the later should have been removed for a long time.

The empty translations please do not touch for now, I don't see how removing them would make transition easier, imho having them in the web UI might actually trigger some will from users to help translating in those languages.

In any case, I'd rather see the whole web-based translation system up and running before we do anything to current one (including docs).

@Blendify Am fine adding you to #Translations, but why do you need it exactly?

I’d rather stay in charge of translation project, including which languages we keep or not… If `es` and `es_ES` are identical, then indeed the later should have been removed for a long time. The empty translations please do not touch for now, I don't see how removing them would make transition easier, imho having them in the web UI might actually trigger some will from users to help translating in those languages. In any case, I'd rather see the whole web-based translation system up and running *before* we do anything to current one (including docs). @Blendify Am fine adding you to #Translations, but why do you need it exactly?

The only reason is to help do the changes I described but if you would like to stay on top of things instead that works too.

In any case, I'd rather see the whole web-based translation system up and running before we do anything to current one (including docs).

I am currently trying to investigate an issue I was having getting the Chinese languages imported on a local server.
Once I figure that out I can work with Dan on getting a server set up so we can do public tests.
The only issue the currently with the layout of the svn directory makes it impossible to load the translations into the website.
I ended up making my own repository that I have been using for testing and I guess we can use this also for a public test and switch to a real repo when we are ready.

The only reason is to help do the changes I described but if you would like to stay on top of things instead that works too. > In any case, I'd rather see the whole web-based translation system up and running *before* we do anything to current one (including docs). I am currently trying to investigate an issue I was having getting the Chinese languages imported on a local server. Once I figure that out I can work with Dan on getting a server set up so we can do public tests. The only issue the currently with the layout of the svn directory makes it impossible to load the translations into the website. I ended up making my own repository that I have been using for testing and I guess we can use this also for a public test and switch to a real repo when we are ready.

Added subscriber: @tim_occ

@Blendify what is your progress on weblate?

I've been working with Weblate on the translation of the Godot manual.
In my opinion, this tool has many advantages over the disadvantages. Above all, the glossary and translation suggestions make it much easier to coordinate between the translators.

With the current SVN system, you need well-founded knowledge in order to contribute to translations at all.

From my side 1+ for Weblate.

@Blendify what is your progress on weblate? I've been working with Weblate on the translation of the Godot manual. In my opinion, this tool has many advantages over the disadvantages. Above all, the glossary and translation suggestions make it much easier to coordinate between the translators. With the current SVN system, you need well-founded knowledge in order to contribute to translations at all. From my side 1+ for Weblate.

Hi, I have been quite busy the past couple months so I have no new progress to report. Hopefully soon we can get this up and running.

Added subscriber: @ChameleonScales

Added subscriber: @Poulpator

Added subscriber: @pioverfour

Added subscriber: @xan2622

Hello @Blendify,

Any news about the Weblate setup?

I saw that there is already a git repository that lists all of the *.po files. This established the trading point between developers and translators. :)
https://github.com/blender/blender-translations/tree/master/po

Let me know if you need any help. Although I have no experience with installing Weblate on a server, I have already set up several projects and components in a Weblate installation and am currently administering a project at my employer.

Hello @Blendify, Any news about the Weblate setup? I saw that there is already a git repository that lists all of the *.po files. This established the trading point between developers and translators. :) https://github.com/blender/blender-translations/tree/master/po Let me know if you need any help. Although I have no experience with installing Weblate on a server, I have already set up several projects and components in a Weblate installation and am currently administering a project at my employer.

Any news about the Weblate setup?

First the foundation needs to formally approve this project

I saw that there is already a git repository that lists all of the *.po files. This established the trading point between developers and translators. :)

I saw that there is already a git repository that lists all of the *.po files. This established the trading point between developers and translators. :)

This repositories is just for testing purposes, the official website would use repositories on developer.blender.org

Let me know if you need any help. Although I have no experience with installing Weblate on a server, I have already set up several projects and components in a Weblate installation and am currently administering a project at my employer.

I have nothing immediately, the hardest part of this will be getting weblate installed and configured on a server.

> Any news about the Weblate setup? First the foundation needs to formally approve this project > I saw that there is already a git repository that lists all of the *.po files. This established the trading point between developers and translators. :) I saw that there is already a git repository that lists all of the *.po files. This established the trading point between developers and translators. :) This repositories is just for testing purposes, the official website would use repositories on developer.blender.org > Let me know if you need any help. Although I have no experience with installing Weblate on a server, I have already set up several projects and components in a Weblate installation and am currently administering a project at my employer. I have nothing immediately, the hardest part of this will be getting weblate installed and configured on a server.

Added subscriber: @Yuro

Added subscriber: @ponderz

Added subscriber: @imhjnoh

Added subscriber: @MetinSeven-1

On January 1st, 2021 I'll start as a part-time Technical Artist for the Blender Foundation, and one of my tasks will be to improve Blender documentation. I'd love to start translating the Blender manual to Dutch, but my first impression of how to realize a translation is that it's cumbersome at the moment, and requires some UNIX knowledge if I've understood it correctly.

I'd very much welcome the Weblate approach, and hope it will be approved soon.

Thanks!

On January 1st, 2021 I'll start as a part-time Technical Artist for the Blender Foundation, and one of my tasks will be to improve Blender documentation. I'd love to start translating the Blender manual to Dutch, but my first impression of how to realize a translation is that it's cumbersome at the moment, and requires some UNIX knowledge if I've understood it correctly. I'd very much welcome the Weblate approach, and hope it will be approved soon. Thanks!

Added subscriber: @T.R.O.Nunes

Small update, the user manual translation file structure has been updated to a format that works with weblate.
There is a small bug in sphinx that affect translations of our HTML templates, this has been reported here:
https://github.com/sphinx-doc/sphinx/issues/9030

Right now the next step would be to iron the details for weblate and how to integrate into our workflow.
We also need to get weblate installed on Blender.org and figure out how to handle user accounts / integrate Blender ID.
I am working on setting up a local demo for myself to test out the workflow and better understand how the system works.

Small update, the user manual translation file structure has been updated to a format that works with weblate. There is a small bug in sphinx that affect translations of our HTML templates, this has been reported here: https://github.com/sphinx-doc/sphinx/issues/9030 Right now the next step would be to iron the details for weblate and how to integrate into our workflow. We also need to get weblate installed on Blender.org and figure out how to handle user accounts / integrate Blender ID. I am working on setting up a local demo for myself to test out the workflow and better understand how the system works.

Added subscriber: @cobisimo

If you need any help in git migration and setting up Weblate tool, maybe I can help.
Only thing is that I am busy during week, so can spend some time during weekends.

If you need any help in git migration and setting up Weblate tool, maybe I can help. Only thing is that I am busy during week, so can spend some time during weekends.

Added subscriber: @Jarrett-Johnson

Added subscriber: @Balint-Csuthy

Hello, I tried a way to convert the translations repo to git. I understand that this topic is under discussion and it may not be helpful, but here goes.

Clone svn repo with git

git svn clone https://svn.blender.org/svnroot/bf-translations/ bf-translations -Tbranches

This has the effect of taking the branches dir and using it as the root for the repo. The trunk and tags are thus dropped, and their associated commits forgotten.

Rewrite history

We then rewrite the history so that all .po files are inside a single po dir, the same way as the current light git repo works. We also remove leftover mo files and move a few python scripts to a new directory. To do that, we use the git-filter-repo tool, with the following config file:

# File bf-translation_convert_to_git-files_list.txt

# Move .pot and .po files
blender.pot==>po/blender.pot
regex:(.*)/([^/]*)\.po$==>po/\2.po

# Move Python scripts
regex:(.*)/([^/]*)\.py$==>scripts/\2.py

# Delete .mo files
regex:(.*)/([^/]*)\.mo$==>

We do a copy of the repo first, and then run the command:
git filter-repo --force --paths-from-file ../bf-translation_convert_to_git-files_list.txt

Delete the leftover trunk branch, which points to the same commit as master:
git branch -D trunk

We now have a 710MB repo with the following structure:

.
├── po
│   ├── ab.po
│   ├── am.po
│   ├── ar.po
│   ├── bg.po
│   ├── blender.pot
│   ├── ca.po
│   ├── ...
│   ├── uz@cyrillic.po
│   ├── uz.po
│   ├── vi.po
│   ├── zh_CN.po
│   └── zh_TW.po
└── scripts
    ├── ar_to_utf.py
    └── fa_to_utf.py

Compare to the current SVN checkout, which is 3.3GB. This includes 1.5GB of tags for old releases (pre 2.7), which are duplicates of data found in the branches anyway.

Name and e-mail remapping

Right now the commits’ authors and e-mail addresses were generated automatically on converting from SVN, looking like this in logs:

Author: pioverfour <pioverfour@a22679aa-88f2-e011-8c9d-001b2159fb48>

They can be remapped while rewriting the history by using:

git filter-repo --force --paths-from-file ../bf-translation_convert_to_git_files.txt --mailmap ../bf-my-mailmap.txt

with my-mailmap.txt consisting of:

Damien Picard <address@domain.tld> pioverfour <pioverfour@a22679aa-88f2-e011-8c9d-001b2159fb48>
...

A list of phabricator authors can be obtained using:

git log --pretty=format:"%aN <%aE>" | sort -u

I’ve got such a partial list if interested.

Hello, I tried a way to convert the translations repo to git. I understand that this topic is under discussion and it may not be helpful, but here goes. ### Clone svn repo with git `git svn clone https://svn.blender.org/svnroot/bf-translations/ bf-translations -Tbranches` This has the effect of taking the `branches` dir and using it as the root for the repo. The trunk and tags are thus dropped, and their associated commits forgotten. ### Rewrite history We then rewrite the history so that all .po files are inside a single po dir, the same way as the current light git repo works. We also remove leftover mo files and move a few python scripts to a new directory. To do that, we use the [git-filter-repo](https:*htmlpreview.github.io/?https:*github.com/newren/git-filter-repo/blob/docs/html/git-filter-repo.html#_filtering_based_on_many_paths) tool, with the following config file: ``` # File bf-translation_convert_to_git-files_list.txt # Move .pot and .po files blender.pot==>po/blender.pot regex:(.*)/([^/]*)\.po$==>po/\2.po # Move Python scripts regex:(.*)/([^/]*)\.py$==>scripts/\2.py # Delete .mo files regex:(.*)/([^/]*)\.mo$==> ``` We do a copy of the repo first, and then run the command: `git filter-repo --force --paths-from-file ../bf-translation_convert_to_git-files_list.txt` Delete the leftover `trunk` branch, which points to the same commit as `master`: `git branch -D trunk` We now have a 710MB repo with the following structure: ``` . ├── po │ ├── ab.po │ ├── am.po │ ├── ar.po │ ├── bg.po │ ├── blender.pot │ ├── ca.po │ ├── ... │ ├── uz@cyrillic.po │ ├── uz.po │ ├── vi.po │ ├── zh_CN.po │ └── zh_TW.po └── scripts ├── ar_to_utf.py └── fa_to_utf.py ``` Compare to the current SVN checkout, which is 3.3GB. This includes 1.5GB of tags for old releases (pre 2.7), which are duplicates of data found in the branches anyway. ### Name and e-mail remapping Right now the commits’ authors and e-mail addresses were generated automatically on converting from SVN, looking like this in logs: ``` Author: pioverfour <pioverfour@a22679aa-88f2-e011-8c9d-001b2159fb48> ``` They can be [remapped while rewriting the history](https:*htmlpreview.github.io/?https:*github.com/newren/git-filter-repo/blob/docs/html/git-filter-repo.html#_user_and_email_based_filtering) by using: ``` git filter-repo --force --paths-from-file ../bf-translation_convert_to_git_files.txt --mailmap ../bf-my-mailmap.txt ``` with my-mailmap.txt consisting of: ``` Damien Picard <address@domain.tld> pioverfour <pioverfour@a22679aa-88f2-e011-8c9d-001b2159fb48> ... ``` A list of phabricator authors can be obtained using: ``` git log --pretty=format:"%aN <%aE>" | sort -u ``` I’ve got such a partial list if interested.

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Labels Milestones

Blender: Going Locale #68588

Issue

Proposal

Weblate

Improved Communication

Implementation

Timeline

Tasks

Questions

Clone svn repo with git

Rewrite history

Name and e-mail remapping