I18n disambiguation using arbitrary string literals as gettext contexts #100991

New Issue

Damien Picard · 2022-09-12T03:02:53+02:00

Damien Picard commented

2022-09-12 03:02:53 +02:00

TL;DR The i18n policy should change so that literal strings may be used as contexts in some specific situations instead of the current fixed defines, to help disambiguation and to ultimately have better translations.

The issue
The translation system currently enforces using specific keywords as contexts, through defines in BLT_translation.h.

In the vast majority of cases, there is no need for contexts at all. In many others, a suitable context may be chosen from one of the existing context defines. But in rare cases, choosing from one of those is just not possible because none fits, or there is a conflict between meanings within a context, or it’s a workaround because the context is only vaguely related to the message, which is an abuse of gettext contexts.

Here is an example of an issue I’m facing, where a context collision occurs.
I’m trying to disambiguate the message “New” from the Time Stretching panel, because not every language uses the same gender for its translation. But basically every ID is already used as context for “New”, because that’s how the button to create an ID is named.
So, which context should this message use? Maybe id_scene, because the property is in Scene? But now there is a conflict between New [Scene] and New [Time]: in French and other languages, they should not have the same translation, but they use the same context.
The next step would be to consider adding a new context define. But should I create a context just for this string, or should I try to find a more generic one? Maybe I could add properties_scene_time, or even just properties_scene. Either of those would work fine, but then I need to add the context in three places just to disambiguate this one message, which is really overkill for this issue, and may not prevent all further conflicts…

Another example:
The message “Mix” is used in many places, with three basic meanings: the mix blending mode, the mix factor between two states, or the action to mix two things together.
Currently, in order to disambiguate between these three meanings, one has to either create new contexts just for these messages (something like “Blending mode”, “Mix factor” and “Verb: mix two things together”, maybe), or use one of the existing contexts, which will create conflicts because Mix is used with different meanings within each possible context. Also, it will ironically create duplicate translations, when only three would have been needed.

The reasons
Three reasons are given in this wiki page:

A typo would go unnoticed, e.g. creating two contexts where only one should exists.
It’s quite hard to know whether a same (or similar) context already exists!
And last but not least, it would not fit into our Python i18n API!

Counter-points
Although consistency is compelling, I’ll try to explain why I believe this should not be so strictly enforced by going through those reasons.

As stated in the wiki page, contexts are a last resort thing in Blender, and by default none needs to be used. So there aren’t actually that many strings using specific contexts, compared to the global number. Even if a new context is created where it shouldn’t be, it only affects one instance of one message. Sure, this new message will need to be translated, which gives more work to translators. But on the other hand, if a developer needs to disambiguate a specific string, it would be because they were asked by the translators to do it in the first place, so they will look at a translation file before adding the context anyway. Also, as shown in my second example, the current system can also lead to creating duplicate translations.
Disambiguation usually happens when translators notice an issue and report it, because, again, no context is used by default. To disambiguate a message, you do not need to know whether a specific context exists. You only need to know if it exists for this specific message. And this is trivial: you just search for the message in the po file. From the list that appears, you either choose the existing relevant context, or choose a new, explicit one. But this is already the case, because when disambiguating a message, you need to know which of the existing defines were used for which translation, so the way to know is to look in one of the po files (or maybe the pot file if you only speak English). I would add that currently, I have a similar problem because I have to keep BLT_translation.h open to know which contexts exist, and a po file to know which were used.
If arbitrary contexts would not fit into our Python i18n API, well, maybe we could change the Python API to accept them. I’m willing to look into it. So far the only issue I could find is that arbitrary contexts in non-official add-ons don’t get properly extracted to the translations_tuple. While this is certainly an issue, I’m sure it can be solved. Besides, this system is not widely used currently as far as I’m aware—apart from Bastien with Render Copy Settings, I’m the only add-on maintainer who’s introduced translations to community add-ons bundled with Blender.

The proposal
Well, you get the idea by now :D
For new messages, everything should remain as it is now: developers don’t worry about context at first and leave the disambiguation reporting to translators.
For existing messages needing to be disambiguated, the developer first looks at existing context defines, and if no suitable one exists, only then uses a literal string.
This way we don’t have to change the existing system, and we can still do more advanced disambiguation where it’s needed—although the existing system could also be changed quite easily, by searching and replacing the existing contexts by literals in the whole code base, but that’s another story.

TL;DR The i18n policy should change so that literal strings may be used as contexts in some specific situations instead of the current fixed defines, to help disambiguation and to ultimately have better translations. ----- **The issue** The translation system currently enforces using specific keywords as contexts, through defines in `BLT_translation.h`. In the vast majority of cases, there is no need for contexts at all. In many others, a suitable context may be chosen from one of the existing context defines. But in rare cases, choosing from one of those is just not possible because none fits, or there is a conflict between meanings within a context, or it’s a workaround because the context is only vaguely related to the message, which is an abuse of [gettext contexts](https://www.gnu.org/software/gettext/manual/html_node/Contexts.html). Here is an example of an issue I’m facing, where a context collision occurs. I’m trying to disambiguate the message “New” from the Time Stretching panel, because not every language uses the same gender for its translation. But basically every ID is already used as context for “New”, because that’s how the button to create an ID is named. So, which context should this message use? Maybe `id_scene`, because the property is in Scene? But now there is a conflict between *New [Scene]* and *New [Time]*: in French and other languages, they should not have the same translation, but they use the same context. The next step would be to consider adding a new context define. But should I create a context just for this string, or should I try to find a more generic one? Maybe I could add `properties_scene_time`, or even just `properties_scene`. Either of those would work fine, but then I need to add the context in three places just to disambiguate this one message, which is really overkill for this issue, and may not prevent all further conflicts… Another example: The message “Mix” is used in many places, with three basic meanings: the mix blending mode, the mix factor between two states, or the action to mix two things together. Currently, in order to disambiguate between these three meanings, one has to either create new contexts just for these messages (something like “Blending mode”, “Mix factor” and “Verb: mix two things together”, maybe), or use one of the existing contexts, which *will* create conflicts because Mix is used with different meanings within each possible context. Also, it will ironically create duplicate translations, when only three would have been needed. **The reasons** Three reasons are given in [this wiki page](https://wiki.blender.org/wiki/Source/Interface/Internationalization#Contexts): 1. A typo would go unnoticed, e.g. creating two contexts where only one should exists. 2. It’s quite hard to know whether a same (or similar) context already exists! 3. And last but not least, it would not fit into our Python i18n API! **Counter-points** Although consistency is compelling, I’ll try to explain why I believe this should not be so strictly enforced by going through those reasons. 1. As stated in the wiki page, contexts are a last resort thing in Blender, and by default none needs to be used. So there aren’t actually that many strings using specific contexts, compared to the global number. Even if a new context is created where it shouldn’t be, it only affects one instance of one message. Sure, this new message will need to be translated, which gives more work to translators. But on the other hand, if a developer needs to disambiguate a specific string, it would be because they were asked *by* the translators to do it in the first place, so they will look at a translation file before adding the context anyway. Also, as shown in my second example, the current system can also lead to creating duplicate translations. 2. Disambiguation usually happens when translators notice an issue and report it, because, again, no context is used by default. To disambiguate a message, you do not need to know whether a specific context exists. You only need to know if it exists *for this specific message*. And this is trivial: you just search for the message in the po file. From the list that appears, you either choose the existing relevant context, or choose a new, explicit one. But this is already the case, because when disambiguating a message, you need to know which of the existing defines were used for which translation, so the way to know is to look in one of the po files (or maybe the pot file if you only speak English). I would add that currently, I have a similar problem because I have to keep `BLT_translation.h` open to know which contexts exist, *and* a po file to know which were used. 3. If arbitrary contexts would not fit into our Python i18n API, well, maybe we could change the Python API to accept them. I’m willing to look into it. So far the only issue I could find is that arbitrary contexts in non-official add-ons don’t get properly extracted to the `translations_tuple`. While this is certainly an issue, I’m sure it can be solved. Besides, this system is not widely used currently as far as I’m aware—apart from Bastien with Render Copy Settings, I’m the only add-on maintainer who’s introduced translations to community add-ons bundled with Blender. **The proposal** Well, you get the idea by now :D For new messages, everything should remain as it is now: developers don’t worry about context at first and leave the disambiguation reporting to translators. For existing messages needing to be disambiguated, the developer first looks at existing context defines, and if no suitable one exists, only *then* uses a literal string. This way we don’t have to change the existing system, and we can still do more advanced disambiguation where it’s needed—although the existing system could also be changed quite easily, by searching and replacing the existing contexts by literals in the whole code base, but that’s another story.

Damien Picard commented

2022-09-12 03:02:53 +02:00

Changed status from 'Needs Triage' to: 'Needs Developer To Reproduce'

Damien Picard commented

2022-09-12 03:02:53 +02:00

Added subscriber: @pioverfour

Bastien Montagne commented

2022-12-13 02:57:07 +01:00

Added subscriber: @mont29

Bastien Montagne commented

2022-12-13 02:57:07 +01:00

I do not really understand why adding a few other predefined contexts is really an issue here... a TIME one would fix your first example, and is potentially re-usable in other places... In your second example, you can already use the operator context for the verb case (operator == action == verb). And adding a BLEND_MODE context could also make sense e.g. in other blending cases. Then last case can just keep using the generic context.

In a nutshell, I'd rather stick to current system for now, do not really see any strong point here justifying accepting the aforementioned drawbacks and spending time and efforts into supporting random contexts.

I do not really understand why adding a few other predefined contexts is really an issue here... a `TIME` one would fix your first example, and is potentially re-usable in other places... In your second example, you can already use the `operator` context for the verb case (operator == action == verb). And adding a `BLEND_MODE` context could also make sense e.g. in other blending cases. Then last case can just keep using the generic context. In a nutshell, I'd rather stick to current system for now, do not really see any strong point here justifying accepting the aforementioned drawbacks and spending time and efforts into supporting random contexts.

Damien Picard commented

2022-12-14 01:49:03 +01:00

Hello, thank you for reading this long rant!

To be very honest, I don’t understand the benefits of this system. I find it confusing from a translator’s point of view; frustrating or even harmful for disambiguation because contexts could be more granular and helpful to translators; and not really helpful to other developers, since they’re not supposed to deal with it in the first place.

This is why I’d like to work toward relying less on it going forward, or even getting rid of it eventually.

Now it’s very possible that I’m missing something, but I tested getting rid of the defines entirely by doing a search and replace in the code base, and it seems to work just as well, including message extraction–and I actually find the code more legible, but maybe that’s just me.

But I also understand that things which work need a good reason to be changed. If you think it’s not worth it, I can’t argue with that.

Hello, thank you for reading this long rant! To be very honest, I don’t understand the benefits of this system. I find it confusing from a translator’s point of view; frustrating or even harmful for disambiguation because contexts could be more granular and helpful to translators; and not really helpful to other developers, since they’re not supposed to deal with it in the first place. This is why I’d like to work toward relying less on it going forward, or even getting rid of it eventually. Now it’s very possible that I’m missing something, but I tested getting rid of the defines entirely by doing a search and replace in the code base, and it seems to work just as well, including message extraction–and I actually find the code more legible, but maybe that’s just me. But I also understand that things which work need a good reason to be changed. If you think it’s not worth it, I can’t argue with that.

Bastien Montagne added this to the User Interface project 2023-02-09 15:34:10 +01:00

Philipp Oeser commented

2023-02-14 12:02:02 +01:00

If we move this out of the triaging queue (by changing status label to Needs Information from Developers), this needs to be the responsibility of a module (so a Module label must be added)

If we move this out of the triaging queue (by changing status label to `Needs Information from Developers`), this needs to be the responsibility of a module (so a `Module` label must be added)

Philipp Oeser added the

Module

User Interface

label 2023-02-14 12:02:18 +01:00

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

I18n disambiguation using arbitrary string literals as gettext contexts #100991