Proposal: use str.format instead of percentage formatting for core-scripts #120453

New Issue

Campbell Barton · 2024-04-10T05:55:04+02:00

Campbell Barton commented

2024-04-10 05:55:04 +02:00

Motivation

Using Python's string formatting (str.format syntax) is generally preferred for modern Python (in tutorials & documentation).
Based on add-ons and scripts maintained outside of scripts/startup percentage-formatting is no longer the preferred method of string formatting used by Blender developers.

Pros & Cons

Pros:

Developers who maintain modern code-bases outside of Blender don't have to learn percentage string formatting since f-string & str.format use same mini-language.
Clearer for closely grouped formatting:
- "0x%.2x%.2x%.2x%.2x" % color: %-format.
- "0x{:02x}{:02x}{:02x}{:02x}".format(*color): str.format.
Also when . is part of the string:
- "%s%.3d.fbx" % (name, index): %-format.
- "{:s}{:03d}.fbx" % (name, index): str.format.
Reduced ambiguity with multiple arguments:
- "%d %s" % var: %-format.
- "{:d} {:s}".format(*var): str.format.
%-format reads like it could be a bug, it's only valid if var is a tuple but this relies on checking the surrounding context to verify.

By contrast str.format requires explicitly unpacking var making the developers intention clear.

There is also the case where you may want to print a tuple, requiring awkward single-tuple syntax:
- "Example %r text" % (var,): %-format.
- "Example {:r} text".format(var): str.format.
str.format is similar to fmt library which is becoming more widely used in C++ code.

Cons:

Performance, str.format is over twice as slow.
More verbose, especially for short expressions:
- "(%d): %s" % (a, b) compared to...
- "({:d}): {:s}".format(a, b).

While performance aspect shows up in micro-benchmarks the absolute times are both very small (~112 vs 255 nanoseconds in my tests).

If we were to consider these kinds of changes to be deal breakers then there would be other changes we could make with the Python API use - such as avoiding function call overhead in the UI... nevertheless being slower is a down side - so it needs to be mentioned.

Proposal

Use str.format for all scripts in: scripts/startup/
Replace percentage formatting with str.format in user facing scripts:
- Docs: doc/python_api/examples/
- Templates: scripts/templates_py/
Update https://developer.blender.org/docs/handbook/guidelines/python/#conventions-for-core-scripts
Prefer str.format over percentage-formatting elsewhere (percentage formatting can be phased out for Blender's scripts).

Details

Always use type specifiers for clarity.
Use positional arguments (with an exception for complex cases, see Keyword arguments below)

Before:

file_preset.write("%s = %r\n" % (rna_path_step, value))

After:

file_preset.write("{:s} = {!r}\n".format(rna_path_step, value))

Before:

layout.label(
    text=iface_("Rename %d %s") % (len(self._data[0]), self._data[2]),
    translate=False,
)

After:

layout.label(
    text=iface_("Rename {:d} {:s}").format(len(self._data[0]), self._data[2]),
    translate=False,
)

Keyword arguments (for clarity when needed)

In situations where it's more difficult to keep track of positional arguments, developers may used keyword arguments:

"Sample text {foo:s} mixed in with the string {bar:d}, etc... {baz:f}".format(
    foo=construct_string(many, arguments),
    bar=calculate_number(even, more, arguments),
    baz=compute_floating_point(lots, more, arguments),
)

Rationale for str.format over f-strings

The reasons to propose str.format over f-strings are as follows:

f-string wont work for translated strings, because the evaluated strings can't be used for translation look-ups.
str.format is a reasonable one-size-fits-(practically)-all solution for core scripts, the existing formatting method can be replaced with str.format in a straightforward way.
Development tools still don't support f-strings as well as they might (autopep8 wont format within f-strings for example, some features in my editor don't work inside f-strings).
Using f-string would push us to make subjective judgement calls, even refactoring to avoid having too much complexity in an f-strings or avoiding overly long expressions which cause the f-string to be split over multiple lines.

While the last point is quite subjective, the issue with translated labels mean even if we wanted to use f-strings we would have to avoid them and keep track of when f-strings can/can't be used, apply this rule consistently, take into account when reviewing patches etc - which I'd rather avoid.

Further, support for passing keywords to str.format largely addresses the same pain-point that f-strings do for more complex situations.

Other Notes

There are only around 120 instances of percentage string formatting, making this project a relatively small refactor.
See previous discussion on this topic: https://devtalk.blender.org/t/python-string-formatting/14040

## Motivation - Using Python's string formatting (`str.format` syntax) is generally preferred for modern Python (in tutorials & documentation). - Based on add-ons and scripts maintained outside of `scripts/startup` percentage-formatting is no longer the preferred method of string formatting used by Blender developers. ## Pros & Cons **Pros:** - Developers who maintain modern code-bases outside of Blender don't have to learn percentage string formatting since `f-string` & `str.format` use same mini-language. - Clearer for closely grouped formatting: - `"0x%.2x%.2x%.2x%.2x" % color`: `%-format`. - `"0x{:02x}{:02x}{:02x}{:02x}".format(*color)`: `str.format`. Also when `.` is part of the string: - `"%s%.3d.fbx" % (name, index)`: `%-format`. - `"{:s}{:03d}.fbx" % (name, index)`: `str.format`. - Reduced ambiguity with multiple arguments: - `"%d %s" % var`: `%-format`. - `"{:d} {:s}".format(*var)`: `str.format`. `%-format` reads like it could be a bug, it's only valid if `var` is a tuple but this relies on checking the surrounding context to verify. By contrast `str.format` requires explicitly unpacking `var` making the developers intention clear. There is also the case where you may want to print a tuple, requiring awkward single-tuple syntax: - `"Example %r text" % (var,)`: `%-format`. - `"Example {:r} text".format(var)`: `str.format`. - `str.format` is similar to `fmt` library which is becoming more widely used in C++ code. **Cons:** - Performance, `str.format` is over twice as slow. - More verbose, especially for short expressions: - `"(%d): %s" % (a, b)` compared to... - `"({:d}): {:s}".format(a, b)`. While performance aspect shows up in [micro-benchmarks](https://stackoverflow.com/questions/62566526) the absolute times are both very small (~112 vs 255 nanoseconds in my tests). If we were to consider these kinds of changes to be deal breakers then there would be other changes we could make with the Python API use - such as avoiding function call overhead in the UI... nevertheless being slower is a down side - so it needs to be mentioned. ## Proposal - Use `str.format` for all scripts in: `scripts/startup/` - [x] Replace percentage formatting with `str.format` in user facing scripts: - [x] Docs: `doc/python_api/examples/` - [x] Templates: `scripts/templates_py/` - Update https://developer.blender.org/docs/handbook/guidelines/python/#conventions-for-core-scripts - Prefer `str.format` over percentage-formatting elsewhere (percentage formatting can be phased out for Blender's scripts). #### Details - Always use type specifiers for clarity. - Use positional arguments _(with an exception for complex cases, see **Keyword arguments** below)_ Before: ``` python file_preset.write("%s = %r\n" % (rna_path_step, value)) ``` After: ``` python file_preset.write("{:s} = {!r}\n".format(rna_path_step, value)) ``` Before: ``` python layout.label( text=iface_("Rename %d %s") % (len(self._data[0]), self._data[2]), translate=False, ) ``` After: ``` python layout.label( text=iface_("Rename {:d} {:s}").format(len(self._data[0]), self._data[2]), translate=False, ) ``` - Keyword arguments (for clarity when needed) In situations where it's more difficult to keep track of positional arguments, developers may used keyword arguments: ``` python "Sample text {foo:s} mixed in with the string {bar:d}, etc... {baz:f}".format( foo=construct_string(many, arguments), bar=calculate_number(even, more, arguments), baz=compute_floating_point(lots, more, arguments), ) ``` ## Rationale for str.format over f-strings The reasons to propose `str.format` over f-strings are as follows: - f-string wont work for translated strings, because the evaluated strings can't be used for translation look-ups. - `str.format` is a reasonable one-size-fits-(practically)-all solution for core scripts, the existing formatting method can be replaced with `str.format` in a straightforward way. - Development tools still don't support f-strings as well as they might (autopep8 wont format within f-strings for example, some features in my editor don't work inside f-strings). - Using f-string would push us to make subjective judgement calls, even refactoring to avoid having too much complexity in an f-strings or avoiding overly long expressions which cause the f-string to be split over multiple lines. While the last point is quite subjective, the issue with translated labels mean even if we wanted to use f-strings we would have to avoid them and keep track of when f-strings can/can't be used, apply this rule consistently, take into account when reviewing patches etc - which I'd rather avoid. Further, support for passing keywords to `str.format` largely addresses the same pain-point that f-strings do for more complex situations. ## Other Notes - There are only around 120 instances of percentage string formatting, making this project a relatively small refactor. - See previous discussion on this topic: https://devtalk.blender.org/t/python-string-formatting/14040

Campbell Barton added the

labels 2024-04-10 05:55:04 +02:00

YimingWu commented

2024-04-10 06:18:12 +02:00

f-string wont work so well for translated strings, where changing logic within the f-string will cause the translation to need updating.

How come it's not going to be an issue with str.format? Once you changed the format string the translation would need to be changed anyway 🤔

> f-string wont work so well for translated strings, where changing logic within the f-string will cause the translation to need updating. How come it's not going to be an issue with `str.format`? Once you changed the format string the translation would need to be changed anyway 🤔

Campbell Barton commented

2024-04-10 06:33:52 +02:00

@ChengduLittleA this is my understanding although @mont29 may have more insights here.

The translation system currently receives "Rename %d %r" and uses a lookup table to replace that exact string.

If we use str.format the string would be "Rename {:d} {!r}" which is roughly equivalent.

From what I can see f-strings can't be used for translated strings because translation lookups on evaluated strings wont work e.g. iface_(f"Rename {a+b} {self.modifier.name}") will fail because it would use the result of a+b and self.modifier.name in the lookup.

Even if it could be made to work having code inlined seems like it's not ideal for the translation system to include as changes to the code would cause the string literal to change.

@ChengduLittleA this is my understanding although @mont29 may have more insights here. The translation system currently receives `"Rename %d %r"` and uses a lookup table to replace that exact string. If we use `str.format` the string would be `"Rename {:d} {!r}"` which is roughly equivalent. From what I can see f-strings can't be used for translated strings because translation lookups on evaluated strings wont work e.g. `iface_(f"Rename {a+b} {self.modifier.name}")` will fail because it would use the result of `a+b` and `self.modifier.name` in the lookup. Even if it could be made to work having code inlined seems like it's not ideal for the translation system to include as changes to the code would cause the string literal to change.

👍 1

Sybren A. Stüvel commented

2024-04-11 10:10:34 +02:00

👍 LGTM!

Personally I've always liked % formatting, but that's just because of familiarity since I've been using printf() for so long. I stand behind all the advantages of str.format() described in this design task though. Especially with f-strings becoming more prevalant (especially in software that doesn't need translation) the format string mini-language is getting more widely known too, so let's go!

:+1: LGTM! Personally I've always liked `%` formatting, but that's just because of familiarity since I've been using `printf()` for so long. I stand behind all the advantages of `str.format()` described in this design task though. Especially with f-strings becoming more prevalant (especially in software that doesn't need translation) the format string mini-language is getting more widely known too, so let's go!

Bastien Montagne commented

2024-04-11 14:29:45 +02:00

I'm not sure I really see the urgency/actual benefit of this change, but am also not opposed to it. Only real downside I see to using str.format is that it's somewhat more verbose than the % C-style way. The performances issues noted in https://stackoverflow.com/questions/62566526/percentage-formatting-twice-as-fast-as-f-strings-in-python-3-x also still seems valid? But this indeed also matches our C++ fmt lib syntax...

Regarding f-strings, the problems with translations arise even before looking up for translations during execution: we cannot even extract f-strings currently.

I18N message extraction uses python's AST module to find strings, and f-strings are... not strings at AST level - never found a way to get anything usable from them at that level of abstraction. In any case, I'm also not sure how we would deal with such strings at translation level. Even the basic % syntax is already an issue (people forget some, or modify their order, and with C-code this can even lead to crashes). I would rather not have interpreted python code in these translations strings! ;)

So as much as I love f-strings, they are indeed an absolute no-go for any UI-related message.

@ChengduLittleA indeed the existing translations will have to be updated, but like @ideasman42 said, it's only a change of syntax, so while annoying to have to deal with it, it's not really a big issue either.

I'm not sure I really see the urgency/actual benefit of this change, but am also not opposed to it. Only real downside I see to using `str.format` is that it's somewhat more verbose than the `%` C-style way. The performances issues noted in https://stackoverflow.com/questions/62566526/percentage-formatting-twice-as-fast-as-f-strings-in-python-3-x also still seems valid? But this indeed also matches our C++ `fmt` lib syntax... Regarding f-strings, the problems with translations arise even before looking up for translations during execution: we cannot even _extract_ f-strings currently. I18N message extraction uses python's AST module to find strings, and f-strings are... not strings at AST level - never found a way to get anything usable from them at that level of abstraction. In any case, I'm also not sure how we would deal with such strings at translation level. Even the basic `%` syntax is already an issue (people forget some, or modify their order, and with C-code this can even lead to crashes). I would rather _not_ have interpreted python code in these translations strings! ;) So as much as I love f-strings, they are indeed an absolute no-go for any UI-related message. @ChengduLittleA indeed the existing translations will have to be updated, but like @ideasman42 said, it's only a change of syntax, so while annoying to have to deal with it, it's not really a big issue either.

Campbell Barton commented

2024-04-11 16:34:02 +02:00

@mont29 the performance issue is still valid unfortunately and something I should have mentioned in the proposal, although this is an area of CPython that had some work - the improvements don't seem significant (in some of my tests str.format is ~40% slower).

Comparing "%s %s %s" with "{:s} {:s} {:s}" the old-style is around twice as fast perhaps a little faster taking out the time to iterate, however this is comparing very small values.

0.000000112 seconds.
0.000000255 seconds.

112 vs 255 nano-seconds, which I think can be overlooked as I doubt we could even measure the difference from Blender's UI in practice. This was a concern previously but since we're no longer writing exporters in Python, the performance of formatting large amounts of data is less of an issue.

@mont29 the performance issue is still valid unfortunately and something I should have mentioned in the proposal, although this is an area of CPython that had some work - the improvements don't seem significant (in some of my tests str.format is ~40% slower). Comparing `"%s %s %s"` with `"{:s} {:s} {:s}"` the old-style is around twice as fast perhaps a little faster taking out the time to iterate, however this is comparing very small values. - `0.000000112` seconds. - `0.000000255` seconds. 112 vs 255 nano-seconds, which I think can be overlooked as I doubt we could even measure the difference from Blender's UI in practice. This was a concern previously but since we're no longer writing exporters in Python, the performance of formatting large amounts of data is less of an issue.

Campbell Barton commented

2024-04-12 02:21:22 +02:00

Added pros & cons to the proposal.

Campbell Barton referenced this issue

2024-04-12 08:10:40 +02:00

Python: refactor scripts under scripts/startup/ to use `str.format` #120552

Campbell Barton commented

2024-04-12 08:12:10 +02:00

Submitted PR to use str.format !120552.

Submitted PR to use `str.format` !120552.

Sybren A. Stüvel commented

2024-04-22 14:22:24 +02:00

Comparing "%s %s %s" with "{:s} {:s} {:s}"

As I also mentioned in #120552, this is not an entirely fair comparison, as these two do not do the same thing. "%s %s %s" should be compared with "{!s} {!s} {!s}" (if you want to enforce the call to str()) or "{} {} {}" (if the arguments are already strings).

This was a concern previously but since we're no longer writing exporters in Python, the performance of formatting large amounts of data is less of an issue.

In those cases I think it would be fine to keep using %-formatting. And then there's of course also the opportunity for benchmarking, writing those results in a comment, and explaining the choices made. IMO such specific cases don't have to stand in the way of the majority of Blender's Python code switching to str.format.

> Comparing `"%s %s %s"` with `"{:s} {:s} {:s}"` As I also mentioned in #120552, this is not an entirely fair comparison, as these two do not do the same thing. `"%s %s %s"` should be compared with `"{!s} {!s} {!s}"` (if you want to enforce the call to `str()`) or `"{} {} {}"` (if the arguments are already strings). > This was a concern previously but since we're no longer writing exporters in Python, the performance of formatting large amounts of data is less of an issue. In those cases I think it would be fine to keep using `%`-formatting. And then there's of course also the opportunity for benchmarking, writing those results in a comment, and explaining the choices made. IMO such specific cases don't have to stand in the way of the majority of Blender's Python code switching to `str.format`.

Campbell Barton referenced this issue from a commit

2024-04-27 08:07:42 +02:00

Refactor: scripts under scripts/startup/ to use `str.format`

Campbell Barton referenced this issue from a commit

2024-04-27 09:00:21 +02:00

Update Python code guidelines

Campbell Barton commented

2024-04-27 09:46:14 +02:00

Comparing "%s %s %s" with "{:s} {:s} {:s}"

As I also mentioned in #120552, this is not an entirely fair comparison, as these two do not do the same thing. "%s %s %s" should be compared with "{!s} {!s} {!s}" (if you want to enforce the call to str()) or "{} {} {}" (if the arguments are already strings).

True, {:s} / {!s} / {} have different results when micro-bench marking (I don't have the numbers handy).
Sine the overall speed is so small I'm not that worried about this, although it's annoying to opt for a measurably slower option.

If this kind of performance was a concern though - I think there are quite a few ways we could when this back (RNA function calls can probably be optimized for e.g.).

This was a concern previously but since we're no longer writing exporters in Python, the performance of formatting large amounts of data is less of an issue.

In those cases I think it would be fine to keep using %-formatting. And then there's of course also the opportunity for benchmarking, writing those results in a comment, and explaining the choices made. IMO such specific cases don't have to stand in the way of the majority of Blender's Python code switching to str.format.

Right, even in this case though I think it's reasonable that there is a measurable real world difference that's not in the sub <10 millisecond realm.

> > Comparing `"%s %s %s"` with `"{:s} {:s} {:s}"` > > As I also mentioned in #120552, this is not an entirely fair comparison, as these two do not do the same thing. `"%s %s %s"` should be compared with `"{!s} {!s} {!s}"` (if you want to enforce the call to `str()`) or `"{} {} {}"` (if the arguments are already strings). True, `{:s}` / `{!s}` / `{}` have different results when micro-bench marking (I don't have the numbers handy). Sine the overall speed is so small I'm not that worried about this, although it's annoying to opt for a measurably slower option. If this kind of performance was a concern though - I think there are quite a few ways we could when this back (RNA function calls can probably be optimized for e.g.). > > This was a concern previously but since we're no longer writing exporters in Python, the performance of formatting large amounts of data is less of an issue. > > In those cases I think it would be fine to keep using `%`-formatting. And then there's of course also the opportunity for benchmarking, writing those results in a comment, and explaining the choices made. IMO such specific cases don't have to stand in the way of the majority of Blender's Python code switching to `str.format`. Right, even in this case though I think it's reasonable that there is a measurable real world difference that's not in the sub <10 millisecond realm.

Campbell Barton commented

2024-04-27 09:48:12 +02:00

Committed 0e3b594edb & updated the developer handbook, closing.

Committed 0e3b594edb910ccb811bd5ac1998868d9ec72ea1 & updated the developer handbook, closing.

Campbell Barton closed this issue

2024-04-27 09:48:22 +02:00

Blender Bot added the

Status

Archived

label 2024-04-27 09:48:31 +02:00

T-112 commented

2024-05-02 04:09:07 +02:00

@ideasman42 Just a note on the performance discrepancy mentioned for f-strings: It really depends on how they are used. Taking the test script from your SO question (https://stackoverflow.com/questions/62566526/percentage-formatting-twice-as-fast-as-f-strings-in-python-3-x?newreg=eda4a27f8a8647f6904f31dfa7f1c12b), if you remove the explicit string specifiers (e.g. {a:s} -> {a}, etc.), then f-strings are actually notably faster than the old modulo style syntax, and they are producing the same resulting string. There are probably cases where one or the other will be faster, depending on how in-depth the formatting specifiers are, and how many are used. For reference, these are the numbers I got when I made the above change (and also changed the .format() to not bother with :s as well):

Percentage formatting: '%s'
0.7048760000000001

'str.format' formatting: '%s'
0.6997130000000001

'f-string' formatting: '%s'
0.5384899999999999

This is on Python 3.12 (the superfluous '%s' in the output should just be ignored). Also, there are some other reputable Python learning source that contend that f-strings are faster: https://realpython.com/python-f-strings/#comparing-performance-f-string-vs-traditional-tools

@ideasman42 Just a note on the performance discrepancy mentioned for f-strings: It really depends on how they are used. Taking the test script from your SO question (https://stackoverflow.com/questions/62566526/percentage-formatting-twice-as-fast-as-f-strings-in-python-3-x?newreg=eda4a27f8a8647f6904f31dfa7f1c12b), if you remove the explicit string specifiers (e.g. `{a:s}` -> `{a}`, etc.), then f-strings are actually notably faster than the old modulo style syntax, and they are producing the same resulting string. There are probably cases where one or the other will be faster, depending on how in-depth the formatting specifiers are, and how many are used. For reference, these are the numbers I got when I made the above change (and also changed the `.format()` to not bother with `:s` as well): ``` Percentage formatting: '%s' 0.7048760000000001 'str.format' formatting: '%s' 0.6997130000000001 'f-string' formatting: '%s' 0.5384899999999999 ``` This is on Python 3.12 (the superfluous `'%s'` in the output should just be ignored). Also, there are some other reputable Python learning source that contend that f-strings are faster: https://realpython.com/python-f-strings/#comparing-performance-f-string-vs-traditional-tools

Campbell Barton commented

2024-05-02 06:02:59 +02:00

@T-112 this is interesting & strange that f-strings are faster without the :s. Replied to the stackoverflow post with these findings.

@T-112 this is interesting & strange that f-strings are faster without the `:s`. Replied to the stackoverflow post with these findings.

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Proposal: use str.format instead of percentage formatting for core-scripts #120453

Motivation

Pros & Cons

Proposal

Details

Rationale for str.format over f-strings

Other Notes