Fix #60947: ffmpeg video colors shifted/banded in some players #130021

Merged

Aras Pranckevicius merged 5 commits from aras_p/blender:ffmpeg-color-metadata into main 2024-11-14 09:04:55 +01:00

Conversation 19 Commits 5 Files Changed 2 +46 -3

Aras Pranckevicius commented 2024-11-08 11:02:33 +01:00

Member

Copy Link

Videos written out of blender were not explicitly indicating the color metadata (color range, primaries, TRC, color space). While some video players assume BT.709, some others do not.

Explicitly indicate BT.709 and limited ("mpeg") YUV range for video codecs that are YUV based (i.e. final image is non-RGB(A)). Overall now the code does the same as these command line ffmpeg parameters: -vf scale=out_color_matrix=bt709 -color_primaries 1 -color_trc 1 -colorspace 1.

Note that there's no need to setup write_colr option; ffmpeg since 4.4 (2020) does that by default when color metadata is provided.

Attached cc-test5.zip test file which is slightly modified repro project from the bug report.

Here's the rendered test file as PNG:

Here's VLC (Windows, 3.0.21) before this PR -- large color shifts. This is rendered into H.265 with perceptually lossless quality.

and VLC with this PR -- looks correct.

Here's ffplay (Windows, 7.1) before this PR -- mostly correct, a tiny bit of gray background darkening.

and ffplay with this PR -- looks correct.

Windows 10 "Movies & TV" app difference is very similar to VLC case.

Videos written out of blender were not explicitly indicating the color metadata (color range, primaries, TRC, color space). While some video players assume BT.709, some others do not. Explicitly indicate BT.709 and limited ("mpeg") YUV range for video codecs that are YUV based (i.e. final image is non-RGB(A)). Overall now the code does the same as these command line ffmpeg parameters: `-vf scale=out_color_matrix=bt709 -color_primaries 1 -color_trc 1 -colorspace 1`. Note that there's no need to setup `write_colr` option; ffmpeg since 4.4 (2020) does that by default when color metadata is provided. Attached `cc-test5.zip` test file which is slightly modified repro project from the bug report. Here's the rendered test file as PNG:  Here's VLC (Windows, 3.0.21) before this PR -- large color shifts. This is rendered into H.265 with perceptually lossless quality.  and VLC with this PR -- looks correct.  Here's ffplay (Windows, 7.1) before this PR -- mostly correct, a tiny bit of gray background darkening.  and ffplay with this PR -- looks correct.  Windows 10 "Movies & TV" app difference is very similar to VLC case.

cc-test5.zip

165 KiB

p8a.png

244 KiB

f-ffplay-h264b.png

233 KiB

f-vlc-h264b.png

234 KiB

f-ffplay-h264a.png

254 KiB

f-vlc-h264a.png

223 KiB

👍 1

Richard Antalik commented 2024-11-10 15:34:40 +01:00

Member

Copy Link

I was just checking this out, and I did specify these parameters to AVCodecParameters instead of AVCodecContext. I did not expect these to be at multiple places at once. Not sure why, since I know what we did with reading framerate...

That said, this PR does not seem to fix this issue when checking with ffplay and VLC.

Command from referenced issue seems to work best and probably is best from comments I read:
ffmpeg -i Original\ Color.png -vcodec libx264 -vf scale=out_color_matrix=bt709 -color_primaries 1 -color_trc 1 -colorspace 1 out/ref.mp4

So to me it seems, that we just need to figure out, what -vf scale=out_color_matrix=bt709 does exactly.

I was just checking this out, and I did specify these parameters to `AVCodecParameters` instead of `AVCodecContext`. I did not expect these to be at multiple places at once. Not sure why, since I know what we did with reading framerate... That said, this PR does not seem to fix this issue when checking with ffplay and VLC. Command from referenced issue seems to work best and probably is best from comments I read: `ffmpeg -i Original\ Color.png -vcodec libx264 -vf scale=out_color_matrix=bt709 -color_primaries 1 -color_trc 1 -colorspace 1 out/ref.mp4` So to me it seems, that we just need to figure out, what `-vf scale=out_color_matrix=bt709` does exactly.

Aras Pranckevicius changed title from Fix #60947: ffmpeg video colors shifted/banded in some players to WIP: Fix #60947: ffmpeg video colors shifted/banded in some players 2024-11-10 16:33:35 +01:00

Aras Pranckevicius commented 2024-11-10 16:37:34 +01:00

Author

Member

Copy Link

I was just checking this out, and I did specify these parameters to AVCodecParameters instead of AVCodecContext. I did not expect these to be at multiple places at once. Not sure why, since I know what we did with reading framerate...

I tried both, and there did not seem to be a difference between them. From the docs it feels like AVCodecParameters are supposed to be only "queried" and not "modified", but then the docs are not exactly clear nor detailed.

That said, this PR does not seem to fix this issue when checking with ffplay and VLC.

Yeah this is funny. From a quick test this seems to fix the issue with ffplay vs VLC on my Mac, but does not fix the issue on Windows. Aaaargh! I'll keep on looking (or I did some tests wrong).

But just for more info: what are your ffplay and VLC versions, on what OS, and on what GPU?

Command from referenced issue seems to work best and probably is best from comments I read:
ffmpeg -i Original\ Color.png -vcodec libx264 -vf scale=out_color_matrix=bt709 -color_primaries 1 -color_trc 1 -colorspace 1 out/ref.mp4
So to me it seems, that we just need to figure out, what -vf scale=out_color_matrix=bt709 does exactly.

I'll look into that, thanks.

> I was just checking this out, and I did specify these parameters to `AVCodecParameters` instead of `AVCodecContext`. I did not expect these to be at multiple places at once. Not sure why, since I know what we did with reading framerate... I tried both, and there did not seem to be a difference between them. From the docs it _feels_ like AVCodecParameters are supposed to be only "queried" and not "modified", but then the docs are not exactly clear nor detailed. > That said, this PR does not seem to fix this issue when checking with ffplay and VLC. Yeah this is funny. From a quick test this seems to fix the issue with ffplay vs VLC on my Mac, but does not fix the issue on Windows. Aaaargh! I'll keep on looking (or I did some tests wrong). But just for more info: what are your ffplay and VLC versions, on what OS, and on what GPU? > Command from referenced issue seems to work best and probably is best from comments I read: > `ffmpeg -i Original\ Color.png -vcodec libx264 -vf scale=out_color_matrix=bt709 -color_primaries 1 -color_trc 1 -colorspace 1 out/ref.mp4` > So to me it seems, that we just need to figure out, what `-vf scale=out_color_matrix=bt709` does exactly. I'll look into that, thanks.

Aras Pranckevicius force-pushed ffmpeg-color-metadata from a10f87583d to f411efc93a 2024-11-10 17:31:50 +01:00 Compare

Aras Pranckevicius changed title from WIP: Fix #60947: ffmpeg video colors shifted/banded in some players to Fix #60947: ffmpeg video colors shifted/banded in some players 2024-11-10 17:34:15 +01:00

Aras Pranckevicius commented 2024-11-10 17:44:23 +01:00

Author

Member

Copy Link

@iss I think now I have it working correctly. The missing -vf scale=out_color_matrix=bt709 part was color space details that need to be set for sws_scale

@iss I think _now_ I have it working correctly. The missing `-vf scale=out_color_matrix=bt709` part was color space details that need to be set for `sws_scale`

Aras Pranckevicius added 1 commit 2024-11-10 17:45:41 +01:00

Merge branch 'main' into ffmpeg-color-metadata ad182f1d57

Aras Pranckevicius commented 2024-11-10 17:46:06 +01:00

Author

Member

Copy Link

@blender-bot build

@blender-bot build

Aras Pranckevicius requested review from Richard Antalik 2024-11-10 19:43:11 +01:00

Aras Pranckevicius requested review from Sergey Sharybin 2024-11-10 19:43:16 +01:00

Sergey Sharybin reviewed 2024-11-11 11:14:20 +01:00

Sergey Sharybin left a comment

Owner

Copy Link

For the default OCIO configuration this makes sense.

What I am trying to understand figure out are a couple of things:

• What are the implications for someone who replaces the default config with, say, ACES. I don't think this situation is worse than compared to prior to this PR. But would be goo to have a second thought on this.
• Something for the future. Is the FFmpeg configuration limited to some harcoded list of primaries and ranges, or can we potentially provide custom primaries and matrices? Something to make the code agnostic to the OCIO configuration.

For the default OCIO configuration this makes sense. What I am trying to understand figure out are a couple of things: - What are the implications for someone who replaces the default config with, say, ACES. I don't think this situation is worse than compared to prior to this PR. But would be goo to have a second thought on this. - Something for the future. Is the FFmpeg configuration limited to some harcoded list of primaries and ranges, or can we potentially provide custom primaries and matrices? Something to make the code agnostic to the OCIO configuration.

Aras Pranckevicius commented 2024-11-11 12:04:51 +01:00

Author

Member

Copy Link

What I am trying to understand figure out are a couple of things:

• What are the implications for someone who replaces the default config with, say, ACES. I don't think this situation is worse than compared to prior to this PR. But would be goo to have a second thought on this.
• Something for the future. Is the FFmpeg configuration limited to some harcoded list of primaries and ranges, or can we potentially provide custom primaries and matrices? Something to make the code agnostic to the OCIO configuration.

Good questions! My understanding of all things color management is very hand-wavy, but here's what I think happens:

• The video codecs only allow expressing a certain set of things like color spaces or color primaries, not "anything could happen" that OCIO is probably able to do. E.g. color primaries are BT.709, BT.2020 and a handful of others; color spaces are BT.709, BT.2020 and a handful of others, color transfer ("encoding"?) is also just a handful of choices etc.
• However, everything that is part of video specification (if we ignore HDR videos for a moment) is effectively just the "display" part of the color management, I think. What happens with the colors before that can still be completely arbitrary and controlled by OCIO, like what sort of tonemapper you use, or what is your look transform, etc. etc. can still be "whatever".
• The current code specifies BT.709 "things", which as far as color primaries go are the same as sRGB. From a quick look, "P3" could also be expressed (in two different ways! DCI P3 or Display P3). Maybe that would be an option to investigate later.

> What I am trying to understand figure out are a couple of things: > - What are the implications for someone who replaces the default config with, say, ACES. I don't think this situation is worse than compared to prior to this PR. But would be goo to have a second thought on this. > - Something for the future. Is the FFmpeg configuration limited to some harcoded list of primaries and ranges, or can we potentially provide custom primaries and matrices? Something to make the code agnostic to the OCIO configuration. Good questions! My understanding of all things color management is _very_ hand-wavy, but here's what I think happens: - The video codecs only allow expressing a certain set of things like color spaces or color primaries, not "anything could happen" that OCIO is probably able to do. E.g. color primaries are BT.709, BT.2020 and a handful of others; color spaces are BT.709, BT.2020 and a handful of others, color transfer ("encoding"?) is also just a handful of choices etc. - However, everything that is part of video specification (if we ignore HDR videos for a moment) is effectively just the "display" part of the color management, I think. What happens with the colors before that can still be completely arbitrary and controlled by OCIO, like what sort of tonemapper you use, or what is your look transform, etc. etc. can still be "whatever". - The current code specifies BT.709 "things", which as far as color primaries go are the same as sRGB. From a quick look, "P3" could also be expressed (in two different ways! DCI P3 or Display P3). Maybe that would be an option to investigate later.

Sergey Sharybin approved these changes 2024-11-11 12:17:37 +01:00

Sergey Sharybin left a comment

Owner

Copy Link

Thanks for sharing the thoughts!

So the only possible downside is for people who use ACES config, and try to output video to something else than sRGB rec709. I'm not even sure if that is something people currently do, and whether it is really working reliably.

From fixing the standard Blender configuration I think it is a good step forward.

The more complete solution indeed would require some thinking, but I don't think we need to solve it now.

One thing I didn't mention is: is there a possibility of covering this with a render test somehow? The fact that "some player" (that is not Blender) is needed makes it kinda non-trivial =\

Thanks for sharing the thoughts! So the only possible downside is for people who use ACES config, and try to output video to something else than sRGB rec709. I'm not even sure if that is something people currently do, and whether it is really working reliably. From fixing the standard Blender configuration I think it is a good step forward. The more complete solution indeed would require some thinking, but I don't think we need to solve it now. One thing I didn't mention is: is there a possibility of covering this with a render test somehow? The fact that "some player" (that is not Blender) is needed makes it kinda non-trivial =\

Richard Antalik commented 2024-11-11 14:06:31 +01:00

Member

Copy Link

Perhaps, this could be for future PR, but Rec. 601, 709 and 2020 standards also define a movie resolutions - SD, HD, 4K/8K respectively. So I think, that we should either make a property for which encoding is to be used or switch encoding based on movie resolution. In case of automatic switching, question would be what to do about non-standard resolutions though.

Not sure how/if Rec.709 and Rec.2020 are "compatible". My point is, that we could avoid report like "Colors in my 4K render are off".

Perhaps, this could be for future PR, but Rec. 601, 709 and 2020 standards also define a movie resolutions - SD, HD, 4K/8K respectively. So I think, that we should either make a property for which encoding is to be used or switch encoding based on movie resolution. In case of automatic switching, question would be what to do about non-standard resolutions though. Not sure how/if Rec.709 and Rec.2020 are "compatible". My point is, that we could avoid report like "Colors in my 4K render are off".

Richard Antalik commented 2024-11-11 14:23:58 +01:00

Member

Copy Link

Tested PR now. The colors are correct in ffplay, but importing rendered movie back to VSE causes color shift. I suspect, that we also need to correct Rec.601 assumed colorspace when converting to RGB. Which could involve another round of AVStream vs AVCodecContext dichotomy lol.

Tested PR now. The colors are correct in ffplay, but importing rendered movie back to VSE causes color shift. I suspect, that we also need to correct Rec.601 assumed colorspace when converting to RGB. Which could involve another round of `AVStream` vs `AVCodecContext` dichotomy lol.

Sergey Sharybin commented 2024-11-11 14:47:55 +01:00

Owner

Copy Link

So I think, that we should either make a property for which encoding is to be used or switch encoding based on movie resolution. In case of automatic switching, question would be what to do about non-standard resolutions though.

While standards like BT.709-6 define image resolution, frame rate, chromaticities, those are things that can/ should be tackled separately.

The chromaticities are defined by the color management (in our case, the config.ocio). They define how colors are interpreted, transformed, displayed etc. The resolution is just the number of pixels in the image.

I suspect, that we also need to correct Rec.601 assumed colorspace when converting to RGB

Where does the Rec.601 come from? Standard Blender configuration uses Rec.709 primaries.

> So I think, that we should either make a property for which encoding is to be used or switch encoding based on movie resolution. In case of automatic switching, question would be what to do about non-standard resolutions though. While standards like `BT.709-6` define image resolution, frame rate, chromaticities, those are things that can/ should be tackled separately. The chromaticities are defined by the color management (in our case, the `config.ocio`). They define how colors are interpreted, transformed, displayed etc. The resolution is just the number of pixels in the image. > I suspect, that we also need to correct Rec.601 assumed colorspace when converting to RGB Where does the Rec.601 come from? Standard Blender configuration uses Rec.709 primaries.

Richard Antalik commented 2024-11-11 14:59:11 +01:00

Member

Copy Link

To me it sounds that these standards define a "thing" that has particular properties. So you can't mix and match properties of multiple standards in one "thing".

Where does the Rec.601 come from? Standard Blender configuration uses Rec.709 primaries.

AFAIK it is ffmpeg assumption, but that should happen only for untagged files, would have to double check that

To me it sounds that these standards define a "thing" that has particular properties. So you can't mix and match properties of multiple standards in one "thing". > Where does the Rec.601 come from? Standard Blender configuration uses Rec.709 primaries. AFAIK it is ffmpeg assumption, but that should happen only for untagged files, would have to double check that

Sergey Sharybin commented 2024-11-11 17:27:13 +01:00

Owner

Copy Link

To me it sounds that these standards define a "thing" that has particular properties. So you can't mix and match properties of multiple standards in one "thing".

There mighr indeed some confusion. But when we're talking about colors, it is mainly about the chromaticities and such. it is not very handy to refer to the rest of the parameters like resolution or framerate. Outside of whatever strict cinema systems computer can play back almost everything. But it needs to know information colors.

AFAIK it is ffmpeg assumption, but that should happen only for untagged files, would have to double check that

If that's the case, then our writing code is wrong?
Also, a bit strange that FFmpeg will assume rec.601 when we explicitly set rec.709.

> To me it sounds that these standards define a "thing" that has particular properties. So you can't mix and match properties of multiple standards in one "thing". There mighr indeed some confusion. But when we're talking about colors, it is mainly about the chromaticities and such. it is not very handy to refer to the rest of the parameters like resolution or framerate. Outside of whatever strict cinema systems computer can play back almost everything. But it needs to know information colors. > AFAIK it is ffmpeg assumption, but that should happen only for untagged files, would have to double check that If that's the case, then our writing code is wrong? Also, a bit strange that FFmpeg will assume rec.601 when we explicitly set rec.709.

Aras Pranckevicius changed title from Fix #60947: ffmpeg video colors shifted/banded in some players to WIP: Fix #60947: ffmpeg video colors shifted/banded in some players 2024-11-11 19:38:58 +01:00

Aras Pranckevicius commented 2024-11-11 19:40:27 +01:00

Author

Member

Copy Link

I think I have a theory why decoding videos that explicitly have colorspace metadata goes wrong (and that's not a new issue; seemingly has been that way forever -- will also add test coverage). The issue so far seems to be that the YUV->RGB conversion done by sws_scale at decoding time does not get the colorspace information from the AVFrame into the sws_scale part.

Stay tuned! Nice catch Richard on this issue.

I think I have a theory why decoding videos that explicitly have colorspace metadata goes wrong (and that's not a new issue; seemingly has been that way forever -- will also add test coverage). The issue so far seems to be that the YUV->RGB conversion done by sws_scale at decoding time does not get the colorspace information from the AVFrame into the sws_scale part. Stay tuned! Nice catch Richard on this issue.

👍 1

Aras Pranckevicius commented 2024-11-12 18:52:05 +01:00

Author

Member

Copy Link

Tested PR now. The colors are correct in ffplay, but importing rendered movie back to VSE causes color shift. I suspect, that we also need to correct Rec.601 assumed colorspace when converting to RGB. Which could involve another round of AVStream vs AVCodecContext dichotomy lol.

Actually... I can't reproduce this :( @iss do you have details in how does it reproduce for you?

Initially I thought I could reproduce it with movies produced by command line ffmpeg that explicitly indicate "yeah I'm BT.709", but turns out I was using ffmpeg command line incorrectly.

I did add a bunch of render test coverage with movies using various color space settings & whatnot (not comitted yet), and they also confirm that seemingly there's no problem. What I thought was a problem (sws_scale), was just my confusion; it gets both the YUV range and the colorspace correctly.

> Tested PR now. The colors are correct in ffplay, but importing rendered movie back to VSE causes color shift. I suspect, that we also need to correct Rec.601 assumed colorspace when converting to RGB. Which could involve another round of `AVStream` vs `AVCodecContext` dichotomy lol. Actually... I can't reproduce this :( @iss do you have details in how does it reproduce for you? Initially I thought I could reproduce it with movies produced by command line ffmpeg that explicitly indicate "yeah I'm BT.709", but turns out I was using ffmpeg command line incorrectly. I did add a bunch of render test coverage with movies using various color space settings & whatnot (not comitted yet), and they also confirm that seemingly there's no problem. What I thought was a problem (sws_scale), was just my confusion; it gets both the YUV range and the colorspace correctly.

Sergey Sharybin commented 2024-11-13 13:57:16 +01:00

Owner

Copy Link

@aras_p Sorry, wall-o-text coming...

I've attached a test scene which, I believe, demonstrates the issue Richard run into. It might another issue, but an issue nevertheless :)

Steps to reprodce:

• Render animation
• Add rendered movie file to the same timeline
• Toggle movie strip visiiblity to see the difference in colors

Some important observations:

• Same issue happens in the main branch before this PR.
• The issue is coming from the use of proxies.If you disable proxies then colors of the original PNG and rendered movie looks the same when added to the sequencer.

@aras_p Sorry, wall-o-text coming... I've attached a test scene which, I believe, demonstrates the issue Richard run into. It might another issue, but an issue nevertheless :) Steps to reprodce: - Render animation - Add rendered movie file to the same timeline - Toggle movie strip visiiblity to see the difference in colors Some important observations: - Same issue happens in the `main` branch before this PR. - The issue is coming from the use of proxies.If you disable proxies then colors of the original PNG and rendered movie looks the same when added to the sequencer.

test.blend

277 KiB

p8a.png

244 KiB

Aras Pranckevicius commented 2024-11-13 14:23:22 +01:00

Author

Member

Copy Link

• Same issue happens in the main branch before this PR.

Ah, so this is not something caused by this PR? What Richard said sounded like an issue introduced by this PR. That said...

• The issue is coming from the use of proxies.If you disable proxies then colors of the original PNG and rendered movie looks the same when added to the sequencer.

I can't reproduce the issue with your file. I have verified that proxies work (by explicitly trying proxy resolution & quality settings - I can see lower res & at low quality I see compression artifacts). But I don't see the color changes.

I'm testing on both 4.3 and current main. Windows 10, regular sRGB monitor. Which brings the question: which OS do you see this on, and what kind of monitor?

> - Same issue happens in the `main` branch before this PR. Ah, so this is not something caused by this PR? What Richard said sounded like an issue introduced by this PR. That said... > - The issue is coming from the use of proxies.If you disable proxies then colors of the original PNG and rendered movie looks the same when added to the sequencer. I can't reproduce the issue with your file. I have verified that proxies work (by explicitly trying proxy resolution & quality settings - I can see lower res & at low quality I see compression artifacts). But I don't see the color changes. I'm testing on both 4.3 and current main. Windows 10, regular sRGB monitor. Which brings the question: which OS do you see this on, and what kind of monitor?

Sergey Sharybin commented 2024-11-13 14:59:00 +01:00

Owner

Copy Link

Ah, so this is not something caused by this PR? What Richard said sounded like an issue introduced by this PR. That said...

On more tests, I might be wrong. I can no longer reproduce the issue on main. Maybe I forgot to delete BL_Proxy folder? Or, deleted something else by accident?

I'm on Apple M2 Ultra, macOS 14.6.1, using XDR screen.

> Ah, so this is not something caused by this PR? What Richard said sounded like an issue introduced by this PR. That said... On more tests, I might be wrong. I can no longer reproduce the issue on `main`. Maybe I forgot to delete `BL_Proxy` folder? Or, deleted something else by accident? I'm on Apple M2 Ultra, macOS 14.6.1, using XDR screen.

Richard Antalik commented 2024-11-13 17:34:13 +01:00

Member

Copy Link

Tested PR now. The colors are correct in ffplay, but importing rendered movie back to VSE causes color shift. I suspect, that we also need to correct Rec.601 assumed colorspace when converting to RGB. Which could involve another round of AVStream vs AVCodecContext dichotomy lol.

Actually... I can't reproduce this :( @iss do you have details in how does it reproduce for you?

I used png file from #127663.
Steps to repro:

• Add image to timeline
• Render movie
• Add movie to timeline
• Compare images visually - there is very clear difference

Here are the files, with movie rendered on my machine, just in case. This does not happen in main. Also no proxies involved.

Speaking of proxies, We should add colorspace/encoding metadata to these as well.

> > Tested PR now. The colors are correct in ffplay, but importing rendered movie back to VSE causes color shift. I suspect, that we also need to correct Rec.601 assumed colorspace when converting to RGB. Which could involve another round of `AVStream` vs `AVCodecContext` dichotomy lol. > > Actually... I can't reproduce this :( @iss do you have details in how does it reproduce for you? I used png file from #127663. Steps to repro: - Add image to timeline - Render movie - Add movie to timeline - Compare images visually - there is very clear difference Here are the files, with movie rendered on my machine, just in case. This does not happen in main. Also no proxies involved. Speaking of proxies, We should add colorspace/encoding metadata to these as well.

test.zip

1.1 MiB

Aras Pranckevicius commented 2024-11-13 17:46:10 +01:00

Author

Member

Copy Link

Here are the files, with movie rendered on my machine, just in case. This does not happen in main. Also no proxies involved.
Speaking of proxies, We should add colorspace/encoding metadata to these as well.

There are proxies involved, and they are exactly the problem (proxies are involved by default; if you turn off "use proxies" in the preview N panel then the problem goes away).

And your last sentence is exactly the solution, actually. The problem is that proxies are built by using or scaling AVFrame contents directly (without going through YUV->RGB->YUV steps), but if the source video contains colorspace information, then the proxy file will not contain the colorspace metadata. Yet the actual frame data will be in that particular colorspace.

This means that the "bug with proxies" has existed for a long time, if your source video clips are in some specific colorspace. I think I have a fix, hold on!

(phew, this bug has confused me at least 7 times this week...)

> Here are the files, with movie rendered on my machine, just in case. This does not happen in main. Also no proxies involved. > Speaking of proxies, We should add colorspace/encoding metadata to these as well. There *are* proxies involved, and they are exactly the problem (proxies are involved by default; if you turn off "use proxies" in the preview N panel then the problem goes away). And your last sentence is exactly the solution, actually. The problem is that proxies are built by using or scaling AVFrame contents directly (without going through YUV->RGB->YUV steps), but if the source video contains colorspace information, then the proxy file will not contain the colorspace metadata. Yet the actual frame data will be in that particular colorspace. This means that the "bug with proxies" has existed for a long time, if your source video clips are in some specific colorspace. I think I have a fix, hold on! _(phew, this bug has confused me at least 7 times this week...)_

Aras Pranckevicius added 2 commits 2024-11-13 17:56:35 +01:00

Merge branch 'main' into ffmpeg-color-metadata 0bb50909d6

Fix: VSE movie proxies have color shifts when source movie has colorspace info ... 1ca47d4e83

Proxies are built by directly using decoded AVFrame data (possibly
downscaling it). However if source movie contained colorspace
metadata, the resulting proxy was not containing it, leading to
incorrect decoding.

Aras Pranckevicius commented 2024-11-13 17:57:03 +01:00

Author

Member

Copy Link

@Sergey @iss ok now the issue with proxies should be fixed.

@Sergey @iss ok *now* the issue with proxies should be fixed.

Sergey Sharybin reviewed 2024-11-13 18:29:59 +01:00

Sergey Sharybin left a comment

Owner

Copy Link

I don't fully understand why metadata on the source file would break colors in the proxy.

With the test.blend from my previous post I've rendered it in main, in the PR, pulled both renders into VSE without proxies. Their color matched. Just to confirm that there wouldn't be some regressions from the way how Blender handles videos.

One thing I've notices is that in Finder on macOS the original p8a.png is darker than the video rendered from test.blend with this PR applied. The colors of the videos after this PR looks more correct.
We don't necessarily have to fix all the issues with this PR, it was the easiest way I could verify colors outside of Blender. If this PR makes things correct with some players (vlc? ffplay? maybe other editing software?) then we don't need to make thing more complicated than they should and can land this PR.

I don't fully understand why metadata on the source file would break colors in the proxy. With the `test.blend` from my previous post I've rendered it in `main`, in the PR, pulled both renders into VSE without proxies. Their color matched. Just to confirm that there wouldn't be some regressions from the way how Blender handles videos. One thing I've notices is that in Finder on macOS the original `p8a.png` is darker than the video rendered from `test.blend` with this PR applied. The colors of the videos after this PR looks more correct. We don't necessarily have to fix all the issues with this PR, it was the easiest way I could verify colors outside of Blender. If this PR makes things correct with some players (vlc? ffplay? maybe other editing software?) then we don't need to make thing more complicated than they should and can land this PR.

Aras Pranckevicius changed title from WIP: Fix #60947: ffmpeg video colors shifted/banded in some players to Fix #60947: ffmpeg video colors shifted/banded in some players 2024-11-13 20:35:14 +01:00

Richard Antalik commented 2024-11-13 21:05:23 +01:00

Member

Copy Link

Here are the files, with movie rendered on my machine, just in case. This does not happen in main. Also no proxies involved.
Speaking of proxies, We should add colorspace/encoding metadata to these as well.

There are proxies involved, and they are exactly the problem (proxies are involved by default; if you turn off "use proxies" in the preview N panel then the problem goes away).

Ah sorry, I have checked incorrect location, you are correct, the proxy was a cause. Also did not notice building process because file was so small. That could be bug in logic for automatic proxy building if there are not enough frames to do heuristic analysis. Will check that.

I don't fully understand why metadata on the source file would break colors in the proxy.

I wanted to say, that it's the mismatch of metadata between proxy and original, but actually that is weird, because proxy should have same metadata as render in main, which is displayed correctly.

Looking at the code, it could be, because sws_scale did change the colorspace, but file was still tagged as bt.601.

We don't necessarily have to fix all the issues with this PR, it was the easiest way I could verify colors outside of Blender. If this PR makes things correct with some players (vlc? ffplay? maybe other editing software?) then we don't need to make thing more complicated than they should and can land this PR.

Sure, the issue I raised originally looked like regression. I will test this early tomorrow, but would say, go ahead and merge if you have both tested this.

> > Here are the files, with movie rendered on my machine, just in case. This does not happen in main. Also no proxies involved. > > Speaking of proxies, We should add colorspace/encoding metadata to these as well. > > There *are* proxies involved, and they are exactly the problem (proxies are involved by default; if you turn off "use proxies" in the preview N panel then the problem goes away). Ah sorry, I have checked incorrect location, you are correct, the proxy was a cause. Also did not notice building process because file was so small. That could be bug in logic for automatic proxy building if there are not enough frames to do heuristic analysis. Will check that. > I don't fully understand why metadata on the source file would break colors in the proxy. I wanted to say, that it's the mismatch of metadata between proxy and original, but actually that is weird, because proxy should have same metadata as render in main, which is displayed correctly. Looking at the code, it could be, because `sws_scale` did change the colorspace, but file was still tagged as bt.601. > We don't necessarily have to fix all the issues with this PR, it was the easiest way I could verify colors outside of Blender. If this PR makes things correct with some players (vlc? ffplay? maybe other editing software?) then we don't need to make thing more complicated than they should and can land this PR. Sure, the issue I raised originally looked like regression. I will test this early tomorrow, but would say, go ahead and merge if you have both tested this.

Richard Antalik approved these changes 2024-11-14 06:59:20 +01:00

Aras Pranckevicius merged commit 527e33cef6 into main 2024-11-14 09:04:55 +01:00

Aras Pranckevicius deleted branch ffmpeg-color-metadata 2024-11-14 09:04:59 +01:00

Aras Pranckevicius referenced this issue from a commit 2024-11-14 09:05:03 +01:00

Fix #60947: ffmpeg video colors shifted/banded in some players

Sign in to join this conversation.

Reviewers

No reviewers

Sergey Sharybin

Richard Antalik

Labels

Clear labels   

Interest

Alembic

  

Interest

Animation & Rigging

  

Interest

Asset System

Asset datablocks, libraries, browser and shelf

  

Interest

Audio

  

Interest

Automated Testing

  

Interest

Blender Asset Bundle

  

Interest

BlendFile

  

Interest

Code Documentation

Code comments, Python/RNA API descriptions

  

Interest

Collada

  

Interest

Compatibility

Backward and forward compatibility

  

Interest

Compositing

  

Interest

Core

  

Interest

Cycles

  

Interest

Dependency Graph

  

Interest

Development Management

  

Interest

EEVEE

  

Interest

FBX

FBX I/O related topics

  

Interest

Freestyle

  

Interest

Geometry Nodes

  

Interest

glTF

glTF format I/O topics

  

Interest

Grease Pencil

  

Interest

ID Management

  

Interest

Images & Movies

  

Interest

Import Export

  

Interest

Line Art

  

Interest

Masking

  

Interest

Metal

  

Interest

Modeling

  

Interest

Modifiers

  

Interest

Motion Tracking

  

Interest

Nodes & Physics

  

Interest

OpenGL

  

Interest

Overlay

  

Interest

Overrides

  

Interest

Performance

  

Interest

Physics

  

Interest

Pipeline & IO

  

Interest

Platforms, Builds & Tests

  

Interest

Python API

  

Interest

Render & Cycles

  

Interest

Render Pipeline

  

Interest

Sculpt, Paint & Texture

  

Interest

Text Editor

  

Interest

Translations

  

Interest

Triaging

  

Interest

Undo

  

Interest

USD

  

Interest

User Interface

  

Interest

UV Editing

  

Interest

VFX & Video

  

Interest

Video Sequencer

  

Interest

Viewport & EEVEE

  

Interest

Virtual Reality

  

Interest

Vulkan

  

Interest

Wayland

Wayland windowing on Unix

  

Interest

Workbench

  

Interest: X11

Xorg/X11 windowing

  

Legacy

Asset Browser Project

Archived

  

Legacy

Blender 2.8 Project

Archived

  

Legacy

Milestone 1: Basic, Local Asset Browser

Archived

  

Legacy

OpenGL Error

Archived

  

Meta

Good First Issue

  

Meta

Papercut

  

Meta

Retrospective

  

Meta

Security

Related to security, see policy: https://developer.blender.org/docs/handbook/bug_reports/vulnerability_reports/

  

Module

Animation & Rigging

  

Module

Asset System

  

Module

Core

  

Module

Development Management

  

Module

Grease Pencil

  

Module

Modeling

  

Module

Nodes & Physics

  

Module

Pipeline & IO

  

Module

Platforms, Builds & Tests

  

Module

Python API

  

Module

Render & Cycles

  

Module

Sculpt, Paint & Texture

  

Module

Triaging

  

Module

User Interface

  

Module

VFX & Video

  

Module

Viewport & EEVEE

  

Platform

FreeBSD

  

Platform

Linux

  

Platform

macOS

  

Platform

Windows

  

Severity

High

  

Severity

Low

  

Severity

Normal

  

Severity

Unbreak Now!

  

Status

Archived

  

Status

Confirmed

  

Status

Duplicate

  

Status

Needs Info from Developers

  

Status

Needs Information from User

  

Status

Needs Triage

  

Status

Resolved

  

Type

Bug

  

Type

Design

  

Type

Known Issue

  

Type

Patch

Archived

  

Type

Report

Archived

  

Type

To Do

No Label

Interest

Alembic

Interest

Animation & Rigging

Interest

Asset System

Interest

Audio

Interest

Automated Testing

Interest

Blender Asset Bundle

Interest

BlendFile

Interest

Code Documentation

Interest

Collada

Interest

Compatibility

Interest

Compositing

Interest

Core

Interest

Cycles

Interest

Dependency Graph

Interest

Development Management

Interest

EEVEE

Interest

FBX

Interest

Freestyle

Interest

Geometry Nodes

Interest

glTF

Interest

Grease Pencil

Interest

ID Management

Interest

Images & Movies

Interest

Import Export

Interest

Line Art

Interest

Masking

Interest

Metal

Interest

Modeling

Interest

Modifiers

Interest

Motion Tracking

Interest

Nodes & Physics

Interest

OpenGL

Interest

Overlay

Interest

Overrides

Interest

Performance

Interest

Physics

Interest

Pipeline & IO

Interest

Platforms, Builds & Tests

Interest

Python API

Interest

Render & Cycles

Interest

Render Pipeline

Interest

Sculpt, Paint & Texture

Interest

Text Editor

Interest

Translations

Interest

Triaging

Interest

Undo

Interest

USD

Interest

User Interface

Interest

UV Editing

Interest

VFX & Video

Interest

Video Sequencer

Interest

Viewport & EEVEE

Interest

Virtual Reality

Interest

Vulkan

Interest

Wayland

Interest

Workbench

Interest: X11

Legacy

Asset Browser Project

Legacy

Blender 2.8 Project

Legacy

Milestone 1: Basic, Local Asset Browser

Legacy

OpenGL Error

Meta

Good First Issue

Meta

Papercut

Meta

Retrospective

Meta

Security

Module

Animation & Rigging

Module

Asset System

Module

Core

Module

Development Management

Module

Grease Pencil

Module

Modeling

Module

Nodes & Physics

Module

Pipeline & IO

Module

Platforms, Builds & Tests

Module

Python API

Module

Render & Cycles

Module

Sculpt, Paint & Texture

Module

Triaging

Module

User Interface

Module

VFX & Video

Module

Viewport & EEVEE

Platform

FreeBSD

Platform

Linux

Platform

macOS

Platform

Windows

Severity

High

Severity

Low

Severity

Normal

Severity

Unbreak Now!

Status

Archived

Status

Confirmed

Status

Duplicate

Status

Needs Info from Developers

Status

Needs Information from User

Status

Needs Triage

Status

Resolved

Type

Bug

Type

Design

Type

Known Issue

Type

Patch

Type

Report

Type

To Do

Milestone

Clear milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

No Assignees

3 Participants

Notifications

Subscribe

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender#130021

No description provided.