Video Output: Support higher bits per pixel formats #118493

New Issue

Sergey Sharybin · 2024-02-20T11:36:34+01:00

Sergey Sharybin commented

2024-02-20 11:36:34 +01:00

Currently the video output is hard-wired to use byte buffers (passed as int* structure, assuming integer is always 32bit). This is more of a historical code from the times when video files did not support anything beyond 8bits.

Nowadays, there are codecs which support 10 (and possibly above) bits depth, which is not possible to utilize within the video output API in Blender. This task is about unlocking such possibility.

There seems to be a few steps achieving the goal:

Make bMovieHandle::append_movie receive ImBuf instead of pixels, rectx , rectx
Ensure there is 10bit+ codec support compiled into FFmpeg shipped with Blender
- Ideally we'll add support of h265, but maybe there is a way to avoid such dependency between tasks
- h264 seems to support 10bit, but not all video players support if
- AV1 should support 10bit, so perhaps it is the codec to use to verify changes in the encoding code
Adopt BKE_ffmpeg_append to support float buffer from the ImBuf
Avoid unnecessary float -> byte conversions (if possible)
Figure out what sort of settings (e.g. 8 vs 10 vs 12 bit, HDR profiles, etc.) should be exposed in video UI

Possibly relevant notes:

https://projects.blender.org/nathanvegdahl/.profile/src/branch/main/notes/hdr_video_output.md
https://academysoftwarefoundation.github.io/EncodingGuidelines/
Overall issue wrt HDR workflows: #105714

Currently the video output is hard-wired to use byte buffers (passed as `int*` structure, assuming integer is always 32bit). This is more of a historical code from the times when video files did not support anything beyond 8bits. Nowadays, there are codecs which support 10 (and possibly above) bits depth, which is not possible to utilize within the video output API in Blender. This task is about unlocking such possibility. There seems to be a few steps achieving the goal: - [x] Make `bMovieHandle::append_movie` receive `ImBuf` instead of `pixels`, `rectx` , `rectx` - [x] Ensure there is 10bit+ codec support compiled into FFmpeg shipped with Blender - Ideally we'll add support of h265, but maybe there is a way to avoid such dependency between tasks - h264 seems to support 10bit, but not all video players support if - AV1 should support 10bit, so perhaps it is the codec to use to verify changes in the encoding code - [ ] Adopt `BKE_ffmpeg_append` to support float buffer from the `ImBuf` - [ ] Avoid unnecessary float -> byte conversions (if possible) - [ ] Figure out what sort of settings (e.g. 8 vs 10 vs 12 bit, HDR profiles, etc.) should be exposed in video UI Possibly relevant notes: - https://projects.blender.org/nathanvegdahl/.profile/src/branch/main/notes/hdr_video_output.md - https://academysoftwarefoundation.github.io/EncodingGuidelines/ - Overall issue wrt HDR workflows: #105714

Sergey Sharybin added the

labels 2024-02-20 11:36:34 +01:00

Sergey Sharybin referenced this issue

2024-02-20 11:37:01 +01:00

Supporting HDR workflows in Blender #105714

Nathan Vegdahl commented

2024-02-20 11:57:31 +01:00

@Sergey Here are the notes you asked me to post:
https://projects.blender.org/nathanvegdahl/.profile/src/branch/main/notes/hdr_video_output.md

There's nothing groundbreaking in there, but it summarizes what's needed to encode an HDR video file with ffmpeg, including necessary metadata so that it's recognized as HDR. @SimonThommes and I verified that the resulting files work when uploading to YouTube, at least.

Edit: one thing to note is that the zscale filter stuff shouldn't be necessary when encoding directly from Blender, since we can just handle the transfer functions and color space conversion directly. It's only needed on the command line due to the Blender-rendered files being linear Rec.709 EXR files.

@Sergey Here are the notes you asked me to post: https://projects.blender.org/nathanvegdahl/.profile/src/branch/main/notes/hdr_video_output.md There's nothing groundbreaking in there, but it summarizes what's needed to encode an HDR video file with ffmpeg, including necessary metadata so that it's recognized as HDR. @SimonThommes and I verified that the resulting files work when uploading to YouTube, at least. Edit: one thing to note is that the zscale filter stuff shouldn't be necessary when encoding directly from Blender, since we can just handle the transfer functions and color space conversion directly. It's only needed on the command line due to the Blender-rendered files being linear Rec.709 EXR files.

👍 2

Andrea Monzini commented

2024-02-20 15:00:39 +01:00

@Sergey
there are three main encoders for AV1 ( 10 bit can be enabled as encode options )

libaom-av1 ( compile FFmpeg with --enable-libaom )
SVT-AV1 ( compile FFmpeg with --enable-libsvtav1 )
rav1e ( compile FFmpeg with --enable-librav1e )

reference:
https://trac.ffmpeg.org/wiki/Encode/AV1

but i would consider the very interesting project:

SVT-AV1-PSY ( 10 bit default ):
https://github.com/gianni-rosato/svt-av1-psy?tab=readme-ov-file

with very good docs:
https://github.com/gianni-rosato/svt-av1-psy/blob/master/README.md

10 bit and HDR docs:
https://github.com/gianni-rosato/svt-av1-psy/blob/master/Docs/CommonQuestions.md#8-or-10-bit-encoding

the "full parameters description" link is broken , the correct one:
https://github.com/gianni-rosato/svt-av1-psy/blob/master/Docs/Parameters.md

@Sergey there are three main encoders for AV1 ( 10 bit can be enabled as encode options ) * libaom-av1 ( compile FFmpeg with --enable-libaom ) * SVT-AV1 ( compile FFmpeg with --enable-libsvtav1 ) * rav1e ( compile FFmpeg with --enable-librav1e ) reference: https://trac.ffmpeg.org/wiki/Encode/AV1 but i would consider the very interesting project: * SVT-AV1-PSY ( 10 bit default ): https://github.com/gianni-rosato/svt-av1-psy?tab=readme-ov-file with very good docs: https://github.com/gianni-rosato/svt-av1-psy/blob/master/README.md 10 bit and HDR docs: https://github.com/gianni-rosato/svt-av1-psy/blob/master/Docs/CommonQuestions.md#8-or-10-bit-encoding the "full parameters description" link is broken , the correct one: https://github.com/gianni-rosato/svt-av1-psy/blob/master/Docs/Parameters.md

👍 1

Aras Pranckevicius referenced this issue

2024-02-21 12:37:18 +01:00

Cleanup: Video: change bMovieHandle function to take ImBuf as input #118559

Aras Pranckevicius referenced this issue from a commit

2024-02-21 12:57:49 +01:00

Cleanup: Video: change bMovieHandle function to take ImBuf as input

Aras Pranckevicius commented

2024-02-21 13:26:30 +01:00

Ensure there is 10bit+ codec support compiled into FFmpeg shipped with Blender

Checked, and today in 4.2 main (at least on Windows), the compiled ffmpeg supports:

H.264: 10-bit (YUV 4:2:0)
AV1: 10-bit (YUV 4:2:0, 4:2:2, 4:4:4), and 12-bit (YUV 4:2:0, 4:2:2, 4:4:4)

So looks like it's while maybe something extra (like H.265) could be added, there's nothing missing to start tinkering around with more than 8-bit/channel video encoding.

> Ensure there is 10bit+ codec support compiled into FFmpeg shipped with Blender Checked, and today in 4.2 main (at least on Windows), the compiled ffmpeg supports: - H.264: 10-bit (YUV 4:2:0) - AV1: 10-bit (YUV 4:2:0, 4:2:2, 4:4:4), and 12-bit (YUV 4:2:0, 4:2:2, 4:4:4) So looks like it's while maybe something extra (like H.265) could be added, there's nothing missing to start tinkering around with more than 8-bit/channel video encoding.

👍 3

Sergey Sharybin referenced this issue

2024-03-18 09:51:13 +01:00

Library changes for Blender 4.2 #118455

Aras Pranckevicius self-assigned this 2024-03-22 09:57:34 +01:00

Aras Pranckevicius commented

2024-03-27 09:13:57 +01:00

Question about representation of "video format types" within Blender codebase.

Some parts of UI and logic, like "which bit depths does a file format support?" are purely based on ImageFormatData::imtype enum. Like for JPEG2000 it, as expected, shows 8, 12, 16 choices in the UI:
However, all video is just a single type, R_IMF_IMTYPE_FFMPEG and displayed as such within the UI. Which makes it hard to do decisions on, like "can this do HDR?" or "can this do more than 8 bpp?", since whether "ffmpeg video" can do that depends on other options, which are not encoded within imtype (and not even part of ImageFormatData struct).
However! The enum does contain entries like R_IMF_IMTYPE_H264, R_IMF_IMTYPE_AV1 and so on, they are just not used (?!), since any & all video just gets a "yeah this is ffmpeg" imtype.

So a question is: should video "codecs" get their own imtype? I think it might also make the UI of choices somewhat easier to use, like (a quick mockup):

Most of the needed enums already exist, they are just not being used. Why, I'm not entirely sure.

Question about representation of "video format types" within Blender codebase. - Some parts of UI and logic, like "which bit depths does a file format support?" are purely based on `ImageFormatData::imtype` enum. Like for JPEG2000 it, as expected, shows 8, 12, 16 choices in the UI: ![output-bpp.png](/attachments/e399b410-2b4f-4387-b161-070472a23322) - However, *all* video is just a single type, `R_IMF_IMTYPE_FFMPEG` and displayed as such within the UI. Which makes it hard to do decisions on, like "can this do HDR?" or "can this do more than 8 bpp?", since whether "ffmpeg video" can do that depends on other options, which are not encoded within `imtype` (and not even part of `ImageFormatData` struct). - However! The enum *does* contain entries like `R_IMF_IMTYPE_H264`, `R_IMF_IMTYPE_AV1` and so on, they are just not used (?!), since any & all video just gets a "yeah this is ffmpeg" imtype. So a question is: **should video "codecs" get their own imtype**? I think it might also make the UI of choices somewhat easier to use, like (a quick mockup): ![output-videos.png](/attachments/9a55149a-861f-4a20-bf4a-6e604bdc40de) Most of the needed enums already exist, they are just not being used. Why, I'm not entirely sure.

output-bpp.png

39 KiB

output-videos.png

44 KiB

Sergey Sharybin commented

2024-03-27 09:52:50 +01:00

The codecs should not get their own imtype. If they do, it will be a step back from what the UI was supposed to be moving to. You can see details in the 2.79 Release Notes.

More correctly would be to remove the unused enums, and their associated code paths.
Doing so will keep it easy to have presets for different types of outputs.

I am not sure it worth trying to unify the color depth setting, and I am even less sure the check can be reliably done based on static rules. From my understanding, things like bitness support in h265 depends on the h265's library build flags, so while we can have some static rules for our own libraries, those rules could become invalid for Blender packaged in different distros. So it feels like some probing of FFmpeg API would be needed to see what bitness is supported. Not sure it is possible, but if it is then it seems a better way to go.

The codecs should not get their own imtype. If they do, it will be a step back from what the UI was supposed to be moving to. You can see details in the [2.79 Release Notes](https://developer.blender.org/docs/release_notes/2.79/more_features/). More correctly would be to remove the unused enums, and their associated code paths. Doing so will keep it easy to have presets for different types of outputs. I am not sure it worth trying to unify the color depth setting, and I am even less sure the check can be reliably done based on static rules. From my understanding, things like bitness support in h265 depends on the h265's library build flags, so while we can have some static rules for our own libraries, those rules could become invalid for Blender packaged in different distros. So it feels like some probing of FFmpeg API would be needed to see what bitness is supported. Not sure it is possible, but if it is then it seems a better way to go.

👍 1

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Video Output: Support higher bits per pixel formats #118493