VSE 2.0: Media transform redesign #78988

New Issue

Sergey Sharybin · 2020-07-16T14:31:18+02:00

Sergey Sharybin commented

2020-07-16 14:31:18 +02:00

Current behavior of the media added to the sequencer stack: it gets stretched to fill the entire resolution of the final frame. In order to transform the media some special option is to be enabled, which makes media to have its original resolution in the final result. This is not very intuitive and rather cumbersome.

More expected behavior is to make media to scale-to-fit the final sequencer resolution, and consider this a baseline. All the transformation (scale, position, crop) are relative to this size.

For example, if sequencer is configured to be FullHD, adding 4K image will scale it down, so artists see the entire image. Scaling the image by 0.5 makes it so it covers half of the final frame. Transform operates in the final frame space.

Crop tool should not move the image, it should just remove pixels from sides, leaving rest of the pixels where they are (to compare with the current behavior, the desired crop is a combination of current crop plus current translation).

Mockup from the art team:

Amending change tasks:

#82755 (Sequencer: Improve motion-picture workflow)
#84535 (Resolve inconsistency of sequencer import setting in header)

Current behavior of the media added to the sequencer stack: it gets stretched to fill the entire resolution of the final frame. In order to transform the media some special option is to be enabled, which makes media to have its original resolution in the final result. This is not very intuitive and rather cumbersome. More expected behavior is to make media to scale-to-fit the final sequencer resolution, and consider this a baseline. All the transformation (scale, position, crop) are relative to this size. For example, if sequencer is configured to be FullHD, adding 4K image will scale it down, so artists see the entire image. Scaling the image by 0.5 makes it so it covers half of the final frame. Transform operates in the final frame space. Crop tool should not move the image, it should just remove pixels from sides, leaving rest of the pixels where they are (to compare with the current behavior, the desired crop is a combination of current crop plus current translation). Mockup from the art team: ![vse_improvements_02.jpg](https://archive.blender.org/developer/F8699080/vse_improvements_02.jpg) Amending change tasks: - #82755 (Sequencer: Improve motion-picture workflow) - #84535 (Resolve inconsistency of sequencer import setting in header)

Sergey Sharybin commented

2020-07-16 14:31:18 +02:00

Changed status from 'Needs Triage' to: 'Confirmed'

Sergey Sharybin commented

2020-07-16 14:31:18 +02:00

Added subscribers: @Sergey, @fsiddi, @iss, @dfelinto

Braden Currah commented

2020-07-17 23:50:56 +02:00

Added subscriber: @TakingFire

Richard Antalik commented

2020-07-27 09:18:14 +02:00

@Sergey, @fsiddi I have started working on this issue recently, and arrived at similar design with few differences and open questions:

Do we need to expose anchor point for scale and rotation? Alternative would be to use operator to manipulate with image interactively in preview and calculate correct position, rotation and scale with anchor pernamently set to image center.
I expect with current pipeline and WIP patch to use nearest interpolation for preview (fastest). We can chose bilinear or bicubic for render. Should we pick highest quality possible (slowest)? Or should we expose this option to user?
Should we be able to use relative values for pixels?
I think we no longer need "Use Crop" checkbox - it is using same codepath as "Offset", so it has no impact on performance. Should it be removed and should we use only seq->strip->transform?
To initialize scale to fit, when strips are added to timeline, I think we should flag strips with SEQ_FLAG_SCALE_NOT_CALCULATED and calculate scale on demand.

Over all, I would ask, how this feature should be different to existing transform effect strip?

Ideally we should be able to perform all transform operaions at once, I have used code from transform effect (2D matrix transform). However, it is measurably slower, than current implementation:

2D matrix transform: 62ms
Current scaling method: ~1ms

Therefore I think it would be better to use matrix transform code only when rotation is used. Scaling can be probably implemented in faster method as well.

WIP patch (made before this task was created, so it's not as proposed) D8393: Media transform redesign

@Sergey, @fsiddi I have started working on this issue recently, and arrived at similar design with few differences and open questions: - Do we need to expose anchor point for scale and rotation? Alternative would be to use operator to manipulate with image interactively in preview and calculate correct position, rotation and scale with anchor pernamently set to image center. - I expect with current pipeline and WIP patch to use nearest interpolation for preview (fastest). We can chose bilinear or bicubic for render. Should we pick highest quality possible (slowest)? Or should we expose this option to user? - Should we be able to use relative values for pixels? - I think we no longer need "Use Crop" checkbox - it is using same codepath as "Offset", so it has no impact on performance. Should it be removed and should we use only `seq->strip->transform`? - To initialize scale to fit, when strips are added to timeline, I think we should flag strips with `SEQ_FLAG_SCALE_NOT_CALCULATED` and calculate scale on demand. Over all, I would ask, how this feature should be different to existing transform effect strip? Ideally we should be able to perform all transform operaions at once, I have used code from transform effect (2D matrix transform). However, it is measurably slower, than current implementation: - 2D matrix transform: 62ms - Current scaling method: ~1ms Therefore I think it would be better to use matrix transform code only when rotation is used. Scaling can be probably implemented in faster method as well. WIP patch (made before this task was created, so it's not as proposed) [D8393: Media transform redesign](https://archive.blender.org/developer/D8393)

Sergey Sharybin commented

2020-07-27 10:03:06 +02:00

Do we need to expose anchor point for scale and rotation?

General programming approach: do not create entities unless is absolutely needed.
If the anchor point is just a slider, it will be really counter-intuitive to use. If it's a tool to scale and rotate, it can use translation for anchor.

I expect with current pipeline and WIP patch to use nearest interpolation for preview (fastest). We can chose bilinear or bicubic for render. Should we pick highest quality possible (slowest)? Or should we expose this option to user?

Please stop exposing settings to user. This is not a solution to anything.

Should we be able to use relative values for pixels?

Not sure what you mean by this. Users might still operate in the pixel values measured in the final 100% resolution. But the system should work in a way that scaling final resolution up or down the system should transparently scale pixel values accordingly.

Long story short: rendering 4K movie from 2K scene should be as simple as setting render scale to 200%.

I think we no longer need "Use Crop" checkbox

Once the old behavior of image fitting is removed, there is no need in Use Crop option at all. It is super easy to check crop values to be 0 and avoid any processing of the image in this case.

To initialize scale to fit, when strips are added to timeline, I think we should flag strips with SEQ_FLAG_SCALE_NOT_CALCULATED and calculate scale on demand.

This is an implementation detail.

It also seems too fragile, because the resolution in the file can change outside of Blender. The system could be smart enough to always do proper fitting.

The scale for fitting is purely runtime, it is not a state, and is not something users are interacting with.

Over all, I would ask, how this feature should be different to existing transform effect strip?

Transofmration is rather very fundamental operation during video editing. Having it as an effect seems weird. While it might work the same, requirement to always add effect every time one need to translate seems an overkill.

However, it is measurably slower, than current implementation

The new code is non-vectorized single-threaded implementation. This is not how one does real-time video processing tool in 2020.

> Do we need to expose anchor point for scale and rotation? General programming approach: do not create entities unless is absolutely needed. If the anchor point is just a slider, it will be really counter-intuitive to use. If it's a tool to scale and rotate, it can use translation for anchor. > I expect with current pipeline and WIP patch to use nearest interpolation for preview (fastest). We can chose bilinear or bicubic for render. Should we pick highest quality possible (slowest)? Or should we expose this option to user? Please stop exposing settings to user. This is not a solution to anything. > Should we be able to use relative values for pixels? Not sure what you mean by this. Users might still operate in the pixel values measured in the final 100% resolution. But the system should work in a way that scaling final resolution up or down the system should transparently scale pixel values accordingly. Long story short: rendering 4K movie from 2K scene should be as simple as setting render scale to 200%. > I think we no longer need "Use Crop" checkbox Once the old behavior of image fitting is removed, there is no need in Use Crop option at all. It is super easy to check crop values to be 0 and avoid any processing of the image in this case. > To initialize scale to fit, when strips are added to timeline, I think we should flag strips with `SEQ_FLAG_SCALE_NOT_CALCULATED` and calculate scale on demand. This is an implementation detail. It also seems too fragile, because the resolution in the file can change outside of Blender. The system could be smart enough to always do proper fitting. The scale for fitting is purely runtime, it is not a state, and is not something users are interacting with. > Over all, I would ask, how this feature should be different to existing transform effect strip? Transofmration is rather very fundamental operation during video editing. Having it as an effect seems weird. While it might work the same, requirement to always add effect every time one need to translate seems an overkill. > However, it is measurably slower, than current implementation The new code is non-vectorized single-threaded implementation. This is not how one does real-time video processing tool in 2020.

Francesco Siddi commented

2020-07-27 15:55:32 +02:00

Hi, do you have a screen-capture/screenshot of the patch in action?

Richard Antalik commented

2020-07-27 17:11:21 +02:00

In #78988#985973, @fsiddi wrote:
Hi, do you have a screen-capture/screenshot of the patch in action?

2020-07-27 16-45-00.mkv

It is quick proof of concept, values work exactly as in transform strip (that's why I was asking questions above) with exception, that it works on files bigger than project resolution.

Also I have one more detail I want to point out - in mockup, crop values are described as relative to transformed image. I can imagine situations where you want crop applied pre-transform, as well as post. Later, you can do easily with "dummy" strip, former you can't do easily. Not sure what would be absolutely optimal solution if you wanted both or which situation is more used. I would guess pre-transform one is better option.

In #78988#985645, @Sergey wrote:
Please stop exposing settings to user. This is not a solution to anything.

I don't want to. Anchor point is in mockup picture and interpolation is pickable in transform strip.

The new code is non-vectorized single-threaded implementation. This is not how one does real-time video processing tool in 2020.

Sure I was using blocks I had available. My question was perhaps a bit stupid, but what I was trying to ask is how hard we should try to optimize performance to absolute maximum. I guess answer would be "depends".

> In #78988#985973, @fsiddi wrote: > Hi, do you have a screen-capture/screenshot of the patch in action? [2020-07-27 16-45-00.mkv](https://archive.blender.org/developer/F8723289/2020-07-27_16-45-00.mkv) It is quick proof of concept, values work exactly as in transform strip (that's why I was asking questions above) with exception, that it works on files bigger than project resolution. Also I have one more detail I want to point out - in mockup, crop values are described as relative to transformed image. I can imagine situations where you want crop applied pre-transform, as well as post. Later, you can do easily with "dummy" strip, former you can't do easily. Not sure what would be absolutely optimal solution if you wanted both or which situation is more used. I would guess pre-transform one is better option. > In #78988#985645, @Sergey wrote: > Please stop exposing settings to user. This is not a solution to anything. I don't want to. Anchor point is in mockup picture and interpolation is pickable in transform strip. > The new code is non-vectorized single-threaded implementation. This is not how one does real-time video processing tool in 2020. Sure I was using blocks I had available. My question was perhaps a bit stupid, but what I was trying to ask is how hard we should try to optimize performance to absolute maximum. I guess answer would be "depends".

Sergey Sharybin commented

2020-07-27 17:43:53 +02:00

Also I have one more detail I want to point out - in mockup, crop values are described as relative to transformed image.

Not sure what exactly that means. In the patch crop sliders seems to not do anything.

Sure I was using blocks I had available. My question was perhaps a bit stupid, but what I was trying to ask is how hard we should try to optimize performance to absolute maximum.

There is IMB_processor_apply_threaded and IMB_processor_apply_threaded.

From technical side, performance is achieved by both technology (threading, vectorization) and by minimizing amount of work to be done. There is #78992 a parent task for performance.

It also feels wrong to talk about patch which does not implement this design.
Do we all agree on the design level?

> Also I have one more detail I want to point out - in mockup, crop values are described as relative to transformed image. Not sure what exactly that means. In the patch crop sliders seems to not do anything. > Sure I was using blocks I had available. My question was perhaps a bit stupid, but what I was trying to ask is how hard we should try to optimize performance to absolute maximum. There is `IMB_processor_apply_threaded` and `IMB_processor_apply_threaded`. From technical side, performance is achieved by both technology (threading, vectorization) and by minimizing amount of work to be done. There is #78992 a parent task for performance. --- It also feels wrong to talk about patch which does not implement this design. Do we all agree on the design level?

Richard Antalik commented

2020-07-27 19:00:13 +02:00

In #78988#986056, @Sergey wrote:

Also I have one more detail I want to point out - in mockup, crop values are described as relative to transformed image.

Not sure what exactly that means. In the patch crop sliders seems to not do anything.

I meant this mockup:

It also feels wrong to talk about patch which does not implement this design.
Do we all agree on the design level?

Yes overall I agree, I just wanted to clarify some details

> In #78988#986056, @Sergey wrote: >> Also I have one more detail I want to point out - in mockup, crop values are described as relative to transformed image. > > Not sure what exactly that means. In the patch crop sliders seems to not do anything. I meant this mockup: ![vse_improvements_02.jpg](https://archive.blender.org/developer/F8699080/vse_improvements_02.jpg) > It also feels wrong to talk about patch which does not implement this design. > Do we all agree on the design level? Yes overall I agree, I just wanted to clarify some details

tintwotin commented

2020-07-27 23:37:53 +02:00

Added subscriber: @tintwotin

tintwotin commented

2020-07-27 23:37:53 +02:00

In the mock-up, shouldn't Scale be in percentage?

Maybe these values should also be exposed as tools in the Toolbar like in the Transform Tools add-on(https://github.com/doakey3/VSE_Transform_Tools):
{F8723706, size = full}

Exposed anchor point in an enum and a drawn cross in the Transform Tools add-on:
{F8724574, size = full}

In the mock-up, shouldn't Scale be in percentage? Maybe these values should also be exposed as tools in the Toolbar like in the Transform Tools add-on(https://github.com/doakey3/VSE_Transform_Tools): {[F8723706](https://archive.blender.org/developer/F8723706/Transform_toolbar.gif), size = full} Exposed anchor point in an enum and a drawn cross in the Transform Tools add-on: {[F8724574](https://archive.blender.org/developer/F8724574/Anchor.gif), size = full}

Sergey Sharybin commented

2020-07-28 09:16:36 +02:00

Also I have one more detail I want to point out - in mockup, crop values are described as relative to transformed image.

Not sure what exactly that means. In the patch crop sliders seems to not do anything.

I meant this mockup

Ah, if I understand it correct (and @fsiddi will correct me if i'm wrong), those are the pixel values after the image has been auto-scaled to fit. As in, the crop is measured in the final frame pixels at 100% resolution.

It is also not unreasonable to expect that user-defined scale and rotation is applied after crop: otherwise one would need to constantly re-adjust crop after scaling the image.

In the mock-up, shouldn't Scale be in percentage?

Percentage is same as 0 to 1 factor. To me it seems that exposing it as a factor is more intuitive: scale 1.5 means the image is 1.5 times bigger. Scale of 150% is well, some people might have different idea of how that works (one might consider this additive).

>>> Also I have one more detail I want to point out - in mockup, crop values are described as relative to transformed image. >> Not sure what exactly that means. In the patch crop sliders seems to not do anything. > I meant this mockup Ah, if I understand it correct (and @fsiddi will correct me if i'm wrong), those are the pixel values after the image has been auto-scaled to fit. As in, the crop is measured in the final frame pixels at 100% resolution. It is also not unreasonable to expect that user-defined scale and rotation is applied after crop: otherwise one would need to constantly re-adjust crop after scaling the image. > In the mock-up, shouldn't Scale be in percentage? Percentage is same as 0 to 1 factor. To me it seems that exposing it as a factor is more intuitive: scale 1.5 means the image is 1.5 times bigger. Scale of 150% is well, some people might have different idea of how that works (one might consider this additive).

Francesco Siddi commented

2020-08-10 16:37:56 +02:00

In #78988#986382, @Sergey wrote:

Also I have one more detail I want to point out - in mockup, crop values are described as relative to transformed image.

Not sure what exactly that means. In the patch crop sliders seems to not do anything.

I meant this mockup

Ah, if I understand it correct (and @fsiddi will correct me if i'm wrong), those are the pixel values after the image has been auto-scaled to fit. As in, the crop is measured in the final frame pixels at 100% resolution.

Yes, those are the pixel values after the image has been auto-scaled to fit.

Regarding @tintwotin previous post: having visual transforms is desirable, but let's cover the basics first.

> In #78988#986382, @Sergey wrote: >>>> Also I have one more detail I want to point out - in mockup, crop values are described as relative to transformed image. >>> Not sure what exactly that means. In the patch crop sliders seems to not do anything. >> I meant this mockup > > Ah, if I understand it correct (and @fsiddi will correct me if i'm wrong), those are the pixel values after the image has been auto-scaled to fit. As in, the crop is measured in the final frame pixels at 100% resolution. > Yes, those are the pixel values after the image has been auto-scaled to fit. Regarding @tintwotin previous post: having visual transforms is desirable, but let's cover the basics first.

Andy Cuccaro commented

2020-08-13 16:27:52 +02:00

Added subscriber: @AndyCuccaro

Nils-Erik commented

2020-08-19 02:59:47 +02:00

Added subscriber: @clayhands

blender-admin commented

2020-11-02 22:15:52 +01:00

This issue was referenced by e1665c3d31

This issue was referenced by e1665c3d31906aed195bfdc10a4dd7582f31b1f5

blender-admin commented

2020-12-16 20:34:26 +01:00

This issue was referenced by 5713626422

This issue was referenced by 571362642201a743168cdf4c827a59c09c40414b

blender-admin commented

2021-01-10 16:37:18 +01:00

This issue was referenced by 9441413cb2

This issue was referenced by 9441413cb242f549ba1f86621ff72e90418313df

Richard Antalik commented

2021-04-01 11:59:36 +02:00

Marking as resolved, all relevant commits are attached here.

Richard Antalik commented

2021-04-01 11:59:56 +02:00

Changed status from 'Confirmed' to: 'Resolved'

Richard Antalik closed this issue

2021-04-01 11:59:56 +02:00

Richard Antalik self-assigned this 2021-04-01 11:59:56 +02:00

Konstantin Dmitriev (Morevna Project) commented

2023-01-20 11:11:12 +01:00

Added subscriber: @zelgadis

Konstantin Dmitriev (Morevna Project) commented

2023-01-20 11:11:12 +01:00

In #78988#985645, @Sergey wrote:
It also seems too fragile, because the resolution in the file can change outside of Blender. The system could be smart enough to always do proper fitting.

@Sergey "the resolution in the file can change outside of Blender" - this is exactly our case.
Unfortunately, since Blender 2.90 the situation when resolution changes outside of Blender is not handled properly.
This breaks workflow of our studio, I wrote in detail here - https://developer.blender.org/T78986#1477138
Any help is appreciated.

> In #78988#985645, @Sergey wrote: > It also seems too fragile, because the resolution in the file can change outside of Blender. The system could be smart enough to always do proper fitting. @Sergey "the resolution in the file can change outside of Blender" - this is exactly our case. Unfortunately, since Blender 2.90 the situation when resolution changes outside of Blender is not handled properly. This breaks workflow of our studio, I wrote in detail here - https://developer.blender.org/T78986#1477138 Any help is appreciated.

Richard Antalik referenced this issue

2023-12-11 20:13:34 +01:00

VSE 2.0: The Big Picture #78986

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

VSE 2.0: Media transform redesign #78988