Motion tracking marker's search area is not expanding with tracker, requiring constant manual scaling up. #107487

Open
opened 2023-04-30 23:32:01 +02:00 by michael campbell · 13 comments

System Information
Operating system: Windows-10-10.0.19044-SP0 64 Bits
Graphics card: NVIDIA GeForce GTX 1070/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 526.98

Blender Version
Broken: version: 3.5.1 Release Candidate, branch: blender-v3.5-release, commit date: 2023-04-21 11:31, hash: 93dac7da2c6a
Worked: Unsure

Short description of error
motion tracking marker's search area is shrinking during tracking, requiring constant pause and manual scale back up.

Exact steps for others to reproduce the error

  • Open blender, go to motion tracking workspace
  • load attached clip
    • check why first one loads in upside down too if you can
  • set global motion model to loc rot scale
  • add some trackers with the detect features operator
  • make sure search is enabled in the clip display settings (can't paste images here for some reason)
  • set tracking speed to realtime so you can see it happening.
  • track forward

The search area shrinks until it's touching the pattern area, resulting in less accuracy and requirement for manual re-scaling. This is exceptionally problematic with many markers, particularly as there's no way to re-scale the search area of multiple markers simultaneously, and it means you always have to track in realtime to ensure the problem is recognised.

The first thought might be this is expected because it's supposed to scale in this mode, but this seems unrelated to the expected scaling, because it happens when patterns are increasing in size, so if anything it should be expanding. Also if that were the case, then it would retain it's relative scale to the shrinking pattern area. I wonder if in the code perhaps the search area has accidentally been set to contract when the pattern areas grows and vice versa.

**System Information** Operating system: Windows-10-10.0.19044-SP0 64 Bits Graphics card: NVIDIA GeForce GTX 1070/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 526.98 **Blender Version** Broken: version: 3.5.1 Release Candidate, branch: blender-v3.5-release, commit date: 2023-04-21 11:31, hash: `93dac7da2c6a` Worked: Unsure **Short description of error** motion tracking marker's search area is shrinking during tracking, requiring constant pause and manual scale back up. **Exact steps for others to reproduce the error** - Open blender, go to motion tracking workspace - load attached clip - check why first one loads in upside down too if you can - set global motion model to loc rot scale - add some trackers with the detect features operator - make sure search is enabled in the clip display settings (can't paste images here for some reason) - set tracking speed to realtime so you can see it happening. - track forward The search area shrinks until it's touching the pattern area, resulting in less accuracy and requirement for manual re-scaling. This is exceptionally problematic with many markers, particularly as there's no way to re-scale the search area of multiple markers simultaneously, and it means you always have to track in realtime to ensure the problem is recognised. The first thought might be this is expected because it's supposed to scale in this mode, but this seems unrelated to the expected scaling, because it happens when patterns are increasing in size, so if anything it should be expanding. Also if that were the case, then it would retain it's relative scale to the shrinking pattern area. I wonder if in the code perhaps the search area has accidentally been set to contract when the pattern areas grows and vice versa.
michael campbell added the
Priority
Normal
Type
Report
Status
Needs Triage
labels 2023-04-30 23:32:02 +02:00
Member

Hi @3di , I could not reproduce "search area shrinks", but when using global motion model to loc rot scale, the patter would grow as the viewpoint moves, and when some patterns grows to fill the search area, it stops tracking. Is that what you are experiencing? As in "search area stays the same size but pattern size grows"?

Hi @3di , I could not reproduce "search area shrinks", but when using `global motion model to loc rot scale`, the patter would grow as the viewpoint moves, and when some patterns grows to fill the search area, it stops tracking. Is that what you are experiencing? As in "search area stays the same size but pattern size grows"?
YimingWu added
Status
Needs Information from User
Interest
Motion Tracking
and removed
Status
Needs Triage
labels 2023-05-01 07:07:02 +02:00
Member

The video up side down issue is probably blender didn't handle metatag rotation correctly. ffprob -i on that file reported:

  Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 1920x1080, 15603 kb/s, 29.98 fps, 29.97 tbr, 600 tbn, 1200 tbc (default)
    Metadata:
      rotate          : 180
      creation_time   : 2023-04-24T18:38:11.000000Z
      handler_name    : Core Media Video
      vendor_id       : [0][0][0][0]
      encoder         : H.264
    Side data:
      displaymatrix: rotation of -180.00 degrees

This rotation issue could be a similar mechanism of #93588.

The video up side down issue is probably blender didn't handle metatag rotation correctly. `ffprob -i` on that file reported: ``` Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 1920x1080, 15603 kb/s, 29.98 fps, 29.97 tbr, 600 tbn, 1200 tbc (default) Metadata: rotate : 180 creation_time : 2023-04-24T18:38:11.000000Z handler_name : Core Media Video vendor_id : [0][0][0][0] encoder : H.264 Side data: displaymatrix: rotation of -180.00 degrees ``` This rotation issue could be a similar mechanism of #93588.
YimingWu added the
Module
VFX & Video
Interest
Video Sequencer
labels 2023-05-01 07:09:48 +02:00

Hi @3di , I could not reproduce "search area shrinks", but when using global motion model to loc rot scale, the patter would grow as the viewpoint moves, and when some patterns grows to fill the search area, it stops tracking. Is that what you are experiencing? As in "search area stays the same size but pattern size grows"?

ah ok, this could very well be the problem actually yes. So really it needs the search area to grow/shrink in unison with the pattern area, so it maintains it's relative size. Otherwise you have to manually re-scale hundreds of markers.

> Hi @3di , I could not reproduce "search area shrinks", but when using `global motion model to loc rot scale`, the patter would grow as the viewpoint moves, and when some patterns grows to fill the search area, it stops tracking. Is that what you are experiencing? As in "search area stays the same size but pattern size grows"? ah ok, this could very well be the problem actually yes. So really it needs the search area to grow/shrink in unison with the pattern area, so it maintains it's relative size. Otherwise you have to manually re-scale hundreds of markers.
Member

@3di In that case I think it's a reasonable issue. I'll confirm it and see if video guys have any input on this.

@3di In that case I think it's a reasonable issue. I'll confirm it and see if video guys have any input on this.
YimingWu added
Status
Confirmed
and removed
Status
Needs Information from User
labels 2023-05-01 09:37:11 +02:00
YimingWu changed title from motion tracking marker's search area is shrinking during tracking, requiring constant pause and manual scale back up. to Motion tracking marker's search area is not expanding with tracker, requiring constant manual scaling up. 2023-05-01 09:37:48 +02:00

This is indeed annoying, but the exact strategy of when and how to scale is not really clear. The search area can not be scaled proportionally to the pattern area: doing so will quickly make the search area unreasonably huge.

A quick quality life improvement could be to ensure some amount of padding between the pattern and search area after tracking step.

@sebastian_k Does this sound reasonable to you?

In a more ideal world the search would need to be scaled according to a detected motion, but this is more in the realms of working on an automated tracker than a bug fix.

This is indeed annoying, but the exact strategy of when and how to scale is not really clear. The search area can not be scaled proportionally to the pattern area: doing so will quickly make the search area unreasonably huge. A quick quality life improvement could be to ensure some amount of padding between the pattern and search area after tracking step. @sebastian_k Does this sound reasonable to you? In a more ideal world the search would need to be scaled according to a detected motion, but this is more in the realms of working on an automated tracker than a bug fix.

@Sergey I would actually love this! Especially if the padding is maintained when the pattern shrinks again (or grows). And also, when using perspective or affine the pattern can become extremely long and stretched (often it still tracks fine though). In that case the search should not grow unreasonably large. Maybe some kind of padding average can be used or so?

@Sergey I would actually love this! Especially if the padding is maintained when the pattern shrinks again (or grows). And also, when using perspective or affine the pattern can become extremely long and stretched (often it still tracks fine though). In that case the search should not grow unreasonably large. Maybe some kind of padding average can be used or so?
Blender Bot added
Status
Archived
and removed
Status
Confirmed
labels 2023-05-01 12:52:11 +02:00

Oops, I’m sorry, didn’t mean to close this.

Oops, I’m sorry, didn’t mean to close this.
Blender Bot added
Status
Needs Triage
and removed
Status
Archived
labels 2023-05-01 12:53:13 +02:00
Pratik Borhade added
Status
Confirmed
and removed
Status
Needs Triage
labels 2023-05-01 12:54:20 +02:00

Especially if the padding is maintained when the pattern shrinks again

Think this is doable. Kind of to maintain previously keyframed search size.

When using perspective or affine the pattern can become extremely long and stretched

Yeah,, this is a concern. Not sure what you mean by padding average though.

A proper solution I think is to constraint the pattern area from becoming collapsing to singularity.

> Especially if the padding is maintained when the pattern shrinks again Think this is doable. Kind of to maintain previously keyframed search size. > When using perspective or affine the pattern can become extremely long and stretched Yeah,, this is a concern. Not sure what you mean by padding average though. A proper solution I think is to constraint the pattern area from becoming collapsing to singularity.
Sergey Sharybin added
Type
Bug
and removed
Type
Report
labels 2023-05-01 17:23:39 +02:00
Sergey Sharybin self-assigned this 2023-05-01 17:23:43 +02:00

Maybe something like this:

for marker in current_frame.markers:

    max_search = bpy.context.scene.max_search 
    min_search = bpy.context.scene.min_search
    original_pattern_bounds = marker.original_size.bounding_box
    current_pattern_bounds = marker.current_size.bounding_box
    original_search_size = marker.original_search_size
    search_ratio_x = original_search_size[0] / original_pattern_bounds[0]
    search_ration_y = original_search_size[1] / original_pattern_bounds[1]

    current_search_size = marker.current_search.size

    new_search_size_x = current_pattern_bounds[0] * search_ratio_x
    new_search_size_y = current_pattern_bounds [1] * search_ration_y

    if ((new_search_size_x,new_search_size_y) < (max_search[0], max_search[1]) and (new_search_size_x,new_search_size_y) > (min_search[0], min_search[1])):
        marker.current_search_size = (new_search_size_x,new_search_size_y)
    else:
        #pattern is either too large or too small.  if it's too small then there are too few pixels for accuracy, if it's too large then there are more pixels than necessary
        
        #set pattern size to original size at the same point.  so if it's to small, this'll expand the pattern area to sufficient pixels to recognise a new pattern, or if too big, this'll shrink back down to sufficient pixels to recognise a new pattern in the same spot.

        marker.pattern_size = marker.original_size
        marker.search_size = marker.original_search_size

        #update pattern
        marker.update_pattern() #if the new pattern isn't very trackable, then nothing lost, because it would have failed with the old pattern anyway at this point.

Then it'll keep going while it can without any user intervention.

What would also be nice is as soon as one becomes disabled during tracking, then automatically create a new one at the most contrasty point of the previous frame, then track just the new marker from the previous frame to the current, before continuing to track all markers again.

Maybe something like this: ```Py for marker in current_frame.markers: max_search = bpy.context.scene.max_search min_search = bpy.context.scene.min_search original_pattern_bounds = marker.original_size.bounding_box current_pattern_bounds = marker.current_size.bounding_box original_search_size = marker.original_search_size search_ratio_x = original_search_size[0] / original_pattern_bounds[0] search_ration_y = original_search_size[1] / original_pattern_bounds[1] current_search_size = marker.current_search.size new_search_size_x = current_pattern_bounds[0] * search_ratio_x new_search_size_y = current_pattern_bounds [1] * search_ration_y if ((new_search_size_x,new_search_size_y) < (max_search[0], max_search[1]) and (new_search_size_x,new_search_size_y) > (min_search[0], min_search[1])): marker.current_search_size = (new_search_size_x,new_search_size_y) else: #pattern is either too large or too small. if it's too small then there are too few pixels for accuracy, if it's too large then there are more pixels than necessary #set pattern size to original size at the same point. so if it's to small, this'll expand the pattern area to sufficient pixels to recognise a new pattern, or if too big, this'll shrink back down to sufficient pixels to recognise a new pattern in the same spot. marker.pattern_size = marker.original_size marker.search_size = marker.original_search_size #update pattern marker.update_pattern() #if the new pattern isn't very trackable, then nothing lost, because it would have failed with the old pattern anyway at this point. ``` Then it'll keep going while it can without any user intervention. What would also be nice is as soon as one becomes disabled during tracking, then automatically create a new one at the most contrasty point of the previous frame, then track just the new marker from the previous frame to the current, before continuing to track all markers again.

See my initial reply here. It is not ideal to scale search area proportionally to the pattern. The amount of movement of a feature is not linearly dependent on the size of the feature. Overestimating the size of the search area at a best will cause considerable performance drop, and in average it will degrade quality of track. That is why I was talking about padding and not scaling.

See my initial reply here. It is not ideal to scale search area proportionally to the pattern. The amount of movement of a feature is not linearly dependent on the size of the feature. Overestimating the size of the search area at a best will cause considerable performance drop, and in average it will degrade quality of track. That is why I was talking about padding and not scaling.

that's what the user definable max_search_size would be for. If it fails due to too much movement, then it would fail anyway. The above is if it's failing unnecessarily due to the search size being insufficient for the expanded pattern size rather than movement beyond the bounds of the search size.

that's what the user definable max_search_size would be for. If it fails due to too much movement, then it would fail anyway. The above is if it's failing unnecessarily due to the search size being insufficient for the expanded pattern size rather than movement beyond the bounds of the search size.

it basically keeps the search size relative to the pattern until unfeasable (either because the pattern and search size have grown too big for good performance, or if the pattern has shrunk to too few pixels for a recognisable pattern), then sets the marker pattern and search back to the original dimensions and registers a new pattern in the same location for the following frames, to ensure performance. It's doing what otherwise has to be done manually.

I guess it's similar to what happens with match set to previous frame, except it always ensures the pattern doesn't have too many or too few pixels. Always keeping the search area relative within the bounds of good performance.

it basically keeps the search size relative to the pattern until unfeasable (either because the pattern and search size have grown too big for good performance, or if the pattern has shrunk to too few pixels for a recognisable pattern), then sets the marker pattern and search back to the original dimensions and registers a new pattern in the same location for the following frames, to ensure performance. It's doing what otherwise has to be done manually. I guess it's similar to what happens with match set to previous frame, except it always ensures the pattern doesn't have too many or too few pixels. Always keeping the search area relative within the bounds of good performance.

imagine if you have 10 markers whose pattern has grown to the bounds of the search area. Currently you have to select all markers and scale them back down to register the new pattern, but the horrible bit is, you then have to re-expand the search area one marker at a time by dragging the little triangle. With a hundred markers, it's tear inducing.

You can't use the alt enter method in the search size textbox, because not all markers search areas have changed by the same amount.

imagine if you have 10 markers whose pattern has grown to the bounds of the search area. Currently you have to select all markers and scale them back down to register the new pattern, but the horrible bit is, you then have to re-expand the search area one marker at a time by dragging the little triangle. With a hundred markers, it's tear inducing. You can't use the alt enter method in the search size textbox, because not all markers search areas have changed by the same amount.
Sign in to join this conversation.
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset Browser
Interest
Asset Browser Project Overview
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
EEVEE & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Interest: X11
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
EEVEE & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No project
No Assignees
4 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender#107487
No description provided.