GHOST: Process full mouse move resolution in Win32 #125949

Open
YimingWu wants to merge 3 commits from ChengduLittleA/blender:ghost-win32-mousemove into main

When changing the target branch, be careful to rebase the branch in your fork to match. See documentation.
Member

In Win32, WM_MOUSEMOVE don't usually report all the mouse move events.
So when using a non-pen device to draw stuff, like grease pencil or texture paint, you will very easily see jagged lines.
This is just how WM_MOUSEMOVE event behaves. It doesn't give you more granulated events unlike on X11.

This patch attempts to get all the mouse events and send them in sequence so a smoother input can be achieved, and
any generic pointer device would be benefited.

Update:

There's likely a bug with GetMouseMovePointsEx in Win32 that prevents this from working properly. So probably can't implement this since if the event doesn't work you aren't even able to manipulate user interface.

Before After
image image
In Win32, `WM_MOUSEMOVE` don't usually report all the mouse move events. So when using a non-pen device to draw stuff, like grease pencil or texture paint, you will very easily see jagged lines. This is just how `WM_MOUSEMOVE` event behaves. It doesn't give you more granulated events unlike on X11. This patch attempts to get all the mouse events and send them in sequence so a smoother input can be achieved, and any generic pointer device would be benefited. Update: There's likely a bug with `GetMouseMovePointsEx` in Win32 that prevents this from working properly. So probably can't implement this since if the event doesn't work you aren't even able to manipulate user interface. | Before | After | |-----|-----| | ![image](/attachments/50ed9459-402c-4cc4-84b5-f1b64487fad6) | ![image](/attachments/23542435-2651-4af9-86fb-57b45e091877) |
YimingWu added the
Module
Core
label 2024-08-06 10:16:22 +02:00
YimingWu added 2 commits 2024-08-06 10:16:29 +02:00
YimingWu added 1 commit 2024-08-06 10:36:16 +02:00
YimingWu changed title from WIP: GHOST: Process full mouse move resolution in Win32 to GHOST: Process full mouse move resolution in Win32 2024-08-06 10:36:45 +02:00
YimingWu requested review from Harley Acheson 2024-08-06 10:37:02 +02:00
Member

I haven't actually tested this yet, or looked at it much, so the following comments could be all wrong and/or dumb...

I think you can pass your timestamp in the input MOUSEMOVEPOINT's time. That way if there are multiple positions with the same values it will use the timestamp to get the right one. "If your application supplies a time stamp, the GetMouseMovePointsEx function will use it to differentiate between two equal points that were recorded at different times."

The docs on this API function are a bit vague about issues with multiple monitors. It mentions that in the case of multiple monitors it can give bad data if you give it negative values or if returned coordinates are negative, which is says "can occur if multiple monitors are present". But then it goes on to give an example of how to correct for this yet it is only for the case of using GMMP_USE_HIGH_RESOLUTION_POINTS, not GMMP_USE_DISPLAY_POINTS. But the remarks don't explicitly say that you can get erroneous values in those situations only if using GMMP_USE_DISPLAY_POINTS. Nor does it say that you should use GMMP_USE_HIGH_RESOLUTION_POINTS for multiple monitor support.

Not trying to second-guess Microsoft, but their example code looks dodgy to me in the case of monitors that differ in DPI or user scale.

Their example is here: https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-getmousemovepointsex

The above confusion about multiple monitors led me to this document about how Paint.net removed their use of GetMouseMovePointsEx specifically because it worked badly with multiple monitors and/or with differing DPI or scale.

https://blog.getpaint.net/2019/11/14/paint-net-4-2-6-alpha-build-7258/

So in total this makes me a bit nervous that MS added this function a long time ago (Windows 2000) and haven't update it or the docs to consider differing scale or dpi. Multiple monitors were a thing back then but not with different scale or dpi.

I haven't actually tested this yet, or looked at it much, so the following comments could be all wrong and/or dumb... I _think_ you can pass your timestamp in the input MOUSEMOVEPOINT's time. That way if there are multiple positions with the same values it will use the timestamp to get the right one. "If your application supplies a time stamp, the GetMouseMovePointsEx function will use it to differentiate between two equal points that were recorded at different times." The docs on this API function are a bit vague about issues with multiple monitors. It mentions that in the case of multiple monitors it can give bad data if you give it negative values or if returned coordinates are negative, which is says "can occur if multiple monitors are present". But then it goes on to give an example of how to correct for this yet it is only for the case of using GMMP_USE_HIGH_RESOLUTION_POINTS, not GMMP_USE_DISPLAY_POINTS. But the remarks don't explicitly say that you can get erroneous values in those situations only if using GMMP_USE_DISPLAY_POINTS. Nor does it say that you should use GMMP_USE_HIGH_RESOLUTION_POINTS for multiple monitor support. Not trying to second-guess Microsoft, but their example code looks dodgy to me in the case of monitors that differ in DPI or user scale. Their example is here: https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-getmousemovepointsex The above confusion about multiple monitors led me to this document about how Paint.net removed their use of GetMouseMovePointsEx specifically because it worked badly with multiple monitors and/or with differing DPI or scale. https://blog.getpaint.net/2019/11/14/paint-net-4-2-6-alpha-build-7258/ So in total this makes me a bit nervous that MS added this function a long time ago (Windows 2000) and haven't update it or the docs to consider differing scale or dpi. Multiple monitors were a thing back then but not with different scale or dpi.
Author
Member

@Harley I actually have tested this with my own painting program with dual screen setup, if it's using GMMP_USE_DISPLAY_POINTS it will be correct in this logic regardless of screen configurations (and in their conversion code example they also just pass that along). I haven't tried to use it with DPI scale, I will be ensuring that works (or if it doesn't I'll try find a way to do it lol)

Also I think Blender always use full screen resolution (not scaled by windows), so DPI scale shouldn't really be a problem...

@Harley I actually have tested this with my own painting program with dual screen setup, if it's using `GMMP_USE_DISPLAY_POINTS` it will be correct in this logic regardless of screen configurations (and in their conversion code example they also just pass that along). I haven't tried to use it with DPI scale, I will be ensuring that works (or if it doesn't I'll try find a way to do it lol) Also I think Blender always use full screen resolution (not scaled by windows), so DPI scale shouldn't really be a problem...
Author
Member

Hi @Harley ! Actually it's not really the problem of DPI scale, but rather the placement of the second screen. (I'm on win11 atm)

I can have events working as long as screens are configured in layout 1 regardless of DPI scale (small screen is main screen). if in layout 2 then blender would only register the events below half of the screen (below origin of the main screen).

aaaand GMMP_USE_HIGH_RESOLUTION_POINTS seems to be broken as well, So this does seem to be a windows bug unfortunately :(

Hi @Harley ! Actually it's not really the problem of DPI scale, but rather the placement of the second screen. (I'm on win11 atm) I can have events working as long as screens are configured in layout 1 regardless of DPI scale (small screen is main screen). if in layout 2 then blender would only register the events below half of the screen (below origin of the main screen). aaaand `GMMP_USE_HIGH_RESOLUTION_POINTS` seems to be broken as well, So this does seem to be a windows bug unfortunately :(
Author
Member

Maybe one way to get pas this is to check whether we have more than 1 monitors plugged in (and also detect every time that configuration changes), sounds less optimal but would generally lead to a nicer mouse experience overall under windows. e.g. I typically use mouse to sculpt

Maybe one way to get pas this is to check whether we have more than 1 monitors plugged in (and also detect every time that configuration changes), sounds less optimal but would generally lead to a nicer mouse experience overall under windows. e.g. I typically use mouse to sculpt
Member

if available maybe ping @PrototypeNM1 he likely walked this minefield before and may have insights.

if available maybe ping @PrototypeNM1 he likely walked this minefield before and may have insights.
Member

Is it just that it can screw up in some circumstances for a window on a particular monitor if that monitor is set to a high dpi or scale? If that is the case we could just test for that and not run your code in that case. We are informed as our windows are moved between monitors (so we can adjust for the changes in the sizes of the title bar, etc).

Or can it also screw up on a regular monitor if there is a high-dpi one between it and the origin? That would pretty hard to deal with.

Or is it something more specific, like a high-dpi monitor that has parts below the origin point? We could test for that.

Is it just that it can screw up in some circumstances for a window on a particular monitor if that monitor is set to a high dpi or scale? If that is the case we could just test for that and not run your code in that case. We are informed as our windows are moved between monitors (so we can adjust for the changes in the sizes of the title bar, etc). Or can it also screw up on a regular monitor if there is a high-dpi one between it and the origin? That would pretty hard to deal with. Or is it something more specific, like a high-dpi monitor that has parts below the origin point? We could test for that.
Member

if available maybe ping @PrototypeNM1 he likely walked this minefield before and may have insights.

Yes, I have been having lots of Déjà vu about this particular API call. I think we've tried this before.

> if available maybe ping @PrototypeNM1 he likely walked this minefield before and may have insights. Yes, I have been having lots of Déjà vu about this particular API call. I think we've tried this before.
Author
Member

According to my tests, this always works:

  • have primary screen on left and top most so no coordinates would ever go below 0,0 for the origin of the entire frame buffer
  • different DPI configuration in this setup doesn't seem to matter, mouse events always worked.

This won't work:

  • any of the portion of the screen has exceeded the top/left border of the primary screen
    • in which case when your mouse move to those positions the coordinates can be negative related to the screen, however that function only seem to accept positive points (?)
    • still seems to work fine in all positions that's to the right and below the origin point, including on main monitor and extended monitor regardless of DPI setting.

I think if I print and look at the coordinates I could find out how it works (that is, if calls like "client to screen" always worked)...

According to my tests, this always works: - have primary screen on left and top most so no coordinates would ever go below 0,0 for the origin of the entire frame buffer - different DPI configuration in this setup doesn't seem to matter, mouse events always worked. This won't work: - any of the portion of the screen has exceeded the top/left border of the primary screen - in which case when your mouse move to those positions the coordinates can be negative related to the screen, however that function only seem to accept positive points (?) - still seems to work fine in all positions that's to the right and below the origin point, including on main monitor and extended monitor regardless of DPI setting. I think if I print and look at the coordinates I could find out how it works (that is, if calls like "client to screen" always worked)...
Member

Oh, so if that thing can't handle negative values then we could probably just skip your code if the window rect (relative to the combined desktop) contains negative values?

Oh, so if that thing can't handle negative values then we could probably just skip your code if the window rect (relative to the combined desktop) contains negative values?
Author
Member

I'm not sure what the exact value is. I have to check tomorrow. The problems is that none of the coordinates are supposed to be negative in the first place, the framebuffer origin should be as low as you go so I'm not entirely sure about the mechanism at work here.

I'm not sure what the exact value is. I have to check tomorrow. The problems is that none of the coordinates are supposed to be negative in the first place, the framebuffer origin should be as low as you go so I'm not entirely sure about the mechanism at work here.

GetMouseMovePointsEx returns corrupt data for quadrants 2 through 4 of the screen when the input is in absolute coordinates. This most commonly occurs with tablets, remote desktop, and Win32 calls to set the mouse position. This is likely due to Q2-4 having a leading 1 when reporting position, which combines with the aforementioned "negative" values issue due to bit casting in Windows mouse handling code (outside of our control).

I believe I had previously chased down that we can't tell if the source of mouse movement is absolute or relative without using the RawInput API, which itself would provide values that are not usable with GetMouseMovePointsEx due to not factoring in mouse acceleration. To the best of my knowledge, the unfortunate situation on Windows is that there is no way to handle all mouse input without this breakage. Also unfortunate is that we have learned this lesson several times, last time I put a warning against using GetMouseMovePointsEx but that seems to have been removed at some point so I anticipate we'll continue relearning this lesson into the future. :(

`GetMouseMovePointsEx` returns corrupt data for quadrants 2 through 4 of the screen when the input is in absolute coordinates. This most commonly occurs with tablets, remote desktop, and Win32 calls to set the mouse position. This is likely due to Q2-4 having a leading 1 when reporting position, which combines with the aforementioned "negative" values issue due to bit casting in Windows mouse handling code (outside of our control). I believe I had previously chased down that we can't tell if the source of mouse movement is absolute or relative without using the RawInput API, which itself would provide values that are not usable with `GetMouseMovePointsEx` due to not factoring in mouse acceleration. To the best of my knowledge, the [unfortunate](https://blog.getpaint.net/2019/11/14/paint-net-4-2-6-alpha-build-7258/) situation on Windows is that there is no way to handle all mouse input without this breakage. Also unfortunate is that we have learned this lesson several times, last time I put a warning against using `GetMouseMovePointsEx` but that seems to have been removed at some point so I anticipate we'll continue relearning this lesson into the future. :(
Author
Member

thanks @PrototypeNM1 ! I wonder if reporting this to microsoft would do any good... I guess since pen input works, this could be like... not a high priority but it kinda sucks this way compared to X11 🤔

thanks @PrototypeNM1 ! I wonder if reporting this to microsoft would do any good... I guess since pen input works, this could be like... not a high priority but it kinda sucks this way compared to X11 🤔
Member

Is it an option to use RAW mouse pointer input? I know that means we don't get pointer acceleration, but perhaps we could do that ourselves? https://stackoverflow.com/questions/36862013/raw-input-and-cursor-acceleration

Is it an option to use RAW mouse pointer input? I know that means we don't get pointer acceleration, but perhaps we could do that ourselves? https://stackoverflow.com/questions/36862013/raw-input-and-cursor-acceleration
Author
Member

@Harley doesn't this mean that the user might experience different acceleration than what would be through the OS?

@Harley doesn't this mean that the user might experience different acceleration than what would be through the OS?
Member

Is it an option to use RAW mouse pointer input? I know that means we don't get pointer acceleration, but perhaps we could do that ourselves? https://stackoverflow.com/questions/36862013/raw-input-and-cursor-acceleration

Trying to match an undisclosed algorithm for mouse pointer ballistics pretty much guarantees we're gonna screw it up and not match the OS which will lead to "the mouse in blender in weird compared to other apps" tickets, which will likely be much more numerous than the number of the mouse resolution is low tickets we currently get.

Given there is no perfect solution, maybe like tablet input, we could offer a few backends and give the user the choice between low res with ballistics using the current method and a DirectInput or Raw HID input that'll have better resolution but have other drawbacks.

> Is it an option to use RAW mouse pointer input? I know that means we don't get pointer acceleration, but perhaps we could do that ourselves? https://stackoverflow.com/questions/36862013/raw-input-and-cursor-acceleration Trying to match an undisclosed algorithm for mouse pointer ballistics pretty much guarantees we're gonna screw it up and not match the OS which will lead to "the mouse in blender in weird compared to other apps" tickets, which will likely be much more numerous than the number of the mouse resolution is low tickets we currently get. Given there is no perfect solution, maybe like tablet input, we could offer a few backends and give the user the choice between low res with ballistics using the current method and a DirectInput or Raw HID input that'll have better resolution but have other drawbacks.

Is it an option to use RAW mouse pointer input? I know that means we don't get pointer acceleration, but perhaps we could do that ourselves? https://stackoverflow.com/questions/36862013/raw-input-and-cursor-acceleration

If we use raw mouse input then we'll also need to handle drawing a virtual cursor so that we don't have visual de-syncs from raw mouse and cursor having different accelerations. This is doable, but I'm not sure we want to do it given the ballistics inconsistency @LazyDodo mentioned.

It might be worth studying the dwExtraInfo field from MOUSEMOVEPOINT to determine if there's a pattern we can use to differentiate relative from absolute input, and disregard the latter if present. We would need to inspect that for mouse, tablet, SendInput, SetCursorPos, remote desktop. If there's a reliable identifier for absolute position input I believe we could use GetMouseMovePointsEx without issue. Note, afaik there is not public documentation (from Microsoft or elsewhere) on the dwExtraInfo field.

> Is it an option to use RAW mouse pointer input? I know that means we don't get pointer acceleration, but perhaps we could do that ourselves? https://stackoverflow.com/questions/36862013/raw-input-and-cursor-acceleration If we use raw mouse input then we'll also need to handle drawing a virtual cursor so that we don't have visual de-syncs from raw mouse and cursor having different accelerations. This is doable, but I'm not sure we want to do it given the ballistics inconsistency @LazyDodo mentioned. It might be worth studying the `dwExtraInfo` field from `MOUSEMOVEPOINT ` to determine if there's a pattern we can use to differentiate relative from absolute input, and disregard the latter if present. We would need to inspect that for mouse, tablet, `SendInput`, `SetCursorPos`, remote desktop. If there's a reliable identifier for absolute position input I believe we could use `GetMouseMovePointsEx` without issue. Note, afaik there is not public documentation (from Microsoft or elsewhere) on the `dwExtraInfo` field.
Member

If we use raw mouse input then we'll also need to handle drawing a virtual cursor so that we don't have visual de-syncs from raw mouse and cursor having different accelerations. This is doable, but I'm not sure we want to do it given the ballistics inconsistency @LazyDodo mentioned.

No, that sounds like the showstopper for that idea. Thanks!

> If we use raw mouse input then we'll also need to handle drawing a virtual cursor so that we don't have visual de-syncs from raw mouse and cursor having different accelerations. This is doable, but I'm not sure we want to do it given the ballistics inconsistency @LazyDodo mentioned. No, that sounds like the showstopper for that idea. Thanks!
Merge conflict checking is in progress. Try again in few moments.

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u ghost-win32-mousemove:ChengduLittleA-ghost-win32-mousemove
git checkout ChengduLittleA-ghost-win32-mousemove
Sign in to join this conversation.
No reviewers
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset System
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Code Documentation
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
FBX
Interest
Freestyle
Interest
Geometry Nodes
Interest
glTF
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Viewport & EEVEE
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Interest: X11
Legacy
Asset Browser Project
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Asset System
Module
Core
Module
Development Management
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Module
Viewport & EEVEE
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Severity
High
Severity
Low
Severity
Normal
Severity
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No project
No Assignees
4 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender#125949
No description provided.