Draft: Added getPixelUnderCursor for MacOS_X #111397

Caleb-Cleavinger · 2023-08-22T17:17:43+02:00

Caleb-Cleavinger commented

2023-08-22 17:17:43 +02:00

Added feature in issue #111303 for MacOS. Note I have not tested this b/c of a lack of access to an Apple device to test on.

Caleb-Cleavinger added 1 commit 2023-08-22 17:17:56 +02:00

buildbot/vexp-code-patch-coordinator Build done. Details

bd04376bac Added getPixelUnderCursor for MacOS_X

Caleb-Cleavinger referenced this pull request

2023-08-22 17:19:45 +02:00

UI: Linux & Mac Support for Eyedropper Outside Blender #111303

Harley Acheson reviewed 2023-08-22 19:57:12 +02:00

intern/ghost/intern/GHOST_SystemCocoa.hh Outdated

						
				@ -11,3 +11,3 @@

				#ifndef __APPLE__

				#  error Apple OSX only!

				  #  error Apple OSX only!

Harley Acheson commented

2023-08-22 19:57:12 +02:00

unintended formatting change

Harley Acheson commented

2023-08-22 19:59:24 +02:00

@blender-bot build macos

Harley Acheson commented

2023-08-22 20:04:28 +02:00

This looks like the right thing, keeping in mind that I don't build for Mac.

When we are getting colors from outside of the Blender window we probably need to keep a consistent colorspace, which for now is sRGB. That way once it comes in we know what it is and can transform it further if needed.

It is my (possibly wrong) understanding that the NSColor object can create a new color object in an different colorspace with usingColorSpace and sRGB as argument.

This looks like the right thing, keeping in mind that I don't build for Mac. When we are getting colors from outside of the Blender window we probably need to keep a consistent colorspace, which for now is sRGB. That way once it comes in we know what it is and can transform it further if needed. It is my (possibly wrong) understanding that the [NSColor](https://developer.apple.com/documentation/appkit/nscolor) object can create a new color object in an different colorspace with [usingColorSpace](https://developer.apple.com/documentation/appkit/nscolor/1527379-usingcolorspace) and [sRGB](https://developer.apple.com/documentation/appkit/nscolorspace/1412071-srgb) as argument.

Harley Acheson commented

2023-08-22 20:52:18 +02:00

This does compile without errors:

If you are enjoying this so far, why not try updating this PR without the format change and with the sRGB colorspace change, and then we can create an installable build for users to test?

This does compile without errors: * [darwin-x86_64](https://builder.blender.org/admin/#/builders/134/builds/2205) * [darwin-arm64](https://builder.blender.org/admin/#/builders/135/builds/2167) If you are enjoying this so far, why not try updating this PR without the format change and with the sRGB colorspace change, and then we can create an installable build for users to test?

Caleb-Cleavinger commented

2023-08-22 21:54:58 +02:00

NSColor is already sRGB I thought. I can fix the indentation mistake pretty quickly.

Harley Acheson commented

2023-08-22 21:57:00 +02:00

NSColor is already sRGB I thought. I can fix the indentation mistake pretty quickly.

Ah, I honestly didn't know. If there are any colorspace issues they will come up in user testing anyway.

> NSColor is already sRGB I thought. I can fix the indentation mistake pretty quickly. Ah, I honestly didn't know. If there are any colorspace issues they will come up in user testing anyway.

Caleb-Cleavinger added 1 commit 2023-08-22 22:47:54 +02:00

buildbot/vexp-code-patch-coordinator Build done. Details

e676fe3ae9 Fixed formatting back to original

Harley Acheson commented

2023-08-22 22:54:28 +02:00

@blender-bot package macos

Blender Bot commented

2023-08-22 22:54:32 +02:00

Package build started. Download here when ready.

Package build started. [Download here](https://builder.blender.org/download/patch/PR111397) when ready.

Harley Acheson commented

2023-08-23 00:14:08 +02:00

@blender-bot package macos

Blender Bot commented

2023-08-23 00:14:11 +02:00

Package build started. Download here when ready.

Package build started. [Download here](https://builder.blender.org/download/patch/PR111397) when ready.

Alaska commented

2023-08-23 02:30:10 +02:00

Testing on a M1 Pro mac on macOS 13.4.1, this doesn't work.
As in, the colour picker still works inside Blender, but not outside Blender. I have attached a video showing this.

Things I tested:

Use the colour picker to select a colour inside the current Blender window (Worked)
Use the colour picker to select a colour inside a child Blender window (Didn't work, also doesn't work in Main)
Use the colour picker to select a colour outside of Blender (Didn't work)
I didn't show it in the video, I tried selecting a colour outside of Blender, then bringing Blender into focus (by selecting the title bar), just to see if Blender wasn't accepting the chosen colour because it was out of focus (Didn't work)

If you would like me to run specific tests, even tests with a debugger and a active break point, just ask and I'll get back to you as soon as I can with the results from the test.

Testing on a M1 Pro mac on macOS 13.4.1, this doesn't work. As in, the colour picker still works inside Blender, but not outside Blender. I have attached a video showing this. Things I tested: 1. Use the colour picker to select a colour inside the current Blender window (Worked) 2. Use the colour picker to select a colour inside a child Blender window (Didn't work, also doesn't work in Main) 3. Use the colour picker to select a colour outside of Blender (Didn't work) 4. I didn't show it in the video, I tried selecting a colour outside of Blender, then bringing Blender into focus (by selecting the title bar), just to see if Blender wasn't accepting the chosen colour because it was out of focus (Didn't work) If you would like me to run specific tests, even tests with a debugger and a active break point, just ask and I'll get back to you as soon as I can with the results from the test.

macOS colour picker experiment 1.mp4

4.4 MiB

Harley Acheson commented

2023-08-23 02:49:38 +02:00

Testing on a M1 Pro mac on macOS 13.4.1, this doesn't work.

Thanks for testing this. And special thanks for the video.

With @Caleb-Cleavinger not having the hardware we'll probably have to wait until someone can get this in a debugger.

eyedropper_color_sample_fl() in eyedropper_color.cc should return black if WM_capabilities_flag() was not set correctly or if getPixelAtCursor returned false. The video shows it doing nothing, in that the color remains as it is.

> Testing on a M1 Pro mac on macOS 13.4.1, this doesn't work. Thanks for testing this. And special thanks for the video. With @Caleb-Cleavinger not having the hardware we'll probably have to wait until someone can get this in a debugger. eyedropper_color_sample_fl() in eyedropper_color.cc should return black if WM_capabilities_flag() was not set correctly or if getPixelAtCursor returned false. The video shows it doing nothing, in that the color remains as it is.

Alaska commented

2023-08-23 04:27:45 +02:00

Here's some extra information.

I built a debug version of this pull request and opened it with break points in getPixelAtCursor (GHOST_SystemCocoa.mm) and eyedropper_color_sample_fl (eyedropper_color.cc) just to see what was happening.

When using the colour picker inside the Blender window, only eyedropper_color_sample_fl is called (as expected?)
When clicking out side of Blender, neither are called (That's why it's not working).
When clicking on the edge of Blender in a certain way, both functions are called and macOS brings up a window basically saying "Do you want to allow Blender to record your screen?". If I ignore that message. Blender will sample the colour of my wallpaper under my cursor (not the colour of the pixel under my cursor)
If I then accept Blender having the ability to screen record, re-open Blender, tests 1 and 2 remain the same. When doing test 3 the colour picked is now white instead of my wallpaper.

I have attached a video demonstrating this. Where I do test 1, test 2, test 3, then accept the permissions and do test 1, 2, 3, again.

If I understand it right, it seems there's two issues.

Blender doesn't recognise a click outside of Blender on macOS.
When Blender does recognise a click outside of Blender in certain situations (E.G. grabbing the edge of Blender), the colour information it gets back is either the wallpaper or white when it should actually be the pixel under the cursor.

If any further testing or information is needed, just ask and I'll try and get you that information.

Here's some extra information. I built a debug version of this pull request and opened it with break points in `getPixelAtCursor` (GHOST_SystemCocoa.mm) and `eyedropper_color_sample_fl` (eyedropper_color.cc) just to see what was happening. 1. When using the colour picker inside the Blender window, only eyedropper_color_sample_fl is called (as expected?) 2. When clicking out side of Blender, neither are called (That's why it's not working). 3. When clicking on the edge of Blender in a certain way, both functions are called and macOS brings up a window basically saying "Do you want to allow Blender to record your screen?". If I ignore that message. Blender will sample the colour of my wallpaper under my cursor (not the colour of the pixel under my cursor) 4. If I then accept Blender having the ability to screen record, re-open Blender, tests 1 and 2 remain the same. When doing test 3 the colour picked is now white instead of my wallpaper. I have attached a video demonstrating this. Where I do test 1, test 2, test 3, then accept the permissions and do test 1, 2, 3, again. If I understand it right, it seems there's two issues. 1. Blender doesn't recognise a click outside of Blender on macOS. 2. When Blender does recognise a click outside of Blender in certain situations (E.G. grabbing the edge of Blender), the colour information it gets back is either the wallpaper or white when it should actually be the pixel under the cursor. --- If any further testing or information is needed, just ask and I'll try and get you that information.

macOS colour picker experiment 2.mp4

18 MiB

Harley Acheson commented

2023-08-23 18:26:11 +02:00

@Alaska

Thanks for all this!

When using the colour picker inside the Blender window, only eyedropper_color_sample_fl is called (as expected?)

Yes, while inside the blender window the sampling is handled by separate code so that we can deal specially for some areas of Blender like in Image editor or Clip Editor.

When clicking out side of Blender, neither are called (That's why it's not working).

This is the kicker. The seems to imply that our event system on Mac is not getting events for some or all mouse events when they are outside of our windows. eyedropper_modal is probably not being called for the click event. I'm assuming this isn't a platform limitation in that other programs, like Gimp, can sample outside of their windows? So for this to differ between platforms is probably a difference in mouse event handling in ghost, possibly in GHOST_SystemCocoa::handleMouseEvent.

If any further testing or information is needed, just ask and I'll try and get you that information.

This is getting pretty weedy for me. I have (rough) plans to get a portable Mac for developing early-ish next year, so I could work while away. So I might be able to look at this then.

@Alaska Thanks for all this! > 1. When using the colour picker inside the Blender window, only eyedropper_color_sample_fl is called (as expected?) Yes, while inside the blender window the sampling is handled by separate code so that we can deal specially for some areas of Blender like in Image editor or Clip Editor. > 2. When clicking out side of Blender, neither are called (That's why it's not working). This is the kicker. The _seems_ to imply that our event system on Mac is not getting events for some or all mouse events when they are outside of our windows. `eyedropper_modal` is probably not being called for the click event. I'm assuming this isn't a platform limitation in that other programs, like Gimp, can sample outside of their windows? So for this to differ between platforms is probably a difference in mouse event handling in ghost, possibly in GHOST_SystemCocoa::handleMouseEvent. > If any further testing or information is needed, just ask and I'll try and get you that information. This is getting pretty weedy for me. I have (rough) plans to get a portable Mac for developing early-ish next year, so I could work while away. So I might be able to look at this then.

Caleb-Cleavinger added 1 commit 2023-08-23 19:19:42 +02:00

d6e88ebd47 Changed method to allow multi monitor screenshot

Caleb-Cleavinger commented

2023-08-23 19:31:56 +02:00

I changed the method to work with multiple monitors (if it actually gets called on those monitors) this code should work when it gets called. After we test the new multimonitor code I think we should push this PR and add a new issue to fix mouse polling on external windows/programs on Mac to hopefully get people who have actual experience with Apple's API and can better debug/test the mouse polling.

Alaska commented

2023-08-24 06:04:55 +02:00

I'm assuming this isn't a platform limitation in that other programs, like Gimp, can sample outside of their windows?

Just wanted to comment on that. Yes, GIMP can sample outside the GIMP window on macOS, so it's not a platform limitation. However it behaves differently from Windows and I have a suspicion on why (I'll explain it at the bottom).

Just for reference, when you don't grant GIMP permission to record the screen, then colours from the GIMP colour picker outside the window are just the wallpaper at the current pixel. Which explains why the wallpaper colour was being selected in some of my tests while I hadn't accepted the permission.

This is the kicker. The seems to imply that our event system on Mac is not getting events for some or all mouse events when they are outside of our windows. eyedropper_modal is probably not being called for the click event.

Quick testing with a debugger suggests it is getting some events (mouse movement?, probably others) outside of the main window, it's just that the code seems to break before doing anything with events from outside the main window. Here's an explanation of what I mean by that:

GHOST_SystemCocoa::processEvents (GHOST_SystemCocoa.mm) is called from somewhere fairly frequently.
In GHOST_SystemCocoa::processEvents on line 1002 there's event = [NSApp nextEventMatchi...];. If Blender is not the active app, event = null, line 1006 breaks when event = null meaning the event doesn't get processed.
If Blender is the active app, then event=something, even when the mouse is outside Blender, and the mouse event gets processed in GHOST_SystemCocoa::handleMouseEvent. However, things like mouse clicks do not.
I believe the mouse clicks do not get processed because macOS processes the mouse click before Blender does, meaning Blender is no longer the active window when Blender comes around to processing the mouse click, which means event = null, and as such the processing of that event exits early. But this is just speculation.

Now about the difference between GIMP on Windows and macOS.

On Windows, you click the eye dropper, then click on a pixel and the colour picker will switch to that colour.
On macOS, you click the eye dropper, and as your mouse moves around, the colour picker updates in real time to show you the colour under the pixel. When you click, that's the colour that ends up being chosen.

Note: Everything below is speculation. The GIMP source code is publicly available and will probably prove this wrong. I'm just not familiar with the GIMP code base so I'm not sure where to look.
This is purely speculation, but I suspect GIMP does this on macOS because they have the same issue. When you click outside GIMP, GIMP is no longer in focus and the mouse click doesn't get processed. So I believe GIMP is constantly sampling the colour under the pixel while GIMP is in focus, and when GIMP loses focus, it processes that as a "click". However, there are various user facing aspects that suggest this isn't how it works

> I'm assuming this isn't a platform limitation in that other programs, like Gimp, can sample outside of their windows? Just wanted to comment on that. Yes, GIMP can sample outside the GIMP window on macOS, so it's not a platform limitation. However it behaves differently from Windows and I have a suspicion on why (I'll explain it at the bottom). Just for reference, when you don't grant GIMP permission to record the screen, then colours from the GIMP colour picker outside the window are just the wallpaper at the current pixel. Which explains why the wallpaper colour was being selected in some of my tests while I hadn't accepted the permission. > This is the kicker. The _seems_ to imply that our event system on Mac is not getting events for some or all mouse events when they are outside of our windows. `eyedropper_modal` is probably not being called for the click event. Quick testing with a debugger suggests it is getting some events (mouse movement?, probably others) outside of the main window, it's just that the code seems to break before doing anything with events from outside the main window. Here's an explanation of what I mean by that: 1. `GHOST_SystemCocoa::processEvents` (`GHOST_SystemCocoa.mm`) is called from somewhere fairly frequently. 2. In `GHOST_SystemCocoa::processEvents` on line 1002 there's `event = [NSApp nextEventMatchi...];`. If Blender is not the active app, `event = null`, line 1006 breaks when `event = null` meaning the event doesn't get processed. 3. If Blender is the active app, then `event=something`, even when the mouse is outside Blender, and the mouse event gets processed in `GHOST_SystemCocoa::handleMouseEvent`. However, things like mouse clicks do not. 4. I believe the mouse clicks do not get processed because macOS processes the mouse click before Blender does, meaning Blender is no longer the active window when Blender comes around to processing the mouse click, which means `event = null`, and as such the processing of that event exits early. **But this is just speculation**. --- Now about the difference between GIMP on Windows and macOS. On Windows, you click the eye dropper, then click on a pixel and the colour picker will switch to that colour. On macOS, you click the eye dropper, and as your mouse moves around, the colour picker updates in real time to show you the colour under the pixel. When you click, that's the colour that ends up being chosen. Note: Everything below is speculation. The GIMP source code is publicly available and will probably prove this wrong. I'm just not familiar with the GIMP code base so I'm not sure where to look. This is purely speculation, but I suspect GIMP does this on macOS because they have the same issue. When you click outside GIMP, GIMP is no longer in focus and the mouse click doesn't get processed. So I believe GIMP is constantly sampling the colour under the pixel while GIMP is in focus, and when GIMP loses focus, it processes that as a "click". However, there are various user facing aspects that suggest this isn't how it works

Alaska commented

2023-08-24 06:39:46 +02:00

Further testing revealed some information.

The colour picker DOES work outside the Blender window! It's just the clicks that aren't being detected. This was discovered by moving the cursor outside the Blender window then pressing Enter/Return on the keyboard. And a pixel outside the Blender window was selected. This isn't an issue with this pull request, but an issue with some other part of Blender.
The vertical axis is backwards. If the mouse is at the top of the screen, then the pixel at the bottom of the screen is the one that's selected. (This was both before and after the commit Changed method to allow multi monitor screenshot). For reference, when the cursor is at the top of the screen, cursorPosition.y = 1440 when it should be 0. (My monitor has 1440 vertical pixels). This is an issue with this pull request.
The Blender title bar is still considered part of the Blender window, and so the "inside Blender" colour picker code is run instead of the "pixel under the cursor" code which leads to the result being black since Blender doesn't have any pixels there. This is not the case on the Windows operating system. Probably an issue somewhere else in Blender.
Selecting a pixel on a second display results in an exception. This is an issue with this pull request.

NSBitmapImageRep *bitmap = [[NSBitmapImageRep alloc] initWithCGImage:image];
Invalid parameter not satisfying: cgImage != NULL

Seems to be because image = null. My guess is that the upside down vertical coordinate is part of what's causing this. But if I put my displays side by side instead of on top of each other (This was done in the macOS settings app), the issue continues. SO maybe there's something else going on. This issue was also there prior to the Changed method to allow multi monitor screenshot commit.

I have included a video with demonstrations.

Further testing revealed some information. 1. The colour picker DOES work outside the Blender window! It's just the clicks that aren't being detected. This was discovered by moving the cursor outside the Blender window then pressing `Enter`/`Return` on the keyboard. And a pixel outside the Blender window was selected. This isn't an issue with this pull request, but an issue with some other part of Blender. 2. The vertical axis is backwards. If the mouse is at the top of the screen, then the pixel at the bottom of the screen is the one that's selected. (This was both before and after the commit `Changed method to allow multi monitor screenshot`). For reference, when the cursor is at the top of the screen, `cursorPosition.y = 1440` when it should be 0. (My monitor has 1440 vertical pixels). This is an issue with this pull request. 3. The Blender title bar is still considered part of the Blender window, and so the "inside Blender" colour picker code is run instead of the "pixel under the cursor" code which leads to the result being black since Blender doesn't have any pixels there. This is not the case on the Windows operating system. Probably an issue somewhere else in Blender. 4. Selecting a pixel on a second display results in an exception. This is an issue with this pull request. ``` NSBitmapImageRep *bitmap = [[NSBitmapImageRep alloc] initWithCGImage:image]; Invalid parameter not satisfying: cgImage != NULL ``` Seems to be because `image = null`. My guess is that the upside down vertical coordinate is part of what's causing this. But if I put my displays side by side instead of on top of each other (This was done in the macOS settings app), the issue continues. SO maybe there's something else going on. This issue was also there prior to the `Changed method to allow multi monitor screenshot` commit. --- I have included a video with demonstrations.

macOS colour picker experiment 3.mp4

13 MiB

Caleb-Cleavinger added 1 commit 2023-08-24 16:31:33 +02:00

2daadafe26 Added y axis transformation & removed multimonitor

Caleb-Cleavinger commented

2023-08-24 16:40:32 +02:00

I saw something on a stack overflow about changing having to modify y axis var. I also just removed the attempted multimonitor code because it just wasn't working. If I could hand this problem off to someone with a mac or at least more experience with Objective C/C++ that would be great because right now I have like 20 tabs of Apple's cryptic objective c documentation. I have never dealt with Cocoa before or objective C++ before. However I have loads of C++ experience & I can start working on the Wayland pixelUnderCursor which I should be able to actually test on WSL.

Caleb-Cleavinger added 1 commit 2023-08-24 16:43:31 +02:00

78b3fe45b7 Forgot to save file. The code should be updated

Caleb-Cleavinger commented

2023-08-24 16:43:59 +02:00

Sorry merged wrong version of files fixed it.

Alaska commented

2023-08-25 00:16:41 +02:00

If I could hand this problem off to someone with a mac or at least more experience with Objective C/C++ that would be great because right now I have like 20 tabs of Apple's cryptic objective c documentation.

You can do that. Leave a comment here explaining that you'd prefer someone else work on it. Change the name of this pull request to Draft: Added getPixelUnderCursor for MacOS_X or something like that , then leave a comment over on #111303 saying you've stepped down from working on the macOS implementation and others can feel free to take over it and use your work as a starting point or reference. Feel free to include my list of issues from below in your comment.

For reference I have included a list of issues that need to be resolved:

Blender does not recognise mouse clicks outside the Blender window on macOS (This is likely a stopping feature to getting this pull request accepted, but it should be fixed separately in a different pull request). To work around this issue, moving your mouse outside the Blender window and pressing Enter on the keyboard works.
There's no multi-monitor support. Selecting a pixel on a second monitor results in an exception.
The vertical axis was upside down (If your mouse is at the top of the screen, then the pixel at the bottom is sampled). Your latest commit attempted to fix it, but it just results in build errors. Use of undeclared mainScreen()
The title bar of Blender isn't sampled by the "pixel under cursor" code meaning all pixels from that area are wrong (Will likely need to be fixed seperetly).

> If I could hand this problem off to someone with a mac or at least more experience with Objective C/C++ that would be great because right now I have like 20 tabs of Apple's cryptic objective c documentation. You can do that. Leave a comment here explaining that you'd prefer someone else work on it. Change the name of this pull request to `Draft: Added getPixelUnderCursor for MacOS_X` or something like that , then leave a comment over on #111303 saying you've stepped down from working on the macOS implementation and others can feel free to take over it and use your work as a starting point or reference. Feel free to include my list of issues from below in your comment. For reference I have included a list of issues that need to be resolved: 1. Blender does not recognise mouse clicks outside the Blender window on macOS (This is likely a stopping feature to getting this pull request accepted, but it should be fixed separately in a different pull request). To work around this issue, moving your mouse outside the Blender window and pressing `Enter` on the keyboard works. 2. There's no multi-monitor support. Selecting a pixel on a second monitor results in an exception. 3. The vertical axis was upside down (If your mouse is at the top of the screen, then the pixel at the bottom is sampled). Your latest commit attempted to fix it, but it just results in build errors. `Use of undeclared mainScreen()` 4. The title bar of Blender isn't sampled by the "pixel under cursor" code meaning all pixels from that area are wrong (Will likely need to be fixed seperetly).

Caleb-Cleavinger changed title from ~~Added getPixelUnderCursor for MacOS_X~~ to Draft: Added getPixelUnderCursor for MacOS_X

2023-08-25 04:00:53 +02:00

Caleb-Cleavinger commented

2023-08-25 04:04:53 +02:00

Thank you for the suggestion Alaska! I'm gonna step down from this commit. Anyone gonna implement it you can use the code I've committed as a baseline. Just for a note beforehand the code is barely functional and will require someone with at least a little experience with cocoa to get working. Objective C++ is a can of worms I can't really deal with right now b/c of work and school responsibilities.
Thank you for all the help @Harley and @Alaska have given!

Thank you for the suggestion Alaska! I'm gonna step down from this commit. Anyone gonna implement it you can use the code I've committed as a baseline. Just for a note beforehand the code is barely functional and will require someone with at least a little experience with cocoa to get working. Objective C++ is a can of worms I can't really deal with right now b/c of work and school responsibilities. Thank you for all the help @Harley and @Alaska have given!

Pablo Guarneros referenced this pull request

2023-10-09 05:01:48 +02:00

UI: Linux & Mac Support for Eyedropper Outside Blender #111303

This pull request has changes conflicting with the target branch.

intern/ghost/intern/GHOST_System.hh

View command line instructions.

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u CC111303:Caleb-Cleavinger-CC111303

git checkout Caleb-Cleavinger-CC111303

Sign in to join this conversation.

No reviewers

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Draft: Added getPixelUnderCursor for MacOS_X #111397

Checkout