Get pixel color from process / window (or entire screen) in QT
-
Hey, I want to read specific pixel from game, to do action when pixel have specific color. I tried to do it in C++, but it doesnt work when game is in fullscreen (I have to alt-tab to "refresh screen capture"[?]). In windowed mode it's working, but I wanted it to work in fullscreen too. How can I "Get Pixel Color" from entire screen / process / window in Qt way?
-
Hi
When the game is fullscreen it's running its own openGL or DirectX context and i don't think Qt can read that.
When it's rendered to a window, then its also part of the normal screen and Qt can see it.If you are lucky, then maybe @Chris-Kawa is around sometimes and he might know more/if you can do it with
windows API. -
Getting a pixel color from window of another processes is generally a "bad thing" ™, especially when it's an app that uses some accelerated API like DirectX, OpenGL or Vulkan, like games usually do. The whole point of using those APIs is to bypass as much of OS buffering and syncing as possible, and fullscreen mode is the especially aggressive mode in doing so. Reading a pixel from a window of a fullscreen app is forcing OS to do all kinds of ugly and slow synchronization & copy between GPU and CPU memory. A window manager does all sorts of trickery in fullscreen mode. It's likely that it redirects the output of the API to some entirely different, more direct buffer than that of a window when compositing it in windowed mode. This is probably why it doesn't work for you - reading from a window that doesn't get any actual content when in fullscreen.
So best case scenario - you're hindering that game's performance by doing this. Worst - it doesn't work at all (like you observed). The "correct"™ way to do it is to make the game read that value for you directly from its frame buffer in an asynchronous way via queries that given graphics API provides and expose it to your external process (either through window messages, shared memory or some other means).
If you're not in control of the game's code then my question would be - why are you doing this? Are you making some sort of bot? If that's the case I'm not helping you ;) -
Getting a pixel color from window of another processes is generally a "bad thing" ™, especially when it's an app that uses some accelerated API like DirectX, OpenGL or Vulkan, like games usually do. The whole point of using those APIs is to bypass as much of OS buffering and syncing as possible, and fullscreen mode is the especially aggressive mode in doing so. Reading a pixel from a window of a fullscreen app is forcing OS to do all kinds of ugly and slow synchronization & copy between GPU and CPU memory. A window manager does all sorts of trickery in fullscreen mode. It's likely that it redirects the output of the API to some entirely different, more direct buffer than that of a window when compositing it in windowed mode. This is probably why it doesn't work for you - reading from a window that doesn't get any actual content when in fullscreen.
So best case scenario - you're hindering that game's performance by doing this. Worst - it doesn't work at all (like you observed). The "correct"™ way to do it is to make the game read that value for you directly from its frame buffer in an asynchronous way via queries that given graphics API provides and expose it to your external process (either through window messages, shared memory or some other means).
If you're not in control of the game's code then my question would be - why are you doing this? Are you making some sort of bot? If that's the case I'm not helping you ;)@Chris-Kawa
I dont want to do bot, and I dont want to use any sort of "hacking" practise (memory reading or sth like that). I want to read specific pixel to do Minecraft tool, like: When user hold button, it will spam the button until the pixel got changed (use a water bucket on a ground to avoid fall damage).I thought about getting around it, like get image / screenshot of specific part of screen, but it's refresh only when alttab, so it's only possible in Windowed Mode. Thank You anyway.
-
@Chris-Kawa
I dont want to do bot, and I dont want to use any sort of "hacking" practise (memory reading or sth like that). I want to read specific pixel to do Minecraft tool, like: When user hold button, it will spam the button until the pixel got changed (use a water bucket on a ground to avoid fall damage).I thought about getting around it, like get image / screenshot of specific part of screen, but it's refresh only when alttab, so it's only possible in Windowed Mode. Thank You anyway.
@BD9a said:
I dont want to do bot
When user hold button, it will spam the button until the pixel got changedI hate to break it to you but that's basically what a bot is :)
Anyway, like I said, reading a pixel is not as simple thing as it may seem. That pixel is in some buffer in your GPU's memory and to "read" it the OS has to sync that buffer, transfer that pixel to a CPU memory (RAM) and then expose it through the window manager to whatever API you're using for that - WinAPI, GDI or whatever. You can sorta do that when in windowed mode because the OS is already doing that syncing to allow for window composition, but when you enter fullscreen all of those shenanigans get bypassed and the CPU side buffer that you read from is not updated. That's why it only works when you alt-tab out of it.
In short - it's not simple to do, not through Qt's high level abstractions at least, and even if you do it through the OS specific APIs it's very fragile and can easily break after some random update.
Minecraft is pretty sandboxy game and, although I never played with it myself, it seems to have some sort of scripting API. It's always better to make mods with the officially provided tools so I'd look into that. Maybe there's an API there to do what you want without reading pixel colors.
