Get Pixel-Values on Screen (KDE)

tim-hilt

I use KDE as my desktop-environment on my Linux machine which, as you might know. is written in Qt. I wanted to dive a bit into KDE development and wanted to find out how I can grab the pixel-values on screen; more specifically the pixel-values for the currently active window.

Since there doesn't seem to be a KDE-API for this specific functionality, I thought about KDEs screenshot-utility Spectacles. I was pointed to this file, which seems to represent the newest version of Spectacle. Spectacle also implements the functionality to make a screenshot of the currently active window. It seems to be a pretty good basis for my experiments!

I'm arguably pretty inexperienced with C++ in general and Qt in particular, so I honestly can't make a lot of sense of the code.

Can anyone help me to understand how Spectacle is able to get the pixel-values I'm searching for? Is there a more minimal way to do it? I honestly am just looking for a 2D-Array of pixel values, that represent the currently active window without it's window-decorations.

Thanks in advance!

wrosecrans

Spectacle is surprisingly complicated to pick apart for a beginner, because it supports some odd features across multiple platforms, in a slightly convoluted way. For basic screenshots, Qt has a simple example: https://doc.qt.io/qt-6/qtwidgets-desktop-screenshot-example.html

QScreen::grabWindow will do what you want. You can get a screenshot of the whole screen or a specific window's contents. You shouldn't need anything KDE-specific to get started.

mpergand

@tim-hilt said in Get Pixel-Values on Screen (KDE):

pixel values, that represent the currently active window without it's window-decorations.

Do you mean the background color ?
You can retreive basic colors of a widget/window with QPalette
You can use grab() to get a pixel color:

QPixmap QWidget::grab(const QRect &rectangle = QRect(QPoint(0, 0), QSize(-1, -1)))
Renders the widget into a pixmap restricted by the given rectangle. If the widget has any children, then they are also painted in the appropriate positions.
If a rectangle with an invalid size is specified (the default), the entire widget is painted.
This function was introduced in Qt 5.0.
Note: This function can be invoked via the meta-object system and from QML. See Q_INVOKABLE.
See also render() and QPixmap.

I experienced stange bugs with the grag() function unfortunately.

wrosecrans

Spectacle is surprisingly complicated to pick apart for a beginner, because it supports some odd features across multiple platforms, in a slightly convoluted way. For basic screenshots, Qt has a simple example: https://doc.qt.io/qt-6/qtwidgets-desktop-screenshot-example.html

QScreen::grabWindow will do what you want. You can get a screenshot of the whole screen or a specific window's contents. You shouldn't need anything KDE-specific to get started.

tim-hilt

@wrosecrans thanks for your answer! This is very helpful. Two issues have come up when I tried adapting the code from the Screenshot-Example:

The existing code I'm trying to update with the Screenshot-Example was previously using Qt5. I understand that to use the QScreen::grabWindow()-function, I have to use Qt6. I did install Qt6 on my machine, however when I try to change the CMakeLists.txt from

find_package (Qt5 REQUIRED COMPONENTS
    Core
    Gui
)

# ...

target_link_libraries (myprogram
    PUBLIC
        Qt5::Core
        Qt5::Gui

to this:

find_package (Qt6 REQUIRED COMPONENTS
    Core
    Gui
    Widgets
)

# ...

target_link_libraries (myprogram
    PUBLIC
        Qt::Core
        Qt::Gui
        Qt::Widgets

I see the following error in CMakes configure-step:

[cmake] -- Configuring done
[cmake] CMake Error: The INTERFACE_QT_MAJOR_VERSION property of "Qt6::Widgets" does
[cmake] not agree with the value of QT_MAJOR_VERSION already determined
[cmake] for "myprogram".

When trying to incorporate the following:

  QScreen *screen = QGuiApplication::primaryScreen();
  auto a = screen->grabWindow(0);

I get red highlighting under the arrow-accessor along with the following error:

Member access into incomplete type 'QScreen'

I can imagine that the second error will be fixed if get the first error to go away. Any idea on that? Thanks again for your response.

If you want to take a look at the code, you can find the mentioned snippets here (CMakeLists.txt) and here (Decoration.cc).

Christian Ehrlicher

Clean your build directory after switching from Qt5 to Qt6 and use Qt6::Foo instead Qt::Foo to be sure to use the Qt6 libs.

Member access into incomplete type 'QScreen'

As always - when you want to use a class you have to include the header where the class is defined.

tim-hilt

@Christian-Ehrlicher I might really have included the wrong header! Thanks for pointing that out.

As for the build directory: I see the error even if I delete the build-directory and configure Cmake on a clean slate. Anything else I can check?

Christian Ehrlicher

Since you did not show the whole CMakeLists.txt you did not tell us that you're also trying to link against KDE5 libs - this will not work you can't mix Qt5 and Qt6.

tim-hilt

@Christian-Ehrlicher does that mean the idea of @wrosecrans won‘t work?

Christian Ehrlicher

@tim-hilt said in Get Pixel-Values on Screen (KDE):

does that mean the idea of @wrosecrans won‘t work?

It does if you use the qt-5 docs instead the qt-6 ones

tim-hilt

@Christian-Ehrlicher thanks. That did in fact work.

tim-hilt

@Christian-Ehrlicher @wrosecrans The original question is answered so I marked the thread as solved. However, allow me to ask one more question:

My ultimate goal is to find the most prominent color in the first row of pixels in a window. My current, naive implementation looks like this:

  QScreen *screen = QGuiApplication::primaryScreen();
  auto window = screen->grabWindow(0).toImage(); // Convert to QImage

  // I have no way to debug, so I'm logging to journalctl
  syslog(LOG_NOTICE, "[Window Decoration] Width: %d, Height: %d",
         window.width(), window.height());

  std::unordered_map<QRgb, unsigned int> cols{};

  for (int i = 0; i < window.width(); i++) {
    cols[window.pixel(i, 0)]++; // Count color-occurences
  }

However the above code always logs, that the window is 0x0 pixels, so no pixel-value can actually be observed. Should I open another thread for this issue? Any idea what could have gone wrong? I added a 500ms delay before calling the above code, but that didn't change the behavior.

Here you can see the code in context.

mpergand

@tim-hilt
I'm using this kind of code:

       QScreen* screen=qApp->primaryScreen();
       screen=window()->windowHandle()->screen();
       if(!screen) return;
       QPoint pt=window()->geometry().topLeft();
       auto pixmap= screen->grabWindow(this->winId(), pt.x(),pt.y(),300,height());
       auto color = pixmap.toImage().pixelColor(2,2);
       qDebug()<<hex<<color<<color.rgb();

tim-hilt

Hi @mpergand, thanks for the reply. It seems like you have an instance of QGuiApplication that you hold in the variable qApp. I unfortunately don't have any of that. I generally don't really understand your code. Why are you initializing screen with one value, if it's directly overwritten in the second line?

I logged the screen-name that I get when using QGuiApplication::primaryScreen(). It prints eDP-1, which is my laptop-screen. So I correctly get the screen. The issue thus seems to happen when I use grabWindow. screen->grabWindow(0) should grab the pixels of the entire screen, but that doesn't seem to be the case for me.

I also tried screen->grabWindow(QApplication::activeWindow()->winId()) to get the window-id of the currently active window, but that just crashes KDE.

I'm unfortunately out of ideas for now. Any idea what else I could try?

mpergand

@tim-hilt said in Get Pixel-Values on Screen (KDE):

Why are you initializing screen with one value, if it's directly overwritten in the second line?

You're right, that's old (test) code that I don't use because grab() doesn't work well:

the colors of the grabbed image are ligther then the original
there's a strange redraw bug on the original window after the grab operation (at least on mac)

Try with a portion of the screen:
auto pixmap= screen->grabWindow(winId,100,100,200,200);

In my opinion, the grab method is not really reliable.

tim-hilt

@mpergand the pixmap-size is still 0x0 pixels, even if I call grabWindow with coordinates. In my application I call .toImage() on the returned pixmap. However both the pixmap and the image are both of size 0x0 pixels. So it also doesn't get lost in the conversion-process from pixmap to image.

mpergand

Test on KDE 21.04 in virtualbox.

int main(int argc, char *argv[])
{    
 QApplication app(argc, argv);

  QWidget w1;
  w1.show();
  w1.resize(300,200);

  QTimer::singleShot(500, 0,[&w1]()
     {
      QScreen *screen = QGuiApplication::primaryScreen();
      auto screen_img = screen->grabWindow(0).toImage();
      qDebug()<<screen_img;
      auto win_img= screen->grabWindow(w1.winId());
      qDebug()<<win_img;
     }
 );
 return app.exec();
}

QImage(QSize(1891, 972),format=QImage::Format_RGB32,depth=32,devicePixelRatio=1,bytesPerLine=7564,sizeInBytes=7352208)
QPixmap(QSize(300, 200),depth=32,devicePixelRatio=1,cacheKey=0x700000001)

Work as expected.

tim-hilt

@mpergand can you provide a CMakeLists.txt or the compile-command you used?

and just to make that clear: I’m using the same code distilled to

QScreen *screen = QGuiApplication::primaryScreen();
auto screen_img = screen->grabWindow(0).toImage();

So no app.exec or something like that. That shouldn’t matter however, since screen->name did resolve to the expected screen name.

If you have the capacity, you could try running my code (linked above). It‘s actually a window-decoration for KDE! The Readme contains installation-instructions and I added logs that you can read through journalctl.

tim-hilt

Some more news about this:

I used qDebug to log the returned object for QScreen::grabWindow. It prints QPixmap(null)! So the grabWindow-function doesn't seem to work on my machine. Versions are printed below.
I can observe the same behavior when compiling and running the Screenshot-Example that you guys were referring to earlier. When I execute the compiled binary and try to take a screenshot, this is the output I'm getting:

$ ./screenshot
QPixmap::scaled: Pixmap is a null pixmap
QPixmap::scaled: Pixmap is a null pixmap
QPixmap::scaled: Pixmap is a null pixmap

Versions:

qt5-base: 5.15.7+kde+r177-1
kwin_wayland --version: kwin 5.26.4
plasmashell --version: plasmashell 5.26.4
uname -a: Linux t14s 6.1.1-arch1-1 #1 SMP PREEMPT_DYNAMIC Wed, 21 Dec 2022 22:27:55 +0000 x86_64 GNU/Linux

@Christian-Ehrlicher can you give me your guess as to where this issue should be further pursued? Is it likely due to Arch Linux, KDE or Qt?

tim-hilt

I have since found out that grabWindow doesn’t work under Wayland. So I either have to find another way or resort to X11 again.