How does Qt synthesize Mouse Events from Touch Events?

MicHes

Hi all,
I'm trying to find an error in my Qt application (Widget-based on 32-bit x86 Linux using Qt 5.15.2 on x.org, no QML) running on a number of PCs which use a touchscreen made by ILITEK. Those touch screens report finger presses as touch events, which Qt somehow turns into Mouse events, which in turn effect button presses etc.

Except it doesn't, at least not reliably.

I have witnessed the application failing to synthesize any mouse events after an arbitrary running time, this rendering the QApplication inoperable, as those machines have no physical mouse. After a restart of the application (which, also restarts X11), the mouse events return for a while; there seems to be no rhyme or reason what triggers this behaviour; sometimes, this does not happen for days, sometimes it can occur within minutes of application startup.

The QApplication is created using default attributes, which as far as I know includes Qt::AA_SynthesizeMouseForUnhandledTouchEvents. I do not change any attributes during runtime.

So, my questions are:

Where is the generation of mouse events from touch events done in Qt?
What can cause this generation to fail or to cease?
What can I do to remedy the situation?

Any help is appreciated, looking forward to hearing your suggestions. If necessary, I can provide logs of the problem, though you only see that no mouse events are generated.

Best regards, Michael

Asperamanca

Go into Qt sources under qtbase/src/gui and search for usages of Qt::AA_SynthesizeMouseForUnhandledTouchEvents
I find a code block that seems to do the synthsis in

Src\qtbase\src\gui\kernel\qguiapplication.cpp

One workaround is not to rely on synthesized events, but instead handle all touch events in your application. Otherwise, it may be worthwhile to search the Qt bug database for existing bug reports on this.

MicHes

Thanks for providing the pointer to the Qt source, @Asperamanca; I will look into that.

In the end, I do not want to deal with the events myself at all - is there another way to automatically get button click events directly from the touch events? I just want my users to be able to use the buttons and other elements on screen, and when the mouse events cease, it's currently so that the whole application does not react at all anymore to user actions at the touch panel, even though there still are touch events.

Cheers, Michael.

Asperamanca

@MicHes What kind of buttons are you talking about? Widgets? GraphicsView?

MicHes

Hi @Asperamanca,
yes, they are QWidgets; most of them are specifically QPushButtons.

Cheers, Michael

Asperamanca

@MicHes
If you get the effect in a debugger, you could set a breakpoint into the relevant Qt code and see what happens.
As mentioned, I would do a brief bug research. Sometimes, workarounds are mentioned in the comments.

You could try using a newer Qt version. Migrating to Qt6 is going to be some work, but I guess you'll have to do it sooner or later anyway.
I also believe KDAB released some open-source patches for Qt 5.15. I don't know the details since I use a commercial license.

And, as I said, you could do the touch handling yourself. That is probably not as much work as it sounds, as long as you are happy with simple touch gestures like "tap". For example, you can install an event filter at your main window(s), look for touch events, identify the button using QWidget::childAt and QAbstractButton::hitButton, and trigger the button if the touch is on the button.

MicHes

@Asperamanca ,
no, unfortunately, this has so far only happened on PCs deployed; it usually takes a few hours of normal operation (which is difficult to reproduce in testing) for the problem to appear.

I have searched the Qt bugtracker, but haven't really found anything similar. A newer version is out of the question at the moment (we just switched to Qt5 six months ago), and we would not want to do the switch unless there is something specific fixed in that release that could explain this problem.

So, your suggestion would be to implement a EventFilter in QMainWindow which

uses the x/y coordinates of the TouchEvent for childAt() to find out whether a button is at that coordinates
if that returns a widget, then call widget->hitButton(x,y) to check whether the clickable area was hit
and if so, create a MouseButtonPress event for that widget? Or do I just call widget->clicked()?

To avoid double clicks, I should also throw away any events Qt does synthesize by itself, right?

I can try that, though I'd to like make sure that this also works when the touchscreen does generate mouse events by itself.

Note: I cannot call hitButton from the event filter:

mySource.cpp:42: error: ‘virtual bool QPushButton::hitButton(const QPoint&) const’ is protected within this context
mySource.cpp: In member function ‘virtual bool CGlobalEventFilter::eventFilter(QObject*, QEvent*)’:
mySource.cpp:42:54: error: ‘virtual bool QPushButton::hitButton(const QPoint&) const’ is protected within this context
   42 |                 if (thisButton->hitButton(coordinates))
      |

What shall I do instead?

MicHes

Hi @Asperamanca ,
I have now added the following code to the event filter of my by QMainWindow derivate class:

    // TEST ROUTINE: generate button presses from touch events ourselves
    if (event->type() == QEvent::TouchBegin)
    {
        QTouchEvent *thisEv = static_cast<QTouchEvent*>(event);
        QString debugTxt = "eventFilter(): " + screen->objectName() + " received TouchBegin event";
        logger->log(INFO, debugTxt);

        logger->log(INFO,QString("eventFilter(): checking point list..."));
        QList<QTouchEvent::TouchPoint> pointList = thisEv->touchPoints();

        for (QTouchEvent::TouchPoint &point: pointList)
        {
            QPoint coordinates = point.screenPos().toPoint();
            QWidget *thisWidget = screen->childAt(coordinates);
            QAbstractButton *thisButton = dynamic_cast<QAbstractButton*>(thisWidget);

            logger->log(INFO, QString("eventFilter(): checking coordinates (%1/%2)")
                            .arg(coordinates.x()).arg(coordinates.y()));

            if (thisButton != nullptr)
            {
                // commented out because hitButton() is protected and thus not accessible from here
                /*logger->log(INFO, QString("eventFilter(): found button [%3] at (%1/%2)")
                                .arg(coordinates.x()).arg(coordinates.y()).arg(thisButton->objectName()));
                if (thisButton->hitButton(coordinates))
                {*/
                    logger->log(INFO, QString("eventFilter(): issuing click for button [%1]")
                                    .arg(thisButton->objectName()));

                    QTimer::singleShot(0, thisButton, [&]{ thisButton->click(); });
                /*}
                else
                {
                    logger->log(INFO, QString("eventFilter(): (%1/%2) outside hit box for [%3] ignoring event")
                                    .arg(coordinates.x()).arg(coordinates.y()).arg(thisButton->objectName()));
                }*/
            }
            else
            {
                logger->log(INFO, QString("eventFilter(): no button found at (%1/%2)")
                                .arg(coordinates.x()).arg(coordinates.y()));
            }
        }
        CFCGuiInteractionEvent ev(CFCInteractions::AnyTouch);
        notifier->Notify(&ev);
        logger->log(INFO, QString("eventFilter(): point list checked; TouchBegin event processed."));
        return true;
    }

This seems to work most of the time, however, this cases crashes due to segmentation faults at seemingly arbitrary points while the software is in libQt5Core.so.5.15.2 or libQt5Widgets.so.5.15.2.

Could this be due to me calling

thisButton->click();

and not

thisButton->clicked();

in the QTimer function? I replaced this because clicked is a signal and not a function, I could of course try

emit thisButton->clicked();

Would this help?

Asperamanca

I think 'click' is the right slot to use. I don't believe this would trigger the crash. The segmentation fault has no relation to your code directly in the call stack?

You can either ignore synthesized events (see Qt::MouseEventSource from the QMouseEvent)...BTW which type of synthsized event do you get normally
Alternatively, you could try disabling event synthesis, see application attribute Qt::AA_SynthesizeMouseForUnhandledTouchEvents

As for hitButton...you could make your own class derived from QPushButton, but then you'd need to replace all existing QPushButton instances with your own. This may only be a rename, but it may be more complicated (especially if you use the designer, I guess).

Looking at QPushButton's implementation of hitButton, it is probably not trivial code to replicate.
Alternatively, you could synthesize the event yourself, and send it into the event system (instead of calling the 'click' slot). That way, you would also bypass the "hitButton" issue, as the button would just handle a mouse event.

One intermediate step (if that is possible) would be to add pure touch logging with event information and let it run on the deployed PC in order to find the root cause why synthesis stops working.

MicHes

Hi @Asperamanca ,

I've now completed the following solution which works for me at the moment:

        if (event->type() == QEvent::TouchBegin)
        {
            QTouchEvent *thisEv = static_cast<QTouchEvent*>(event);
            QString debugTxt = "eventFilter(): " + screen->objectName() + " received TouchBegin event";
            logger->log(INFO, debugTxt);

            logger->log(INFO, QString("eventFilter(): checking point list..."));
            QList<QTouchEvent::TouchPoint> pointList = thisEv->touchPoints();

            for (QTouchEvent::TouchPoint &point: pointList)
            {
                QPoint coordinates = point.screenPos().toPoint();
                QWidget *thisWidget = screen->childAt(coordinates);
                QAbstractButton *thisButton = dynamic_cast<QAbstractButton*>(thisWidget);

                logger->log(INFO, QString("eventFilter(): checking coordinates (%1/%2)")
                                .arg(coordinates.x()).arg(coordinates.y()));

                if (thisButton != nullptr)
                {
                    if (!thisButton->isVisible())
                    {
                        logger->log(INFO, QString("eventFilter(): inoring invisible button [%1]")
                                        .arg(thisButton->objectName()));
                    }
                    else if (!thisButton->isEnabled())
                    {
                        logger->log(INFO, QString("eventFilter(): inoring disabled button [%1]")
                                        .arg(thisButton->objectName()));
                    }
                    else
                    {
                        logger->log(INFO, QString("eventFilter(): issuing click for button [%1/%2]")
                                        .arg(thisButton->objectName()).arg((uint64_t) thisButton));
                        QTimer::singleShot(0, thisButton, [thisButton](){ if (thisButton != nullptr) 
						                     { qDebug() << QString("eventFilter(): triggering pressed [%1]").arg((uint64_t) thisButton) << endl;
								               thisButton->setDown(true);
                                               emit thisButton->pressed();} 
											 });
                        QTimer::singleShot(100, thisButton, [thisButton](){ if (thisButton != nullptr) 
						                     { qDebug() << QString("eventFilter(): triggering released/clicked [%1]").arg((uint64_t) thisButton) << endl;
                                               emit thisButton->released();
                                               emit thisButton->clicked();
                                               thisButton->setDown(false);} 
											 });
                    }
                }
                else
                {
                    logger->log(INFO, QString("eventFilter(): no button found at (%1/%2)")
                                    .arg(coordinates.x()).arg(coordinates.y()));
                }
            }
            CFCGuiInteractionEvent ev(CFCInteractions::AnyTouch);
            notifier->Notify(&ev);
            logger->log(INFO, QString("eventFilter(): point list checked; TouchBegin event processed."));
            return true;
        }
    if (event->type() == QEvent::MouseButtonPress)
    {
        QMouseEvent *thisEv= static_cast<QMouseEvent *>(event);
        QAbstractButton* button = qobject_cast<QAbstractButton*>(object);

        if ( (thisEv->source() == Qt::MouseEventSource::MouseEventSynthesizedByQt) && (!screen->isHidden()) )
        {
            logger->log(INFO, QString("eventFilter(): ignoring synthesized button press"));
            event->ignore();
            return true;
        }
    }

However, I would still like to find the root cause of Qt5 inadvertently ceasing to provide synthesized mouse events, or at least a more general solution which work for other widgets as well.

You suggested:

"Alternatively, you could synthesize the event yourself, and send it into the event system (instead of calling the 'click' slot). That way, you would also bypass the "hitButton" issue, as the button would just handle a mouse event."

And how would I go about that? I've looked into the Qt5 source code, and as far as I can tell, Qt uses a private member function to generate those synthesized mouse events, so I cannot replicate that myself. Is there another way to generate pressed/released events for the left mouse button?

As for:

"One intermediate step (if that is possible) would be to add pure touch logging with event information and let it run on the deployed PC in order to find the root cause why synthesis stops working."

We are already logging every touch, keyboard, mouse and tablet event that reaches the QMainWindow; that's how we know that Qt5 ceases to synthesize mouse events at some point. What we don't know is why it is doing that - and as far as I can tell, we would have to add debugging to the Qt5 libs to find out, right?

Cheers, Michael.

Asperamanca

@MicHes said in How does Qt synthesize Mouse Events from Touch Events?:

"Alternatively, you could synthesize the event yourself, and send it into the event system (instead of calling the 'click' slot). That way, you would also bypass the "hitButton" issue, as the button would just handle a mouse event."

Ah yes. I only ever did this in a testing environment, and there I created the events myself from scratch (and didn't care whether all parameters of the event were correctly filled, as long as my tests worked). In principle, you can use private APIs form Qt, but you must expect them to break if you switch the Qt version.

@MicHes said in How does Qt synthesize Mouse Events from Touch Events?:

"One intermediate step (if that is possible) would be to add pure touch logging with event information and let it run on the deployed PC in order to find the root cause why synthesis stops working."

So you already know that the touch events keep coming, but event synthetisation doesn't work? I think that's about as far as your logging can go.

BTW, could this be your bug?
https://bugreports.qt.io/browse/QTBUG-98519

malikjahanzeb

Hi @MicHes, @Asperamanca,

It seems I am having a similar issue with PyQt5/PySide2 on a raspberry pi. My application is a simple fullscreen multiscreen one that keeps switching between screens based on user input. The application would freeze (not respond to touch even on buttons) occasionally, especially when a lot of touching is done while the screens are switching. When the freeze happens, an attached mouse to the pi's USB port can still successfully click the buttons, just the touch is failing. The frozen window's timer is still working so that when the window times out, it is switched with another and touch is restored.

I have tried stripping the app into a minal two window switching demo with all extra code removed and each window having one button to switch to the other window. The issue can still be reproduced with it, although it is harder with it due to light UI probably. The issue happens with PyQt5 and PySide2 alike.

We have these Pi's in the field with the app running and every now and then one of them fails. If the user is patient enough to wait for the screen's timeout, they will have the functionality restored or else they are very annoyed.

I am uploading a video demo here.

https://drive.google.com/file/d/15lYmnrrIulGp0JrK56SEAY-BBiXhNlTU/view?usp=sharing

Based on your opinion is this the same issue that @MicHes was having? Do you guys have ideas on how to work around this?

Asperamanca

@malikjahanzeb
Do you know whether the touch events even arrive, and the issue lies in synthesizing the mouse events?
I had an issue with a Touch PC where sometimes (in a kind of standby mode), no events whatsoever would arrive at my application for >5 s.
What version of Qt are you using?