mutex/semaphore wait blocks a slot

[mzimmers]

Hi all -

I discovered some strange behavior while adding some logic to my app today. I attempt to send an HTTP message twice in quick succession (The second attempt begins before the first is finished). I'm finding that my slot is being blocked until the second attempt times out/fails.

Here's a code snippet and the output I get (I've removed a lot of code for brevity):

void SystemModel::requestWeather()
{
    qDebug() << __PRETTY_FUNCTION__ << QDateTime::currentDateTime() << "attempting to lock m_mutex.";
    if (m_mutex.tryLock(3000)) {
        qDebug() << __PRETTY_FUNCTION__ << QDateTime::currentDateTime() << "m_mutex locked.";
        QUrlQuery query;
        m_qnrGetWeather = m_messageMgr->sendGetWeather(query); // performs a QNetworkAccessManager::get().
        QObject::connect(m_qnrGetWeather, &QNetworkReply::finished, this, &SystemModel::processWeather);
    } else {
        qDebug() << __PRETTY_FUNCTION__ << QDateTime::currentDateTime() << "m_mutex NOT locked.";
    }
}

void SystemModel::processWeather() {
    QByteArray qba;
    JsonParser parser;
    QJsonDocument qjd;
    QJsonObject qjo;
    QVariant qv;

    qDebug() << __PRETTY_FUNCTION__ << QDateTime::currentDateTime() << "entering.";
    // do some stuff
    m_qnrGetWeather->disconnect();
    m_qnrGetWeather->deleteLater();
    m_mutex.unlock();
    qDebug() << __PRETTY_FUNCTION__ << QDateTime::currentDateTime() << "m_mutex unlocked.";
}

void SystemModel::requestWeather() QDateTime(2024-01-01 15:13:54.742 Pacific Standard Time Qt::LocalTime) attempting to lock m_mutex.
void SystemModel::requestWeather() QDateTime(2024-01-01 15:13:54.742 Pacific Standard Time Qt::LocalTime) m_mutex locked.
QNetworkReply* MessageManager::sendGetWeather(QUrlQuery) QDateTime(2024-01-01 15:13:54.752 Pacific Standard Time Qt::LocalTime) exiting.
void SystemModel::requestWeather() QDateTime(2024-01-01 15:13:55.194 Pacific Standard Time Qt::LocalTime) attempting to lock m_mutex.
void SystemModel::requestWeather() QDateTime(2024-01-01 15:13:58.193 Pacific Standard Time Qt::LocalTime) m_mutex NOT locked.
void SystemModel::processWeather() QDateTime(2024-01-01 15:13:58.194 Pacific Standard Time Qt::LocalTime) entering.
void SystemModel::processWeather() QDateTime(2024-01-01 15:13:58.194 Pacific Standard Time Qt::LocalTime) m_mutex unlocked.

What should happen: about 500 ms after the GET is sent, the slot should process the response.
What IS happening: nothing until the second attempt fails, then (according to the timestamps) the slot is immediately called.

This seems straightforward, but I'm obviously doing something wrong. Can anyone see what it is?

Thanks...

Oh, I also tried this with semaphores with the same result.

[Chris Kawa]

@mzimmers Well, you do have this in your requestWeather call:

QObject::connect(m_messageMgr, &MessageManager::receivedWeather, this, &SystemModel::processWeatherReply);

and it connects the same things every time, so you have three connections after three calls. When they finish you'll get 3 slot invocations for each, 9 in total. Either do that connection somewhere else once or add Qt::UniqueConnection as the 5th argument to connect, so that you don't get multiple connections for the same arguments.

[Chris Kawa]

No surprises here. This is what happens:

(thread A) First request locks the mutex.
(thread A) First request is scheduled on thread B.
(thread A) Control gets back to the event loop.
(thread A) Second request tries to lock the mutex and waits for it.
⋮
(thread B) First request is finished and queues a slot to be executed on thread A.
⋮
(thread A) Try lock from second request times out.
(thread A) Control gets back to the event loop.
(thread A) Slot is delivered and mutex unlocked.

Slots from async operations are delivered via event queue. No event processing happens while you're waiting on a lock, so slot from your first call won't be executed until you let the thread get back to the event loop.

Btw. if these requests are made from a UI thread you should not be using blocking calls like tryLock(3000) at all. This will freeze the UI.

[mzimmers]

@Chris-Kawa thanks, that makes sense, though I'm not sure where to go from here. Originally (before I discovered this issue) I wasn't using mutexes or semaphores; the request began by testing for a null QNetworkReply, and the handler cleared it when done. Same problem.

Slots from async operations are delivered via event queue

OK, so what options does this leave me? Do I have to rig something up to make the GET call synchronous? That seems rather upside-down. Or, is there another way to go about this?

Thanks...

[Chris Kawa]

@mzimmers said:

Do I have to rig something up to make the GET call synchronous?

No, definitely not. I'd say get rid of any synchronization (especially blocking ones) you can.

Or, is there another way to go about this?

It depends on what were you trying to achieve with the mutex in the first place. Ordering of requests? Ordering of the slots?

[mzimmers]

@Chris-Kawa said in mutex/semaphore wait blocks a slot:

It depends on what were you trying to achieve with the mutex in the first place. Ordering of requests? Ordering of the slots?

In the unlikely event that the server is slow in producing a response to me, I want to avoid sending a second GET until I get a reply to my first one.

I use this method in several places in my app, and it seems to work fine. This particular example is kind of a special case, because I definitely will be trying to make two calls in rapid succession. (The reason for this is at startup, my location is unknown. Once I get my location, I want to get a new weather code.)

[Chris Kawa]

@mzimmers said:

I use this method in several places in my app, and it seems to work fine

It works fine on your network. This is a terrible terrible way of doing this. Very manual and error prone.

If you want a request queue make a request queue - create a vector of requests. Push a request to it and signal. Have a processor that waits for that signal. If there's something in flight already do nothing, if there isn't pop a request from the vector and post it. When a slot arrives or request times out signal the processor again to check if there's more requests to post in the queue.

[mzimmers]

@Chris-Kawa said in mutex/semaphore wait blocks a slot:

OK, I think I can do that. Everything you said makes sense to me except:

Have a processor that waits for that signal

Not sure what you mean by "processor" - do you mean a handler/slot?

[Chris Kawa]

@mzimmers said in mutex/semaphore wait blocks a slot:

Not sure what you mean by "processor" - do you mean a handler/slot?

Yup, something that processes the queue. Can be just a slot or you could encapsulate the queue and its handling in a dedicated class.

[mzimmers]

@Chris-Kawa I'm chasing my tail on this. Is it possible that part of my problem is that I'm attempting to do everything from a single thread?

Here's my network code, as minimal as I can make it:

requestWeather(1);
requestWeather(2);
requestWeather(3);

void SystemModel::requestWeather(int i)
{
    QUrlQuery query; 
    // fill out query
    QObject::connect(m_messageMgr, &MessageManager::receivedWeather, this, &SystemModel::processWeatherReply);
    m_messageMgr->addWeatherRequest(query);
}

void MessageManager::addWeatherRequest(QUrlQuery query)
{
    QNetworkRequest l_request;
    // fill out l_request
    // add to the list.
    m_weatherRequestList.append(l_request);
    processList();
}

void MessageManager::processList() {
    do {
        if (m_weatherReply != nullptr) {
            qWarning() << __PRETTY_FUNCTION__ << "m_weatherReply != nullptr; not attempting to process list.";
            continue;
        }
        while (!m_weatherRequestList.isEmpty()) {
            QNetworkRequest request = m_weatherRequestList.front();
            m_weatherReply = m_manager.get(request);
            if (m_weatherReply->error() != QNetworkReply::NoError) {
                continue;
            }
            QObject::connect(m_weatherReply, &QNetworkReply::finished, this, &MessageManager::notifyRequester);
            m_weatherRequestList.removeFirst();
        }
    } while (false);
}

void MessageManager::notifyRequester() {
    emit receivedWeather(m_weatherReply);
    m_weatherReply->deleteLater();
}

My slot processWeatherReply() isn't called until all three of the calls to requestWeather() have exited. I can provide timestamped telltales if desired.

[Chris Kawa]

@mzimmers The problem is you're trying to empty the whole queue at the same time (while (!m_weatherRequestList.isEmpty()) ), which defeats the purpose of a queue. Don't do that.

Remove both loops from processList. Pop just the first request (if the list is not empty), post it and that's it, exit the function. In the notifyRequester callback, after you finish processing the reply, call processList again, so that it posts another request. This will keep happening until the list is emptied.

[mzimmers]

@Chris-Kawa well, progress of sorts. Here's my cleaned up code (along with the slot):

void MessageManager::processList() {
    if (!m_weatherRequestList.isEmpty()) {
        QNetworkRequest request = m_weatherRequestList.front();
        m_weatherReply = m_manager.get(request);
        QObject::connect(m_weatherReply, &QNetworkReply::finished, this, &MessageManager::notifyRequester);
        m_weatherRequestList.removeFirst();
    }
}

void MessageManager::notifyRequester() {
    emit receivedWeather(m_weatherReply);
    m_weatherReply->deleteLater();
    processList();
}

void SystemModel::processWeatherReply(QNetworkReply *qnr) {
    qba = qnr->readAll(); // only the first read returns anything.
    // lots more stuff
    qnr->disconnect();
    qnr->deleteLater();
    qnr = nullptr;
}

So, the three calls to requestWeather() all complete before the first slot is called, but I suppose that's to be expected now because that routine is doing so little work. (yes?)

REMAINING ISSUE:

The first transaction works correctly and returns the expected payload. Subsequent calls, however, return nothing from readAll().

I tried inserting a 1 second delay between the calls, just in case the server had some kind of DDoS protection, but that didn't change anything.

EDIT:

The one-second delay did indeed help; now all three calls return valid payload. Now the problem is I'm getting an extra signal from the message manager, but I'll dig into that and report back.

[mzimmers]

Well, I think this one is mostly solved. For some reason, my processWeatherReply() slot is getting called more often than its only signal is emitted, which is...weird. The "extra" calls to the slot are using the same qnr as the preceding "good" call, but the readAll() returns nothing.

I can easily code around this issue, though I would like to know how I'm getting those extra calls. Some kind of weird EOL that's not getting picked up in the first read, maybe?

[Chris Kawa]

@mzimmers Don't insert artificial delays. It's one second on your machine and might be 10 on mine.

You removed too much from processList i.e. the check to see if something is already in flight: if (m_weatherReply != nullptr). That needs to stay, because now you're, again, posting multiple requests at the same time and overwriting m_weatherReply, which defeats the purpose of a queue and the reason you're not getting anything from readAll. You're reading multiple times from the same overwritten reply.

Add back the check and exit if m_weatherReply is not null. Remember to set it to null after you call deleteLater() on it, otherwise you'll get a stale pointer in the processList() call that follows.

You're also deleting the reply twice, once in notifyRequester and once in processWeatherReply. It works by accident, since you're not getting back to the event loop inbetween, but that's a bug. Delete it once in notifyRequester. You also don't need to disconnect it beforehand. Deleting an object breaks all of its connections automatically.

[mzimmers]

@Chris-Kawa

I was only using the delays to test whether part of my problem was a DDoS defense (and I think it was). The multiple calls, and the delays between them, have been removed.

I'm still getting some empty reads. Here's my updated network code:

void MessageManager::addWeatherRequest(QUrlQuery query)
{
    QNetworkRequest l_request;

    // add to the list.
    m_weatherRequestList.append(l_request);
    processList();
}

void MessageManager::processList() {
    if (m_weatherReply == nullptr) {
        if (m_weatherRequestList.isEmpty()) {
            qDebug() << __PRETTY_FUNCTION__ << "nothing in list to process.";
        } else {
            QNetworkRequest request = m_weatherRequestList.front();
            m_weatherReply = m_manager.get(request);
            if (m_weatherReply->error() == QNetworkReply::NoError) {
                QObject::connect(m_weatherReply, &QNetworkReply::finished, this, &MessageManager::notifyRequester);
            } 
            m_weatherRequestList.removeFirst();
        }
    } else {
        qWarning() << __PRETTY_FUNCTION__ << "attempting to use m_weatherReply which is already in use.";
    }
}

void MessageManager::notifyRequester() {
    qint64 bytesAvailable = m_weatherReply->bytesAvailable();
    if (bytesAvailable > 0) {
        qDebug() << __PRETTY_FUNCTION__ << "emitting receivedWeather().";
        emit receivedWeather(m_weatherReply);
    }
    m_weatherReply->deleteLater();
    m_weatherReply = nullptr;
    processList();
}

I've verified that the signal is only emitted twice now (which is correct) but I'm entering my slot more often (and getting those empty reads).

[Chris Kawa]

@mzimmers Well, you do have this in your requestWeather call:

QObject::connect(m_messageMgr, &MessageManager::receivedWeather, this, &SystemModel::processWeatherReply);

and it connects the same things every time, so you have three connections after three calls. When they finish you'll get 3 slot invocations for each, 9 in total. Either do that connection somewhere else once or add Qt::UniqueConnection as the 5th argument to connect, so that you don't get multiple connections for the same arguments.

[mzimmers]

@Chris-Kawa bingo! Good catch there - I didn't realize that you could "stack" connections; I thought any additional, identical connections were merely superfluous. I moved that connection to the c'tor of my SystemModel object, and it works fine.

As a postscript to this exercise...does my approach now make sense in general, or is this still an undesirable way to go about this?

[Chris Kawa]

@mzimmers said:

does my approach now make sense in general, or is this still an undesirable way to go about this?

It's a valid solution. Whether or not it's best for your case is something you have to answer yourself.