Periodic producer signal to variable execution time consumer slot
In quite a few Qt based application I encountered the same problem: one thread produces data at a constant rate emitting a signal, and a consumer processes the data received through a slot in another thread. The processing of the data in the consumer may take a bit longer from time to time, much longer than the producer period.
What happens depends a bit on the thread model used. If the latest recommendation is used, the consumer would be moved to a consumer thread and the slot would naturally be executed within the event queue processed by the consumer thread. There are some nice examples around. The problem, however, is that the consumer has no visibility on how many slots are queuing up in the consumer thread's event queue. As long as the average execution time of the consumer is less than the producer interval, the consumer would catch up at some point and empty the event queue.
What to do if one is only interested in processing the latest data while still having some visibility of the lost data, at least by name or a property of the data to be processed that can be logged. There is of course event compression (http://stackoverflow.com/questions/20866996/how-to-compress-slot-calls-when-using-queued-connection-in-qt) but I find it a bit intrusive as it needs private Qt includes.
I wonder whether I should return to the pre 4.7 recommended version, override the virtual run() method, use a while(running) loop blocking on a semaphore. Data would be posted to the thread (owned by the main application thread) into a slot where, protected by a mutex, the data is enqueued. Releasing the semaphore in that slot would wake up the thread and inside the consumer thread's loop, a dequeue would get the item to be processed. While the consumer is working on the item, the producer would still be able to queue data items into the queue. Whenever the consumer has finished processing the slow item, it could always inspect the queue, discard all but the latest and process the latest.
My questions now are: has anybody had some experience in this kind of pattern? What is the appropriate type to block on (semaphore, wait condition)? Where do I need to protect the queue in order not to create deadlocks, and also a situation where the consumer is not woken up immediately after a new data item is available? Also, I would like to avoid many copy operations (only one copy for const& with signal & slot with workers moved to thread, http://www.embeddeduse.com/2013/06/29/copied-or-not-copied-arguments-signals-slots/). How can I avoid using a Qt::DirectConnection if the producer should inform & wakeup the consumer without going through the main application event queue?
In the past I have tried many variations of recommended solutions that come up on the first page on google, but I haven't found and elegant and concise Qt way of addressing the problem.
I am looking forward to hearing some suggestions,