Waterfall Spectrogram UI for Massive Amounts of Data

Yippiyak

Hello,
I am relatively new to Qt, but my first assignment is designing a UI for a Spectrogram. The UI receives a test file (eventually would receive live RF Signal data), that transmits approximately 4 gigabytes of data per second. The project is about 50 or so files and around 10,000 lines of code, so there won't be an easy fix. Like I said, I am relatively new to Qt, so I may not even be able to understand any answers you give me. That said, heres what I need help with;

The processing speed of the waterfall graph is 1/20th of realtime, and this is the biggest issue. Not only does the data pour in slowly, but the UI tracks this data slowly as well. A Qt rep indicated that perhaps Qt 3d could render the data without issue, but as of right now I am not entirely sure how to use Qt 3d as anything other than a fancy version of solid works.
Any time I resize the window all the Data in the waterfall chart resets, but the data itself does not loop (nor should it), so essentially all data is lost.
Zoom crashes the UI. (Though this issue is likely a bug)

So, I am here for any advice you can give me, even if it is as simple as "try googling this," because I currently have a working UI in C++ that is far too slow to be useful. The design challenge is that it has to be in Qt, and the quantity of data must remain at around this size. I have read that I may potentially be able to use CUDA in some manner to offload to the GPU, but I am not sure how feasible this is in Qt. Thank you for any help you can give me,

Y

Edit: My manager recommended trying OpenGL, and perhaps Qt DataVisualization, but that I should avoid qml. Anyone have experience with Qt and OpenGL for rendering.

aha_1980

@Yippiyak

4 gigabyte per second

how can you get data in your PC at this speed?

anyhow, the first thing you need to do is to reduce the amount of data by appropriate filtering.

the data is lost anyway later as your eyes cannot read it from the screen at this rate.

and yes, hardware accelleration might speed up thimgs massively.

J.Hilk

@aha_1980 said in Waterfall Spectrogram UI for Massive Amounts of Data:

how can you get data in your PC at this speed?

there are a couple of ways, I think.

OC-3840 for example is a network line with transmission speeds of up to 200 Gbit/s

or

InfiniBand allows for transmission speeds of up to 600 Gbit/s (75 GB/s)

jsulm

@aha_1980 My understanding is that the data is actually coming from a file:
"The UI receives a test file (eventually would receive live RF Signal data), that transmits approximately 4 gigabytes of data per second."
It is an unclear description. I think 4Gbps is the sample rate or something.

kshegunov

Check out this thread.

@J.Hilk said in Waterfall Spectrogram UI for Massive Amounts of Data:

InfiniBand allows for transmission speeds of up to 600 Gbit/s (75 GB/s)

Do you know the price of an inifiniband interface and the switches, and the infrastructure? It's a pretty neat package for home ... if you can afford a third world country on your own ... ;)

Yippiyak

I suppose its worth making a little clearer, this is not a home project on a simple PC. The hardware is a couple hundred grand haha. Can't specify too much, my task is simply to figure out how to optimize the Qt software to run the UI smoothly. The data itself is a rather fixed input from the array. Its less about reading all of the data at that rate, but simply tracking it smoothly. The code runs well in C#, but Im having difficulty adapting the project to C++ in Qt.

mrjj

@Yippiyak
Hi I would add some timing function to try to find out where the time is spend.
Do you use threads?

Yippiyak

@mrjj I do use some threading. And its less a matter of data processing so much as rendering, but I prefer not to rely on a client having a GPU. Just spoke with my project manager, and he said try avoiding qml, but that OpenGL would be helpful. Anyone have experience with this?
https://doc.qt.io/qt-5.11/qtdatavisualization-index.html

jsulm

@Yippiyak "but I prefer not to rely on a client having a GPU" - a client for data visualisation without a GPU? What would that be? Even smartphones have a GPU.

J.Hilk

@kshegunov said in Waterfall Spectrogram UI for Massive Amounts of Data:

Check out this thread.

@J.Hilk said in Waterfall Spectrogram UI for Massive Amounts of Data:

InfiniBand allows for transmission speeds of up to 600 Gbit/s (75 GB/s)

Do you know the price of an inifiniband interface and the switches, and the infrastructure? It's a pretty neat package for home ... if you can afford a third world country on your own ... ;)

😎 Don't we all want Jeff Bezos income?

@Yippiyak said in Waterfall Spectrogram UI for Massive Amounts of Data:

@mrjj I do use some threading. And its less a matter of data processing so much as rendering, but I prefer not to rely on a client having a GPU. Just spoke with my project manager, and he said try avoiding qml, but that OpenGL would be helpful. Anyone have experience with this?
https://doc.qt.io/qt-5.11/qtdatavisualization-index.html

If it's such a "high profile" case, where cost is a secundary concern, I would try reaching out to the Qt-Company for professional consulting!

Should get you further and quicker than a user forum 😉