Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Get Qt Extensions
  • Unsolved
Collapse
Brand Logo
  1. Home
  2. Qt Development
  3. General and Desktop
  4. Waterfall Spectrogram UI for Massive Amounts of Data

Waterfall Spectrogram UI for Massive Amounts of Data

Scheduled Pinned Locked Moved Solved General and Desktop
10 Posts 6 Posters 3.3k Views
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • Y Offline
    Y Offline
    Yippiyak
    wrote on 27 Jun 2018, 01:17 last edited by Yippiyak
    #1

    Hello,
    I am relatively new to Qt, but my first assignment is designing a UI for a Spectrogram. The UI receives a test file (eventually would receive live RF Signal data), that transmits approximately 4 gigabytes of data per second. The project is about 50 or so files and around 10,000 lines of code, so there won't be an easy fix. Like I said, I am relatively new to Qt, so I may not even be able to understand any answers you give me. That said, heres what I need help with;

    1. The processing speed of the waterfall graph is 1/20th of realtime, and this is the biggest issue. Not only does the data pour in slowly, but the UI tracks this data slowly as well. A Qt rep indicated that perhaps Qt 3d could render the data without issue, but as of right now I am not entirely sure how to use Qt 3d as anything other than a fancy version of solid works.
    2. Any time I resize the window all the Data in the waterfall chart resets, but the data itself does not loop (nor should it), so essentially all data is lost.
    3. Zoom crashes the UI. (Though this issue is likely a bug)

    So, I am here for any advice you can give me, even if it is as simple as "try googling this," because I currently have a working UI in C++ that is far too slow to be useful. The design challenge is that it has to be in Qt, and the quantity of data must remain at around this size. I have read that I may potentially be able to use CUDA in some manner to offload to the GPU, but I am not sure how feasible this is in Qt. Thank you for any help you can give me,

    Y

    Edit: My manager recommended trying OpenGL, and perhaps Qt DataVisualization, but that I should avoid qml. Anyone have experience with Qt and OpenGL for rendering.

    A K 2 Replies Last reply 27 Jun 2018, 05:52
    0
    • Y Yippiyak
      27 Jun 2018, 01:17

      Hello,
      I am relatively new to Qt, but my first assignment is designing a UI for a Spectrogram. The UI receives a test file (eventually would receive live RF Signal data), that transmits approximately 4 gigabytes of data per second. The project is about 50 or so files and around 10,000 lines of code, so there won't be an easy fix. Like I said, I am relatively new to Qt, so I may not even be able to understand any answers you give me. That said, heres what I need help with;

      1. The processing speed of the waterfall graph is 1/20th of realtime, and this is the biggest issue. Not only does the data pour in slowly, but the UI tracks this data slowly as well. A Qt rep indicated that perhaps Qt 3d could render the data without issue, but as of right now I am not entirely sure how to use Qt 3d as anything other than a fancy version of solid works.
      2. Any time I resize the window all the Data in the waterfall chart resets, but the data itself does not loop (nor should it), so essentially all data is lost.
      3. Zoom crashes the UI. (Though this issue is likely a bug)

      So, I am here for any advice you can give me, even if it is as simple as "try googling this," because I currently have a working UI in C++ that is far too slow to be useful. The design challenge is that it has to be in Qt, and the quantity of data must remain at around this size. I have read that I may potentially be able to use CUDA in some manner to offload to the GPU, but I am not sure how feasible this is in Qt. Thank you for any help you can give me,

      Y

      Edit: My manager recommended trying OpenGL, and perhaps Qt DataVisualization, but that I should avoid qml. Anyone have experience with Qt and OpenGL for rendering.

      A Offline
      A Offline
      aha_1980
      Lifetime Qt Champion
      wrote on 27 Jun 2018, 05:52 last edited by
      #2

      @Yippiyak

      4 gigabyte per second

      how can you get data in your PC at this speed?

      anyhow, the first thing you need to do is to reduce the amount of data by appropriate filtering.

      the data is lost anyway later as your eyes cannot read it from the screen at this rate.

      and yes, hardware accelleration might speed up thimgs massively.

      Qt has to stay free or it will die.

      J J 2 Replies Last reply 27 Jun 2018, 06:07
      2
      • A aha_1980
        27 Jun 2018, 05:52

        @Yippiyak

        4 gigabyte per second

        how can you get data in your PC at this speed?

        anyhow, the first thing you need to do is to reduce the amount of data by appropriate filtering.

        the data is lost anyway later as your eyes cannot read it from the screen at this rate.

        and yes, hardware accelleration might speed up thimgs massively.

        J Offline
        J Offline
        J.Hilk
        Moderators
        wrote on 27 Jun 2018, 06:07 last edited by
        #3

        @aha_1980 said in Waterfall Spectrogram UI for Massive Amounts of Data:

        how can you get data in your PC at this speed?

        there are a couple of ways, I think.

        OC-3840 for example is a network line with transmission speeds of up to 200 Gbit/s

        or

        InfiniBand allows for transmission speeds of up to 600 Gbit/s (75 GB/s)


        Be aware of the Qt Code of Conduct, when posting : https://forum.qt.io/topic/113070/qt-code-of-conduct


        Q: What's that?
        A: It's blue light.
        Q: What does it do?
        A: It turns blue.

        1 Reply Last reply
        3
        • A aha_1980
          27 Jun 2018, 05:52

          @Yippiyak

          4 gigabyte per second

          how can you get data in your PC at this speed?

          anyhow, the first thing you need to do is to reduce the amount of data by appropriate filtering.

          the data is lost anyway later as your eyes cannot read it from the screen at this rate.

          and yes, hardware accelleration might speed up thimgs massively.

          J Offline
          J Offline
          jsulm
          Lifetime Qt Champion
          wrote on 27 Jun 2018, 06:20 last edited by
          #4

          @aha_1980 My understanding is that the data is actually coming from a file:
          "The UI receives a test file (eventually would receive live RF Signal data), that transmits approximately 4 gigabytes of data per second."
          It is an unclear description. I think 4Gbps is the sample rate or something.

          https://forum.qt.io/topic/113070/qt-code-of-conduct

          1 Reply Last reply
          2
          • Y Yippiyak
            27 Jun 2018, 01:17

            Hello,
            I am relatively new to Qt, but my first assignment is designing a UI for a Spectrogram. The UI receives a test file (eventually would receive live RF Signal data), that transmits approximately 4 gigabytes of data per second. The project is about 50 or so files and around 10,000 lines of code, so there won't be an easy fix. Like I said, I am relatively new to Qt, so I may not even be able to understand any answers you give me. That said, heres what I need help with;

            1. The processing speed of the waterfall graph is 1/20th of realtime, and this is the biggest issue. Not only does the data pour in slowly, but the UI tracks this data slowly as well. A Qt rep indicated that perhaps Qt 3d could render the data without issue, but as of right now I am not entirely sure how to use Qt 3d as anything other than a fancy version of solid works.
            2. Any time I resize the window all the Data in the waterfall chart resets, but the data itself does not loop (nor should it), so essentially all data is lost.
            3. Zoom crashes the UI. (Though this issue is likely a bug)

            So, I am here for any advice you can give me, even if it is as simple as "try googling this," because I currently have a working UI in C++ that is far too slow to be useful. The design challenge is that it has to be in Qt, and the quantity of data must remain at around this size. I have read that I may potentially be able to use CUDA in some manner to offload to the GPU, but I am not sure how feasible this is in Qt. Thank you for any help you can give me,

            Y

            Edit: My manager recommended trying OpenGL, and perhaps Qt DataVisualization, but that I should avoid qml. Anyone have experience with Qt and OpenGL for rendering.

            K Offline
            K Offline
            kshegunov
            Moderators
            wrote on 27 Jun 2018, 14:49 last edited by
            #5

            Check out this thread.

            @J.Hilk said in Waterfall Spectrogram UI for Massive Amounts of Data:

            InfiniBand allows for transmission speeds of up to 600 Gbit/s (75 GB/s)

            Do you know the price of an inifiniband interface and the switches, and the infrastructure? It's a pretty neat package for home ... if you can afford a third world country on your own ... ;)

            Read and abide by the Qt Code of Conduct

            J 1 Reply Last reply 28 Jun 2018, 04:33
            3
            • Y Offline
              Y Offline
              Yippiyak
              wrote on 27 Jun 2018, 16:14 last edited by
              #6

              I suppose its worth making a little clearer, this is not a home project on a simple PC. The hardware is a couple hundred grand haha. Can't specify too much, my task is simply to figure out how to optimize the Qt software to run the UI smoothly. The data itself is a rather fixed input from the array. Its less about reading all of the data at that rate, but simply tracking it smoothly. The code runs well in C#, but Im having difficulty adapting the project to C++ in Qt.

              mrjjM 1 Reply Last reply 27 Jun 2018, 16:21
              0
              • Y Yippiyak
                27 Jun 2018, 16:14

                I suppose its worth making a little clearer, this is not a home project on a simple PC. The hardware is a couple hundred grand haha. Can't specify too much, my task is simply to figure out how to optimize the Qt software to run the UI smoothly. The data itself is a rather fixed input from the array. Its less about reading all of the data at that rate, but simply tracking it smoothly. The code runs well in C#, but Im having difficulty adapting the project to C++ in Qt.

                mrjjM Offline
                mrjjM Offline
                mrjj
                Lifetime Qt Champion
                wrote on 27 Jun 2018, 16:21 last edited by
                #7

                @Yippiyak
                Hi I would add some timing function to try to find out where the time is spend.
                Do you use threads?

                Y 1 Reply Last reply 27 Jun 2018, 16:33
                0
                • mrjjM mrjj
                  27 Jun 2018, 16:21

                  @Yippiyak
                  Hi I would add some timing function to try to find out where the time is spend.
                  Do you use threads?

                  Y Offline
                  Y Offline
                  Yippiyak
                  wrote on 27 Jun 2018, 16:33 last edited by Yippiyak
                  #8

                  @mrjj I do use some threading. And its less a matter of data processing so much as rendering, but I prefer not to rely on a client having a GPU. Just spoke with my project manager, and he said try avoiding qml, but that OpenGL would be helpful. Anyone have experience with this?
                  https://doc.qt.io/qt-5.11/qtdatavisualization-index.html

                  J 1 Reply Last reply 28 Jun 2018, 04:12
                  0
                  • Y Yippiyak
                    27 Jun 2018, 16:33

                    @mrjj I do use some threading. And its less a matter of data processing so much as rendering, but I prefer not to rely on a client having a GPU. Just spoke with my project manager, and he said try avoiding qml, but that OpenGL would be helpful. Anyone have experience with this?
                    https://doc.qt.io/qt-5.11/qtdatavisualization-index.html

                    J Offline
                    J Offline
                    jsulm
                    Lifetime Qt Champion
                    wrote on 28 Jun 2018, 04:12 last edited by
                    #9

                    @Yippiyak "but I prefer not to rely on a client having a GPU" - a client for data visualisation without a GPU? What would that be? Even smartphones have a GPU.

                    https://forum.qt.io/topic/113070/qt-code-of-conduct

                    1 Reply Last reply
                    1
                    • K kshegunov
                      27 Jun 2018, 14:49

                      Check out this thread.

                      @J.Hilk said in Waterfall Spectrogram UI for Massive Amounts of Data:

                      InfiniBand allows for transmission speeds of up to 600 Gbit/s (75 GB/s)

                      Do you know the price of an inifiniband interface and the switches, and the infrastructure? It's a pretty neat package for home ... if you can afford a third world country on your own ... ;)

                      J Offline
                      J Offline
                      J.Hilk
                      Moderators
                      wrote on 28 Jun 2018, 04:33 last edited by
                      #10

                      @kshegunov said in Waterfall Spectrogram UI for Massive Amounts of Data:

                      Check out this thread.

                      @J.Hilk said in Waterfall Spectrogram UI for Massive Amounts of Data:

                      InfiniBand allows for transmission speeds of up to 600 Gbit/s (75 GB/s)

                      Do you know the price of an inifiniband interface and the switches, and the infrastructure? It's a pretty neat package for home ... if you can afford a third world country on your own ... ;)

                      😎 Don't we all want Jeff Bezos income?

                      @Yippiyak said in Waterfall Spectrogram UI for Massive Amounts of Data:

                      @mrjj I do use some threading. And its less a matter of data processing so much as rendering, but I prefer not to rely on a client having a GPU. Just spoke with my project manager, and he said try avoiding qml, but that OpenGL would be helpful. Anyone have experience with this?
                      https://doc.qt.io/qt-5.11/qtdatavisualization-index.html

                      If it's such a "high profile" case, where cost is a secundary concern, I would try reaching out to the Qt-Company for professional consulting!

                      Should get you further and quicker than a user forum 😉


                      Be aware of the Qt Code of Conduct, when posting : https://forum.qt.io/topic/113070/qt-code-of-conduct


                      Q: What's that?
                      A: It's blue light.
                      Q: What does it do?
                      A: It turns blue.

                      1 Reply Last reply
                      1

                      1/10

                      27 Jun 2018, 01:17

                      • Login

                      • Login or register to search.
                      1 out of 10
                      • First post
                        1/10
                        Last post
                      0
                      • Categories
                      • Recent
                      • Tags
                      • Popular
                      • Users
                      • Groups
                      • Search
                      • Get Qt Extensions
                      • Unsolved