How to synchronize QNetworkAccessManager ?



  • Me dropped [url=http://developer.qt.nokia.com/forums/viewthread/5196/]attempts assemble Qt statically with WebKit[/url]. I turned to use QNetworkAccessManager instead. But got another problem.

    Task: application loads simple text list of image names from some Web-server. Then it has to show and process each image one by one. Quantity of images can be large - tens of thousands, may be even hundreds of thousands. I start this process like this:

    @void MainWindow::paintEvent( QPaintEvent * event )
    {
    if( firstpaint )
    {
    listmanager->get( QNetworkRequest( QUrl( "http://........" ) ) );
    firstpaint = false;
    }
    }@

    sorry, but did not find better place.

    Then after list downloading finishes, Qt calls this method:

    @void MainWindow::finishedList( QNetworkReply * reply )
    {
    if( reply->error() == QNetworkReply::NoError )
    {
    forever
    {
    QByteArray byteline = reply->readLine();
    if( byteline.size() == 0 )
    break;
    urllist << byteline; // urllist is a QStringList
    }
    foreach( QString url, urllist )
    imgmanager->get( QNetworkRequest( url ) ) );
    }
    else
    networkerror = true;
    reply->deleteLater();
    }@

    I made separate reading cycle for future purposes. And there is of course method for image processing. It has to be called after each image downloaded:

    @void MainWindow::finishedImg( QNetworkReply * reply )
    {
    if( reply->error() == QNetworkReply::NoError )
    {
    // show and process but doesn't matter how..............
    }
    else
    networkerror = true;
    reply->deleteLater();
    }@

    If I start this with one image - then I see it in MainWindow and processing works. But if list contains a number of images, then call imgmanager->get( QNetworkRequest( url ) ) ); just pushes requests to network queue. First call of MainWindow::finishedImg( QNetworkReply * reply ) appears only after all image requests was sent to queue. They shoot like a Gatling machine gun. If there are lot of images then I get memory overflow.

    How to force QNetworkAccessManager process image downloading requests right now after get() and do not queue them? I try sync this using QSemaphore (acquire before get() then release in finishedImg()) but without success. Or may be paintEvent is not good place for this all? Which one is?



  • Do not start such things from a paintEvent(). You will start fresh requests every time your MainWindow is painted! That will very quickly cripple you system if you get a lot of paintEvents.

    QNAM has a limit on how many requests it deals with at once (I think it is 6 iirc). But first things first you need to find somewhere more appropriate to issue your network requests. How would you like to trigger the fetching of the images? Automatically at app start? In response to some user action?

    Edit: Ah forget the first part of my response. I didn't read your code properly - sorry. Still a better way might be to simply use your own slot that is called in response to a single shot timer.



  • One way to pace this a little would be to use the receipt of one image to trigger the download of the next one, maybe with a time delay in between.

    Or use the producer/consumer pattern where you maintain a buffer of images, a producer component featches the images and appends them to the buffer. Whenthe buffer is full the producer waits (or blocks). A consumer then removes them at a suitable rate and when it is empty the consumer waits (or blocks).



  • bq. How would you like to trigger the fetching of the images? Automatically at app start?[/quote]

    yes, automatically at app start but must show images - that means MainWindow should be properly initialized and able to support this

    bq. Still a better way might be to simply use your own slot that is called in response to a single shot timer.

    that's an idea... I can start timer in MainWindow::MainWindow()

    bq. One way to pace this a little would be to use the receipt of one image to trigger the download of the next one, maybe with a time delay in between.

    I can sync images using QSemaphore, but I need to finish each one before next without queuing

    bq. Or use the producer/consumer pattern where you maintain a buffer of images, a producer component featches the images and appends them to the buffer.

    this looks too complex and wasting resources



  • Why do you think that the producer/consumer pattern is too expensive and wasteful of resources? You yourself have already suggested using a QSemaphore which is used to synchronise access to data from multiple threads (unless you like blocking GUI's of course ;-)).

    Using the producer consumer pattern just requires 2 semaphores (or equivalently two wait conditions) as shown in the Qt examples.

    Another option that may be of interest to you would be to use a very simple state machine. You would only need a small number of states and transitions.



  • bq. Why do you think that the producer/consumer pattern is too expensive and wasteful of resources?

    I thought you suggest extra image buffering. This is not needed.

    Looks like problem appears not because of get() placed in paintEvent(). Images list I get() as well. But then I try get() images from list finishing slot. That can be a source of problem. One QNAM did not properly finish it's task but another one tries to start new task. I have try sync list processing with images.



  • nope... :-\ get() finishing slot is called somewhere after paintEvent() finished, probably this is because QApplication a.exec() starts later

    I'm weird... where better call QNAM instead of paintEvent()? I don't know something like "main thread" in QApplication or it's parent classes.



  • No, in fact your MainWindow::paintEvent() will not get called until the event loop is entered via your call to QApplication::exec(). I suggest that you go read some of the documentation on Qt basics, the event loop, asynchronous operation (for QNAM). I think this should be possible without threading but you may wish to have a quick look at QThread too - especially if the processing of each image is very CPU intensive. Out of interest what are you wanting to do to each image?

    To trigger the first fetch without using paintEvent() and to process it and request the next one just do something along these lines:

    @
    MainWindow::MainWindow( QWidget* parent )
    : QMainWindow( parent ),
    m_nam( new QNetworkAccessManager( this ) ),
    m_currentIndex( -1 ),
    m_urlList()
    {
    // Prepare url list
    m_urlList.append( "http://..." );
    ...

    // Kick things off
    QTimer::singlShot( 0, SLOT( requestNextImage() ) );
    

    }

    MainWindow::requestNextImage()
    {
    if ( ++m_currentIndex < m_urlList.size() ) {
    QNetworkReply* reply = m_nam->get( m_urlList.at( m_currentIndex ) )
    connect( reply, SIGNAL( finished() ), this, SLOT( processImage() ) );
    // Connect other signal as required
    } else {
    qDebug() << "All Done!";
    }
    }

    MainWindow::processImage()
    {
    QNetworkReply* reply = qobject_cast<QNetworkReply*>( sender() );
    if ( !reply )
    return;

    // Process the image as you need
    QImage myProcessedImage = ...
    
    // Display processed image
    ui->label->setPixmap( myProcessedImage );
    
    // Start fetch of next image in a little while
    QTimer::singleShot( delay, SLOT( requestNextImage() ) );
    

    }
    @



  • bq. No, in fact your MainWindow::paintEvent() will not get called until the event loop is entered via your call to QApplication::exec().

    I know this all and was greatly surprised with results.

    But task looks like SOLVED - not exactly the same way you propose but similar

    1st list must be downloaded from server, this starts with QTimer from MainWindow::Mainwindow()

    2nd after it is downloaded, in finishing slot I start another QTimer to start slot with get() of 1st picture from list

    finally, in the finishing slot for picture, after it's processing, I start another QTimer to start slot with get() of each next picture

    this works...

    Looks like problem was exactly because I tried get() next picture right from finishing slot of previous get(). This does not work. May be this should be explained in QNAM manual. If only it was, I'd save more than day.



  • It is not a problem with QNAM and so this does not belong in its documentation. The problem is that you misunderstood how QNAM and event-driven applications in general work. You need the event loop to be able to process events for things to be able to happen. In this case "things" means asynchronous network requests and painting of images, and yes even your initial call to MainWindow::paintEvent().

    Once you realise the importance of allowing the event loop to spin in between tasks it becomes really quite simple.



  • I know about event loop in Qt since 1997 and in GUI-systems since OS/2. I made some experiments - problem was exactly with paintEvent(). It did not finish before get() - therefore QNAM did not work as I expected. Of course I don't know how EXACTLY event loop works in Qt and how it relates to QNAM. But I read in manual - QNAM is asynchronous. For me that means it does not interfere to main event loop. But exactly it does. It cannot work properly if event processing was not finished. That MUST be signed in documentation.



  • OK. I wasn't trying to insult you. Apologies if I caused offence. I was trying to say that you had misunderstood how QNAM and sockets and QIODevices in general work in conjunction with the event loop in Qt.

    When the above classes are used asynchronously that means that they do require an event loop to be spinning in order for them to work. This line from the docs of QAbstractSocket helps:

    "Programming with a blocking socket is radically different from programming with a non-blocking socket. A blocking socket doesn't require an event loop and typically leads to simpler code. However, in a GUI application, blocking sockets should only be used in non-GUI threads, to avoid freezing the user interface. See the network/fortuneclient and network/blockingfortuneclient examples for an overview of both approaches."

    It is also mentioned in the docs of QIODevice (which is inherited by QNetworkReply):

    "Certain subclasses of QIODevice, such as QTcpSocket and QProcess, are asynchronous. This means that I/O functions such as write() or read() always return immediately, while communication with the device itself may happen when control goes back to the event loop."

    If synchronous operation is preferred then using a worker thread is recommended so that you do not block the event loop of the main thread which is needed to paint (amongst other things).

    If you still feel strongly that QNAM documentation is lacking then feel free to create a merge request that addresses this via Gitorious. Anyway, I'm glad you resolved the problem.


Log in to reply
 

Looks like your connection to Qt Forum was lost, please wait while we try to reconnect.