writeTextElement() of QXmlStreamWriter crashed when writting the content of a big file

MNGL

Hi, Guys

Here is the qt code that crash:

Writing the content of a 130MB binary file will crash, but writing a 2MB binary file will not.

Is there any solution for writeTextElement to write big file content? Thanks

inStream.writeTextElement("Content", file->property("Content"));  
//QXmlStreamWriter inStream 
//file->property("Content") == mContent.toBase64()
//mContent is an object of QByteArray


QFile file(fileName);
  
file.write(mContent);
file.close();

QString DFile::property(const QString &inPropertyName) const
{ 
   if(inPropertyName == "Content")
     return mContent.toBase64();

   return QString();
}

alt text

Christian Ehrlicher

Plkease provide a minimal, compilable example. QXmlStreamWriter does not buffer anything so I doubt it has something to do with Qt but a problem on your side.

MNGL

@Christian-Ehrlicher

env: qt opensource 4.8.7, Visual Studio 2010, Windows 10

big file: https://1drv.ms/u/s!Ap_EAuwC9QkXijOmXqxp9DEav3gm?e=iha0uI

project: https://1drv.ms/u/s!Ap_EAuwC9QkXijWzOlpaWmtzOdtz?e=fDpo93

#include <QtCore/QCoreApplication>
#include <QFile>
#include <QXmlStreamWriter>


int main(int argc, char *argv[])
{
    QCoreApplication a(argc, argv);

    QByteArray mContentBuffer;

    QFile file("C:\\Work\\bigfile.xar");
   
    if(!file.open(QFile::ReadOnly))
    {
        return -1;
    }

    mContentBuffer = file.readAll();
    file.close();

    QFile profile("C:\\Work\\profile.xml");

    if(!profile.open(QFile::WriteOnly|QFile::Truncate))
    {
        return -1;
    }

    QXmlStreamWriter stream(&profile);    

    stream.setAutoFormatting(true);
    stream.writeStartDocument();
    stream.writeStartElement("Profile");

    stream.writeTextElement("Content", mContentBuffer.toBase64());

    stream.writeEndElement(); // Profile
    stream.writeEndDocument();

    return a.exec();
}

Christian Ehrlicher

This is an out-of-memory problem on your side. You load 300MB into memory and then create a base64 QByteArray which is 300/3*4MB = 400MB. Since it's a QString you have 800MB. Internally the data must be escaped which creates another >800MB QString. This needs to get converted to a QByteArray internally to be able to write it out to a QIODevice. Your app is 32bit so you're running out-of-memory.
The only way I see is to simulate writeTextElement by calling the threee functions by yourself:

void QXmlStreamWriter::writeTextElement(const QString &namespaceUri, const QString &name, const QString &text)
{
    writeStartElement(namespaceUri, name);
    writeCharacters(text);
    writeEndElement();
}

Then you can call writeChracters with a small buffer size in a loop.

MNGL

@Christian-Ehrlicher Thanks.

One further problem, I used writeCharacters() and with a small buffer size, but it seems that it only works when writing the file content to a xml, writeCharacters() will crash when wrtting the content of file (300Mb) to a QBuffer. Any soultion for this?

you can reproduce this crash using following code example.
big file: https://1drv.ms/u/s!Ap_EAuwC9QkXijbujBQcQk4Hat_O?e=KemgUY

#include <QtCore/QCoreApplication>
#include <QFile>
#include <QXmlStreamWriter>
#include <QBuffer>

int main(int argc, char *argv[])
{
   
    QCoreApplication a(argc, argv);

    QByteArray mContentBuffer;
    QByteArray arr;
    int pos, filesize;

    QFile file("C:\\Work\\bigfile.xar"); //Szie: 300Mb

    if(!file.open(QFile::ReadOnly))
    {
        return -1;
    }

    mContentBuffer = file.readAll();
    file.close();

    //QFile profile("C:\\Work\\profile.xml");

    //if(!profile.open(QFile::WriteOnly|QFile::Truncate))
    //{
    //    return -1;
    //}
    QBuffer buffer;
    buffer.open(QBuffer::ReadWrite);

    QXmlStreamWriter stream(&buffer);    

    stream.setAutoFormatting(true);
    stream.writeStartDocument();
    stream.writeStartElement("Profile");

    //stream.writeTextElement("Content", mContentBuffer.toBase64());
    stream.writeStartElement("Content");
    pos = 0; 
    filesize = mContentBuffer.size();		  
    while(pos<filesize){
       arr = mContentBuffer.mid(pos, 2000000);
       stream.writeCharacters(arr.toBase64());
       pos+=arr.size();
    }
    stream.writeEndElement();
    stream.writeEndElement(); // Profile
    stream.writeEndDocument();

    return 0;
}

Christian Ehrlicher

@MNGL said in writeTextElement() of QXmlStreamWriter crashed when writting the content of a big file:

riteCharacters() will crash when wrtting the content of file (300Mb) to a QBuffer. Any soultion for this?

What do you expect otherwise - I already explained you why it happens...

MNGL

@Christian-Ehrlicher thanks.

I turned on the option /LARGEADDRESSAWARE, that crash is gone.

Here is the goal for writing a big file into a QBuffer.
Do you have alternative way to compare two xml files ?

bool profileModified()
{
    bool result = true;

    QFile profile("C:\\Work\\profile.xml");
    if(profile.exists())
    {
        QBuffer buffer;
        buffer.open(QBuffer::ReadWrite);
        QXmlStreamWriter stream(&buffer);
        
        exportProfile(stream);

        profile.open(QFile::ReadOnly);
        QByteArray profileArr = profile.readAll();
    
        buffer.seek(0);
      
        QByteArray bufferArr = buffer.buffer();
         
        result = (array_compare(profileArr, bufferArr) != 0);
        
        profile.close();
    }

    return result;
}

JonB

@MNGL said in writeTextElement() of QXmlStreamWriter crashed when writting the content of a big file:

Do you have alternative way to compare two xml files ?

What does array_compare(profileArr, bufferArr) do? If it does anything like a byte-by-byte compare (as opposed to some complex context-sensitive fuzzy comparison), why in the world does this require holding two large file contents in memory? Go for same sort of approach as @Christian-Ehrlicher said earlier

Then you can call writeChracters with a small buffer size in a loop.

Here you should surely open both files but not read them all into memory, rather pass open QFIle/QTextStream handles to array_compare() which reads chunk-by-chunk in a loop comparing as it goes. Also has the advantage that if they are going to compare different it will determine this as soon as they differ without having to read all of both files.

MNGL

@JonB Thanks
Below is code of array_compare(), exportProfile(stream) is the function to writting xml content to a file or buffer,how should I control it to write a chunk of the xml content to a buffer or ByteArray?

int pe_strcmp(const char *str1, const char *str2)
{
    return (str1 && str2) ? strcmp(str1, str2) : (str1 ? 1 : (str2 ? -1 : 0));
}

int array_compare(const QByteArray &a, const QByteArray &b) const
{
    return pe_strcmp(a, b);
}

jsulm

@MNGL said in writeTextElement() of QXmlStreamWriter crashed when writting the content of a big file:

how should I control it to write a chunk of the xml content to a buffer or

By reading the content of the files in small chunks as already explained and comparing these chunks.