ASSERT on QString line 885 return false size by read line by line of a 125MB file.

patrik08

Hi is this my error... the function get the correct value..
but on big file test file i have a error from QString size...

this file 125mb have all Unicode data to test app...

<p/>ASSERT: "uint(i) < uint(size())" in file /Users/dev/Applications/qt5.9.1/lib/QtCore.framework/Headers/qstring.h, line 885 QString(int size, Qt::Initialization);
Abort trap: 6

//// count number in array mixed...
static inline int maxscript(int arr[], int stop, int num)
{
    int result = 0;
    for(int i=0; i<stop; i++)
    {
          if (arr[i] == num) {
            result++;
          }
    }
    return result;
}
//// statistic script() from QChar
int  hit_Script_found( QString text ) {
    const int nox = text.size();
    int *array = new int[nox];
    for (int o = 0; o < text.size(); o++) {
      const QChar vox(text.at(o));
      const int typsc = (int)vox.script();
      array[o]=typsc;
     }
    std::map <int,int, std::greater < int > > mhit;
    int NrScript =2; /// start from latin & go up..
    int hitmaxnumber =0;
    do {
          NrScript++;
              int hits = maxscript(array,nox,NrScript);
                if (hits > 0) {
                    hitmaxnumber = qMax(hitmaxnumber,hits);
                    mhit.insert ( std::pair<int,int>(hits,NrScript) );
                }
    }while(NrScript < (sumscritpsLang -2)); /// sum of avaiable script
    std::map<int,int>::iterator it = mhit.begin();
    for (it=mhit.begin(); it!=mhit.end(); ++it) {
        if (it->first == hitmaxnumber) {
            //// qDebug() << "hit is on by:" << it->second;
            return it->second;
          }
      }
   return -1;
   //// qDebug() << "found:" << hits << " by:" << NrScript;
}

patrik08

Super tanks... now 125mb file iterate line by line in 30sec...
only delete[] array; ... i try the vector int if is faster...

mrjj

@patrik08 said in ASSERT on QString line 885 return false size by read line by line of a 125MB file.:

ASSERT: "uint(i) < uint(size())

Yes its not very informative but it means that you ask it to go beyond its max index
like
QString g;
g[888888] = "";

Also just as a note
int *array = new int[nox];

Do you delete that again ?

patrik08

@mrjj no ... not delete the array at end of use... :-(
tanks...

mrjj

@patrik08

You can avoid it all together with

std::vector<int> array;
array.reserve(nox);

unless you have a really good reason to new it.

patrik08

Super tanks... now 125mb file iterate line by line in 30sec...
only delete[] array; ... i try the vector int if is faster...

mrjj

@patrik08
It wont be faster. Its just offer other benefits, like easy to send as parameter, it knows its
own size. Multiple way to loop over it. etc. On big lists you can use move feature to avoid copy
so its good to just use instead of the more traditional [] array. (IMO)

sierdzio

Have you considered reading the file line by line and not storing the whole 125MB in a string? Or using the stream approach to reading the file? You would save loads of RAM and CPU cycles this way.

patrik08

Hi ... oh yes i read line by line.. each line have other language to detect.. wich so much text in over 200 or more language...
google the file name test_shuffle_1000_48_666.utf8.gz
i tested for https://github.com/pehohlva/QCLD2 for language detection ..
the error was unclosed array cycle that fill ram... not QString... have incremented all line... :-) & must go in assert -