[SOLVED] Qt and CLucene

Hatchi

Dear all

I am not certain if this is a Qt problem but if anybody knows what the problem might be I would be very grateful. I have the following problem. When the application starts, the method loadIndex is called and this method tries to load an existing index. This method works fine when there is an index and when there is no index. However during the execution of the program, the method startIndexing can be called. When a new IndexWriter is created I get the following message:

@Exception at 0x75ccc41f, code: 0xe06d7363: C++ exception, flags=0x1 (execution cannot be continued) (first chance) at c:\users\clucene-core-2.3.3.4\clucene-core-2.3.3.4\src\core\clucene\store\directory.cpp:54@

However when I check the index directory, I see that the index is being generated by the method startIndexing. However when the execution flow reaches the statement

@IndexReader* r = IndexReader::open(indexDirectory.toStdString().c_str());@

the following message is shown:

@:0: error: Exception at 0x56e37bea, code: 0xc0000005: read access violation at: 0x0, flags=0x0 (first chance)@

I have no idea what the problem can be. I use Qt 5.1, Qt creator and MSVC 2010. I compiled the CLucene code as a DLL to which I link in the application.

@void IndexingEngine::loadIndex(Dataset *dataset, QString indexDirectory)
{
QTime timer;
timer.start();

try{
    if ( IndexReader::indexExists(indexDirectory.toStdString().c_str()) ){
        if ( IndexReader::isLocked(indexDirectory.toStdString().c_str()) ){
            printf("Index was locked... unlocking it.\n");
            IndexReader::unlock(indexDirectory.toStdString().c_str());
        }
    }
    else
        return;

    dataset->setIndexDirectory(indexDirectory);
    QStringList *termsInIndex = new QStringList();

    IndexReader* r = IndexReader::open(indexDirectory.toStdString().c_str());

    TermEnum* te = r->terms();
    int32_t nterms;
    for (nterms = 0; te->next() == true; nterms++) {
        termsInIndex->append(QString::fromStdWString(te->term()->text()));
    }

    _CLLDELETE(te);

    r->close();
    _CLLDELETE(r);

}catch(CLuceneError& err){
    printf("Error: %s\n", err.what());
}catch(...){
    printf("Unknown error\n");
}

}

void IndexingEngine::startIndexingTXT(Dataset dataset, bool clearIndex, QString indexDirectory)
{
int i=0;
QTime timer;
timer.start();
const char target = indexDirectory.toStdString().c_str();

IndexWriter* writer = NULL;
lucene::analysis::WhitespaceAnalyzer an;

try{
    if ( !clearIndex && IndexReader::indexExists(target) ){
        if ( IndexReader::isLocked(target) ){
            printf("Index was locked... unlocking it.\n");
            IndexReader::unlock(target);
        }

        writer = _CLNEW IndexWriter( target, &an, false);
    }else{
        writer = _CLNEW IndexWriter( indexDirectory.toStdString().c_str() ,&an, true);
    }

    writer->setMaxFieldLength(0x7FFFFFFFL); // LUCENE_INT32_MAX_SHOULDBE

    // Turn this off to make indexing faster; we'll turn it on later before optimizing
    writer->setUseCompoundFile&#40;false&#41;;

    {
        QDirIterator it2 (dataset->getDatasetDirectory(&#41;, QDir::NoDotAndDotDot | QDir::Files | QDir::Dirs);
        while (it2.hasNext())
        {
            it2.next();
            QFile file;
            file.setFileName(it2.filePath());
            file.open(QIODevice::ReadOnly | QIODevice::Text);

            QString filename = it2.fileName();
            QString contents = file.readAll();

            // Re-use the document object
            Document doc;


            doc.clear();
            doc.add( *_CLNEW Field(_T("filename"), filename.toStdWString().c_str(), Field::STORE_YES | Field::INDEX_UNTOKENIZED ) );
            doc.add( *_CLNEW Field(_T("contents"), contents.toStdWString().c_str(), Field::STORE_YES | Field::INDEX_TOKENIZED) );

            writer->addDocument( &doc );
            i++;

            file.close();
        }
    }

    // Make the index use as little files as possible, and optimize it
    writer->setUseCompoundFile&#40;true&#41;;
    writer->optimize(&#41;;

    // Close and clean up
    writer->close();
    _CLLDELETE(writer);

    dataset->setIndexDirectory(indexDirectory);
    QStringList *termsInIndex = new QStringList();

    IndexReader* r = IndexReader::open(indexDirectory.toStdString().c_str());

    TermEnum* te = r->terms();
    int32_t nterms;
    for (nterms = 0; te->next() == true; nterms++) {
        termsInIndex->append(QString::fromStdWString(te->term()->text()));
    }

    _CLLDELETE(te);

    r->close();
    _CLLDELETE(r);

    int elapsedTime = timer.elapsed();
    statusNotification->setText(QString::number(i) + " reports indexed, " + QString::number(nterms) +
                                " unique terms, " +QTime(0,0,0,0).addMSecs(elapsedTime).toString("hh:mm:ss.zzz")+" time elapsed");
    progress->setValue(100);

}catch(CLuceneError& err){
    printf("Error: %s\n", err.what());
}catch(...){
    printf("Unknown error\n");
}
getStats(indexDirectory.toStdString().c_str());

_lucene_shutdown(); //clears all static memory

}
@

SGaist

Hi,

Did you try to run it with the debugger ? That should give you more information

Hatchi

Dear Sir

All information which I obtained with the debugger is displayed in the post ...
Unless you have any idea for obtaining additional information ..?

Kind regards
Hatchi

SGaist

The only information ? Did you run it with a debug build ?

Hatchi

"http://www.jonaspoelmans.com/error/Error.png":http://www.jonaspoelmans.com/error/Error.png

"http://www.jonaspoelmans.com/error/Error2.png":http://www.jonaspoelmans.com/error/Error2.png

This is all information I have ...

Hatchi

SGaist

Can you provide the backtrace ?

SGaist

Just a quick question, since IndexReader::open takes a const char * (and probably the other functions too) why not use something like:

@QByteArray asciiPath = indexDirectory.toAscii(); // also used in toStdString();
const char * path = asciiPath.constData(); << this will ensure that path will point to valid buffer.
@
?

Hatchi

Dear Sir

I tried your latest suggestion but I still received the same errors. I am pretty sure something is going wrong inside the CLucene library itself, however when I tried to put breakpoints in the CLucene source code which is in the include path of my application, the debugger doesn't go to here. I compiled CLucene from source and linked in debug modus with the debug version of the DLL. Do you have any suggestion for obtaining the call trace inside this DLL?

Kind regards
Hatchi

SGaist

You could already try to get a backtrace from the debug console in QtCreator.

I am however surprised that you can't get in the library, are you sure that your program links agains the debug version of CLucene ?

Hatchi

Dear Sir

This is the backtrace I got from Qt creator:

0 QAbstractScrollArea::setHorizontalScrollBarPolicy Qt5Widgetsd 0x5ebe7bea
1 QAbstractScrollArea::setHorizontalScrollBarPolicy Qt5Widgetsd 0x5ed03f4b
2 QAbstractScrollArea::setHorizontalScrollBarPolicy Qt5Widgetsd 0x5ecf86c3
3 QAbstractScrollArea::setHorizontalScrollBarPolicy Qt5Widgetsd 0x5edf4eb1
4 IndexingEngine::startIndexingTXT indexingengine.cpp 207 0xb6a2e4
5 TextBrowser::indexDocuments textbrowser.cpp 248 0xb62b67
6 TextBrowser::qt_static_metacall moc_textbrowser.cpp 140 0xb90424
7 QObject::destroyed Qt5Cored 0x5df9b210
8 QObject::destroyed Qt5Cored 0x5df9ac41
9 QAbstractScrollArea::setHorizontalScrollBarPolicy Qt5Widgetsd 0x5ebd862b
10 QAbstractScrollArea::setHorizontalScrollBarPolicy Qt5Widgetsd 0x5ebd7d07
11 QAbstractScrollArea::setHorizontalScrollBarPolicy Qt5Widgetsd 0x5ebcf411
12 QAbstractScrollArea::setHorizontalScrollBarPolicy Qt5Widgetsd 0x5ee35a18
13 QAbstractScrollArea::setHorizontalScrollBarPolicy Qt5Widgetsd 0x5ed37dc2
14 QAbstractScrollArea::setHorizontalScrollBarPolicy Qt5Widgetsd 0x5ed38e82
15 QAbstractScrollArea::setHorizontalScrollBarPolicy Qt5Widgetsd 0x5ee34ab0
16 QAbstractScrollArea::setHorizontalScrollBarPolicy Qt5Widgetsd 0x5ec39db5
17 QAbstractScrollArea::setHorizontalScrollBarPolicy Qt5Widgetsd 0x5ed38ca1
18 QAbstractScrollArea::setHorizontalScrollBarPolicy Qt5Widgetsd 0x5ee35a7b
19 QAbstractScrollArea::setHorizontalScrollBarPolicy Qt5Widgetsd 0x5ebf233e
... <More>

I could get info from the library in the stack trace but as soon as execution reaches the point where it always crashes, this is the only information ...

Would it be useful if I would send you the dll and some accompanying code to try and test it yourself?

Hatchi

SGaist

One other thing you might want to try first is to build the clucene code along your application (you can have a look at fulltextsearch.pri to see what files to build.

Hatchi

Thank you for this suggestion. This solved the problem. I see that Qt uses an older version of CLucene while I was using the newest unstable version.

Thanks!

SGaist

You're welcome !

Don't forget to update the thread title to solved so other forum users will know that a solution have been found :)