QCompleter - return result set, limited in size, for large term set for SPEED



  • Hi Everyone,

    I have a QCompleter on a QLineEdit, with a sorted list of 350,000 terms from a taxonomy. Since it's sorted and a binary search can be used for to retrieve results, this should not be a big issue.

    However, for the first one to two characters entered, there is a one to five second delay until the first results show up - I suppse that when I enter, say, "A", all results that start with A are considered and then the top N selected for display.

    So the question is, how can I tell the model to not worry about any but the top N results? Do I need to implement a custom model for the completer - and do you have any pointers for how to do that? (I have built custom models, not sure what's involved with a completer though).

    BTW I am using PySide 1.0.7 or later.

    Thanks!

    Pezzi



  • That delay could be mostly the initial load of the 350000 rows into RAM. For example, if the model is a QSqlQueryModel there may be a lengthy one-time delay as the data is loaded into memory 256 rows at a time using fetchMore(). If so, you could make the model persistent so it is a one-time-only delay.

    I have not found an elegant way to do a limited subset. I used a custom model and connected the line edit's textChanged() signal to a slot in the model feeding the completer. The model would reset and run a query to return the top 20 items. So, the completer was working with a model that essentially already contained the correct results.



  • Thank you for this!

    The data is in RAM already so that's not the source of the delay, but I imagine the transformation of a Python set (yes, type Set) of strings into a string model takes time.

    I guess I'll have to implement my own model then, not a big problem,. My understanding is that QCompleter invokes a QSortFilterProxy model - but what methods do I have to implement for this to work? I haven't found any documentation or sample code and I did spend quite a bit of time looking (maybe in all the wrong places though).

    Could you share a sample of what you have done or point me in the right direction?

    Thank you!

    Pezzi



  • The problem with QCompleter is that it really does not give you a way to customise its internal behaviour in regard to the data it uses. You cannot, for example, replace its internal sort/filter proxy with one that know about limits.

    Here's how you could do it:

    • Write a QAbstractListModel (or table model) model wrapper that accesses the existing in-memory data as-is. That is, don't duplicate the data into a QStringListModel if you can avoid it.
    • Assuming your model exists only for this completer then give it an equivalent of the QCompleter::setCompletionPrefix() and have the model present only the top-20 matches
    • Connect your line edit textChanged() signal to the model setCompletionPrefix()
    • Create your completer using the model and attach it to the line edit.

    I am not a Python-head so I cannot really give you usable example code



  • Hi Chris,

    I have been experimenting with this and a couple of issues came up:

    the setCompletionPrefix signal is emitted after data() is called in the model for that keystroke - so it's hard to modify the model's behavior based on the text (but I can always get the QLineEdit's current text from within the data model itself. A little ugly but doable.

    if I tell the QCompleter that I have a sorted list, it does two binary searches, one for the first and one for the last value in the model with that prefix, and then iterates through the entire interval between these boundaries, retrieving every matching item. This is where the multi-second delay comes from. I could limit the results to a sub-interval, e.g. 100 values starting with the lower boundary. But then when additional keys are entered (prefix gets longer) the QCompleter searches relative to the previous interval. Ouch!

    Any ideas?

    Pezzi



  • I wanted to add that with a minimal implementation I was able to eliminate the annoying time when creating the model - I keep the model around for the lifetime of the app, any QLineEdit w/completer can use it:

    This does not yet solve the problem of long times to load long result list when a fairly common prefix is entered. I would settle for preventing completions when the prefix is only 1-2 characters but have not found a way to do that just yet.

    @
    class BasicCompleterModel(QtCore.QAbstractListModel):
    def init(self, terms, parent = None):
    super(BasicCompleterModel, self).init(parent)
    self.terms = sorted(terms)
    self.num_terms = len(terms)
    self.line_edit = parent

    def rowCount(self, something):
        return self.num_terms  
    
    def data(self, index = None, role = None):
        if role in [QtCore.Qt.DisplayRole, QtCore.Qt.EditRole]:
            return self.terms[index.row()]
        return None
    

    class BasicLineEdit(QtGui.QLineEdit):
    def init(self, terms = None, parent = None):
    super(BasicLineEdit, self).init(parent)
    self.et = initialLoadEmTree()
    terms = sorted(self.et.all_terms_and_syns())
    print 'Number of terms', len(terms)
    self.completion_model = BasicCompleterModel(terms = terms, parent = self)
    self.completer = QtGui.QCompleter([], self)
    self.completer.setModel(self.completion_model)
    self.completer.setCaseSensitivity(QtCore.Qt.CaseInsensitive)
    self.completer.setModelSorting(QtGui.QCompleter.CaseInsensitivelySortedModel)
    self.completer.setMaxVisibleItems(20)
    self.setCompleter(self.completer)
    @



  • I was thinking more of using the QCompleter purely for its popup handling and going around it to have the model only present the desired "Top 20" hits to the completer. The completer then has a very limited set to work with, but your model needs to be smarter. Here is an implementation in C++ using an SQL table of 362880 options but a model that will return at most 20. The model is reset for each time the line edit changes (by user action) and the completer only ever sees 20 or fewer rows.

    @
    #include <QtGui>
    #include <QtSql>
    #include <QDebug>
    #include <algorithm>

    void createTestData()
    {
    QSqlDatabase db = QSqlDatabase::addDatabase("QSQLITE");
    db.setDatabaseName(":memory:");
    if (db.open()) {
    QSqlQuery query;
    query.exec("create table options(val varchar(10))");

        query.prepare("insert into options values (?)");
        QString letters("abcdefghi");
        do {
            query.bindValue(0, letters);
            query.exec&#40;&#41;;
        } while (std::next_permutation(letters.begin(), letters.end()));
    }
    

    }

    class MyModel: public QSqlQueryModel
    {
    Q_OBJECT
    public:
    explicit MyModel(QObject *p = 0): QSqlQueryModel(p) {
    setCompletionPrefix("");
    }

    public slots:
    void setCompletionPrefix(const QString &prefix) {
    qDebug() << Q_FUNC_INFO << prefix;
    QSqlQuery query;
    query.prepare(
    "select val from options where val like ? || '%' "
    "order by val limit 20"
    );
    query.bindValue(0, prefix);
    query.exec();
    setQuery(query);
    }
    };

    int main(int argc, char *argv[])
    {
    QApplication app(argc, argv);
    createTestData();
    QLineEdit le;
    QCompleter *completer = new QCompleter(&le);
    completer->setMaxVisibleItems(10);
    MyModel *model = new MyModel(&le);
    completer->setModel(model);
    le.setCompleter(completer);
    QObject::connect(&le, SIGNAL(textEdited(QString)),
    model, SLOT(setCompletionPrefix(QString)));
    le.show();
    return app.exec();
    }
    #include "main.moc"
    @



  • This is odd: I tried to translate this into Python. The data is loaded just fine and queries work - they return the top 20 results just fine - but no dropdown with the completions is rendered when I type into the QLineEdit. Am I doing something obvious wrong? Do you have any ideas how to put hooks into this thing to see what's going on? (which methods of the QSqlQueryModel does QCompleter call?)

    @
    class TestCompleterModel(QtSql.QSqlQueryModel):
    def init(self, terms, view_terms = 10, parent = None):
    super(TestCompleterModel, self).init(parent)
    self.db = QtSql.QSqlDatabase.addDatabase('QSQLITE')
    self.db.setDatabaseName(':memory:')
    if self.db.open():
    self.query = QtSql.QSqlQuery()
    self.query.exec_('create table options(val varchar(100))')
    self.query.prepare('insert into options values (?)')
    for term in terms:
    self.query.bindValue(0, term)
    self.query.exec_()

            self.query.exec_('select count(*) from options')
            print 'rows in SQLITE:'
            while self.query.next():
                print self.query.value(0)
    
            self.query.prepare("select val from options where val like ? || '%' order by val limit 20")
            self.query.bindValue(0, 'aspi')
            self.query.exec_()
            print 'top 20 matches:'
            while self.query.next():
                print self.query.value(0)
            
        self.view_terms = view_terms
        self.terms = sorted(terms)
        self.num_terms = len(terms)
        self.line_edit = parent  
        self.completion_prefix = ''
        
    @QtCore.Slot()        
    def setCompletionPrefix(self, prefix):
        print 'setCompletionPrefix', prefix
        self.completion_prefix = prefix
        self.query.prepare("select val from options where val like ? || '%' order by val limit 20")
        self.query.bindValue(0, prefix)
        self.query.exec_()
        self.setQuery(self.query)
    

    class TestLineEdit(QtGui.QLineEdit):
    def init(self, terms = None, parent = None):
    super(TestLineEdit, self).init(parent)
    self.et = initialLoadEmTree()
    terms = sorted(self.et.all_terms_and_syns())
    print 'Number of terms', len(terms)

        self.completer = QtGui.QCompleter(self)
        self.completer.setMaxVisibleItems(10)        
        self.completion_model = TestCompleterModel(terms, 10, self)
        self.completer.setModel(self.completion_model)
        self.textEdited.connect(self.completion_model.setCompletionPrefix)  
    

    @

    Many thanks

    Patrick


Log in to reply
 

Looks like your connection to Qt Forum was lost, please wait while we try to reconnect.