Is there a way to optimise QSyntaxHighlighter for really big files?

Fuchsiaff

Or do I have to write the highlighter myself to handle big files?

Because right now it will highlight the whole file, it doesn't care how big it is.

I would want to highlight only a certain portion of it and when the users scrolls then it'll highlight more

[EDIT]

Is there a way to check if text block is visible in a QPlainTextEdit, using QTextBlocks isVisible() method doesn't seem to work since it returns True for blocks that are clearly not visible.

Gojir4

@Fuchsiaff Hi,

This is a very interesting question. I did some try without success. It seems to me the only way is to rewrite at least a part of the QSyntaxHighlighter api. But the "delay" happens only the first time the text is set, then only the modified parts are highlighted again.

About QTextBlock::isVisible(). It's not the same meaning than QWidget::isVisible(). It means then the block is visible in the document. As an example you can use this visible property to make something similar to code folding .

Here is a way to retrieve blocks which are visible (really displayed on the screen):

int startPos = textedit->cursorForPosition(QPoint(0, 0)).position();
QPoint bottom_right(textedit->viewport()->width() - 1, textedit->viewport()->height() - 1);
int endPos = textedit->cursorForPosition(bottom_right).position();

QTextCursor cursor = textedit->textCursor();
cursor.setPosition(startPos);
QTextBlock startBlock = cursor.block();
cursor.setPosition(endPos);
QTextBlock endBlock = cursor.block();

int startLine = startBlock.firstLineNumber();
int endLine = endBlock.firstLineNumber() + cursor.block().lineCount();
qDebug() << "Lines visible: " << startLine << endLine;

int line = 0;
//Iterate visible blocks
for(QTextBlock b = startBlock; b.isValid() && b != endBlock; b = b.next()){    
    qDebug() << line++ << b.text();
}

Fuchsiaff

@Gojir4

Thank you for that piece of code you posted, I wrote a hack that used the scrollbar value to get that info but it was very inconvenient and it missed out on some blocks (at the end)

I will take a shot at writing my own syntax highlighter then, because opening a file and appending that to a QPlainTextEdit takes significantly less time than highlighting it. The speed is a concern for me since I'm developing an IDE.

But I do have one more question: I've tried messing around with character formats before but I could never make them "stick" like QSyntaxHighlighter does it, mine would just disappear after a textChanged signal is called.

Gojir4

@Fuchsiaff I think you could meet your need by modifying some parts of QSyntaxHighlighter base implementation. There is already a mechanism which update only blocks which are modified and it could be changed to update blocks only when they are displayed on the screen, like a MVC architecture.

But before your start you may want to try QScintilla which could be a good alternative. I don't know if it's more efficient, but I think it is not a big work to try compared to write your own highlighter.

@Fuchsiaff said in Is there a way to optimise QSyntaxHighlighter for really big files?:

I've tried messing around with character formats before but I could never make them "stick" like QSyntaxHighlighter does it, mine would just disappear after a textChanged signal is called.

Not sure about that, but based on QSyntaxHighlighter::applyFormatChanges() source code you probably need to apply the format on the layout, using QTextLayout::setFormats(), and call QTextDocument::markContentsDirty() to tell to the document it needs to update part which has changed.

Fuchsiaff

@Gojir4

I've used QScintilla in the past but I found it boring.

Creating my own is more of a challenge and I have more customising options. Later I will add code navigation with ctags and a lot more.

Gojir4

@Fuchsiaff Nice challenge indeed. I would be interested to do the same if I had time for it.
Good luck for this project !!

Fuchsiaff

@Gojir4

So I tried out your code and the line bottomRight = QPoint(self.editor.viewport().width() - 1, self.editor.viewport().height() -1 ) doesn't seem to work

It returns this: PyQt5.QtCore.QPoint(1354, -1)

If I remove the -1's then it doesn't work AKA it thinks that endPos is 0 whereas that's the startPos value

Fuchsiaff

@Fuchsiaff

Nevermind, my issue, works now :)

Fuchsiaff

@Gojir4

Here's a demo of what I have right now (currently it doesn't highlight anything when writing text, haven't added that functionality just yet)

https://imgur.com/a/pkZeJcp

As you can see if you scroll too fast, the highlighting won't keep up, this is because it highlights one line every time the value changes, this SHOULD be an easy fix but if it highlights too many blocks at once then the whole thing will lag and crash.

Gojir4

@Fuchsiaff This looks very promising !

@Fuchsiaff said in Is there a way to optimise QSyntaxHighlighter for really big files?:

this is because it highlights one line every time the value changes

What do you mean by "the value" ? position of the scrollbar ?

Fuchsiaff

@Gojir4

Yeah position of the scrollbar.

It basically works like QSyntaxHighlighter now but with better performance since it doesn't highlight the whole file at once.

The file I opened in the video was 2.56mb

Gojir4

@Fuchsiaff Very Nice ! You may propose this new version to the Qt team :).

Fuchsiaff

@Gojir4

Well there are many many problems with my implementation, since I couldn't find a way to limit the scrolling velocity, scrolling too fast will result in the highlighting not able to keep up since it highlights one block every time the scrollbar is moved.

Now a logical solution to that would be that just highlight more blocks at once. Nope, that will make the performance of scrolling utterly sluggish.

Also, I wonder why when opening a 2.56mb file and after highlighting it, typing very fast and stopping, will result in characters still appearing for a bit, it's like the QPlainTextEdit and the highlighting can't keep up with the keyboard events. That shouldn't happen because highlighting only what's modified shouldn't be too time consuming.