Is there a way to optimise QSyntaxHighlighter for really big files?
-
Thank you for that piece of code you posted, I wrote a hack that used the scrollbar value to get that info but it was very inconvenient and it missed out on some blocks (at the end)
I will take a shot at writing my own syntax highlighter then, because opening a file and appending that to a QPlainTextEdit takes significantly less time than highlighting it. The speed is a concern for me since I'm developing an IDE.
But I do have one more question: I've tried messing around with character formats before but I could never make them "stick" like QSyntaxHighlighter does it, mine would just disappear after a textChanged signal is called.
@Fuchsiaff I think you could meet your need by modifying some parts of QSyntaxHighlighter base implementation. There is already a mechanism which update only blocks which are modified and it could be changed to update blocks only when they are displayed on the screen, like a MVC architecture.
But before your start you may want to try QScintilla which could be a good alternative. I don't know if it's more efficient, but I think it is not a big work to try compared to write your own highlighter.
@Fuchsiaff said in Is there a way to optimise QSyntaxHighlighter for really big files?:
I've tried messing around with character formats before but I could never make them "stick" like QSyntaxHighlighter does it, mine would just disappear after a textChanged signal is called.
Not sure about that, but based on QSyntaxHighlighter::applyFormatChanges() source code you probably need to apply the format on the layout, using
QTextLayout::setFormats(), and callQTextDocument::markContentsDirty()to tell to the document it needs to update part which has changed. -
@Fuchsiaff I think you could meet your need by modifying some parts of QSyntaxHighlighter base implementation. There is already a mechanism which update only blocks which are modified and it could be changed to update blocks only when they are displayed on the screen, like a MVC architecture.
But before your start you may want to try QScintilla which could be a good alternative. I don't know if it's more efficient, but I think it is not a big work to try compared to write your own highlighter.
@Fuchsiaff said in Is there a way to optimise QSyntaxHighlighter for really big files?:
I've tried messing around with character formats before but I could never make them "stick" like QSyntaxHighlighter does it, mine would just disappear after a textChanged signal is called.
Not sure about that, but based on QSyntaxHighlighter::applyFormatChanges() source code you probably need to apply the format on the layout, using
QTextLayout::setFormats(), and callQTextDocument::markContentsDirty()to tell to the document it needs to update part which has changed. -
I've used QScintilla in the past but I found it boring.
Creating my own is more of a challenge and I have more customising options. Later I will add code navigation with ctags and a lot more.
@Fuchsiaff Nice challenge indeed. I would be interested to do the same if I had time for it.
Good luck for this project !! -
@Fuchsiaff Nice challenge indeed. I would be interested to do the same if I had time for it.
Good luck for this project !!So I tried out your code and the line
bottomRight = QPoint(self.editor.viewport().width() - 1, self.editor.viewport().height() -1 )doesn't seem to workIt returns this:
PyQt5.QtCore.QPoint(1354, -1)If I remove the -1's then it doesn't work AKA it thinks that endPos is 0 whereas that's the startPos value
-
So I tried out your code and the line
bottomRight = QPoint(self.editor.viewport().width() - 1, self.editor.viewport().height() -1 )doesn't seem to workIt returns this:
PyQt5.QtCore.QPoint(1354, -1)If I remove the -1's then it doesn't work AKA it thinks that endPos is 0 whereas that's the startPos value
Nevermind, my issue, works now :)
-
Here's a demo of what I have right now (currently it doesn't highlight anything when writing text, haven't added that functionality just yet)
As you can see if you scroll too fast, the highlighting won't keep up, this is because it highlights one line every time the value changes, this SHOULD be an easy fix but if it highlights too many blocks at once then the whole thing will lag and crash.
-
Here's a demo of what I have right now (currently it doesn't highlight anything when writing text, haven't added that functionality just yet)
As you can see if you scroll too fast, the highlighting won't keep up, this is because it highlights one line every time the value changes, this SHOULD be an easy fix but if it highlights too many blocks at once then the whole thing will lag and crash.
@Fuchsiaff This looks very promising !
@Fuchsiaff said in Is there a way to optimise QSyntaxHighlighter for really big files?:
this is because it highlights one line every time the value changes
What do you mean by "the value" ? position of the scrollbar ?
-
@Fuchsiaff This looks very promising !
@Fuchsiaff said in Is there a way to optimise QSyntaxHighlighter for really big files?:
this is because it highlights one line every time the value changes
What do you mean by "the value" ? position of the scrollbar ?
-
Yeah position of the scrollbar.
It basically works like QSyntaxHighlighter now but with better performance since it doesn't highlight the whole file at once.
The file I opened in the video was 2.56mb
@Fuchsiaff Very Nice ! You may propose this new version to the Qt team :).
-
@Fuchsiaff Very Nice ! You may propose this new version to the Qt team :).
Well there are many many problems with my implementation, since I couldn't find a way to limit the scrolling velocity, scrolling too fast will result in the highlighting not able to keep up since it highlights one block every time the scrollbar is moved.
Now a logical solution to that would be that just highlight more blocks at once. Nope, that will make the performance of scrolling utterly sluggish.
Also, I wonder why when opening a 2.56mb file and after highlighting it, typing very fast and stopping, will result in characters still appearing for a bit, it's like the QPlainTextEdit and the highlighting can't keep up with the keyboard events. That shouldn't happen because highlighting only what's modified shouldn't be too time consuming.
