Making Unicode BiDi Control Characters visible in QTextEdit

IgKh

Hi All,

I'm working on a text editor focused on BiDi editing, to be used for documents where Unicode BiDi control characters (LRM/RLI/etc) are likely to be present. My end goal is to make those characters visible in QTextEdit. When they are present, QTextEdit can definitely interact with them - there is cursor "resistance" at the location, they can be deleted, cut/pasted and the like - but being invisible zero-width characters, it is all rather awkward.

Scouring the web for ways to make control characters visible in Qt Rich Text, it seems like there is basically two approaches:

Replace control characters with an alternative printable character / string - replacing back when saving.
Use a custom font.

Approach 1 is really problematic specifically for BiDi control characters - since they have unique impact on the layout of surrounding text, they have to stay. I could theoretically insert some sort of directionality neutral, yet printable, character next to each control character, but this still leaves the awkward editing experience.

So I tried to tackle approach 2 (custom font), but it doesn't seem to actually work. As proof of concept, I took the Unifont font, which has a glyph for every Unicode code point under the sun - including all control and replacement characters. But setting this as the editor font still makes no difference.

It appears to boil down to the shaping logic in QTextEngine giving control characters zero width despite a glyph being present in the font. For example, see the following minimal code:

QChar ch((ushort)0x200e);
QString str = ch;

QFont f1("Unifont", 11);
QFontMetrics fm1(f1);
qDebug() << fm1.horizontalAdvance(str); // Prints 0
qDebug() << fm1.horizontalAdvance(ch);  // Prints 15

QFont f2("Helvetica", 11);
QFontMetrics fm2(f2);
qDebug() << fm2.horizontalAdvance(str); // Prints 0
qDebug() << fm2.horizontalAdvance(ch);  // Prints 0

The QFontMetrics::horizontalAdvance(QChar) overload gives the LRM character some width, but the QFontMetrics::horizontalAdvance(QString) overload (which invokes actual shaping) says it has no width.

Looking deeper this behaviour comes from the underlying Harfbuzz library, which hides glyphs belonging to non-printable Unicode codepoints unless the HB_BUFFER_FLAG_PRESERVE_DEFAULT_IGNORABLES flag is set on the Harfbuzz buffer. QTextEngine does set it, but only if the platform specific QFontEngine implementation says the font uses Adobe's legacy symbol encoding; even if I could exploit it somehow, it looks like a specific workaround that shouldn't be relied on. The whole thing is buried deep in Qt's private parts, and it seems there is no good way to affect it.

Does anyone perhaps have any ideas regarding this, or an alternative way to tackle the original problem?

Thanks!

Qt Version: 6.6.1
OS: Arch Linux

IgKh

For some closure for anyone else interested, approach #2 (custom font) was made possible in Qt 6.9 with QTextOption::ShowDefaultIgnorables flag.

SGaist

Hi,

You might want to check KTextEditor.

IgKh

@SGaist Thanks for the reply!

Can you elaborate a bit on how using KTextEditor might help reaching the goal?

From cursory look at the source code it would appear that rendering on the actual text is still done via QTextLayout, so would use the same text shaping algorithms as Qt's own rich text framework. Indeed, trying a test document in Kate and KWrite, they have the same behavior around BiDi control characters as a regular QTextEdit widget.

SGaist

From memory (which was wrong as it seems), I thought it was built differently. Sorry for the wrong direction.

IgKh

For some closure for anyone else interested, approach #2 (custom font) was made possible in Qt 6.9 with QTextOption::ShowDefaultIgnorables flag.