How to visualize LF, CR and other control pictures in QTextEdit?

Robert Hairgrove

I know that there is a whole Unicode plane devoted to "Control Pictures", but I find the characters render impossibly small in all fonts I have seen so far. Examples (and this is using bold highlighting): ␀␁␂␃␄␅

Should I resort to using inline PNG images or pixmaps, or is there a better way?

SGaist

Hi,

Are you replacing the font for only these characters ?
If so, you could create your own format that renders them at a sensible size.

Robert Hairgrove

@SGaist Thanks for the suggestion. I will have to try this. But I would like to keep the overall glyph size the same as the surrounding characters, so that the line spacing is consistent. If I were doing this in HTML with modern CSS, I could probably come up with some formatting which would give me control over the baseline for single characters, etc. Not sure if QTextEdit has enough CSS support for this, though.

Geany (on Linux) does a pretty good job at highlighting the line endings. I wonder how they do it? Both Geany and Qt Creator, for example, are based on Scintilla. I looked at the source code for Geany but haven't seen anything about this yet.

I should also mention that I need separate characters for CR and LF (as well as for the other control pictures)..

JonB

I am a little lost on/curious about this. I am assuming you are using QTextEdit with its "rich" text (i.e. HTML) mode? Then HTML interprets, say, LF simply as whitespace (unless you are inside, say, a <pre> or similar --- are you?). And does not render anything for it. And really very similar even if it is in plain text mode. So you are doing some work on top of the HTML (or plain text) handling which is inbuilt in QTextEdit?

Robert Hairgrove

@JonB I would like to do what many text editors do when the user selects "Show Spaces" or "Show Line Endings" (like Geany, for example). I believe that Notepad++ and UltraEdit also offer this capability.

I know that QTextEdit does not show anything by itself for LF, etc. so I would use other characters instead of the real CR, LF, etc. which can be visualized. As to HTML, I am trying to stick to the API offered by QTextDocument, QTextBlock and QTextCursor if possible.

Pl45m4

@Robert-Hairgrove said in How to visualize LF, CR and other control pictures in QTextEdit?:

I believe that Notepad++

Check Npp's source code?! Maybe you find something.

JonB

@Robert-Hairgrove
I am trying to understand how you will do this. You are using a QTextEdit, in HTML mode, which already does its own processing of, say, LF/newline to do whatever HTML does with it in its context. You are going to have examine every character of the source document to see if it's a LF and output a visual character for it, in the right place in the visual document, before allowing QTextEdit to do whatever it wants with it? You will output an extra glyph for this from the renderer? Or actually insert some HTML entity character into the current HTML?

Robert Hairgrove

@JonB Yes, something like that (i.e. inserting placeholder characters).

If I have a line of text which comes from a document that someone gave me, for example which was downloaded from a Windows server onto an iMac with the wrong configuration (as "text" instead of "binary"), it might have strange line endings such as this (from a real-life example):

The quick brown fox[CR][CR][LF]
// next line

then I want to show each newline character so that the user can see what is going on.

Note that this is for a read-only view; no editing is involved.

JonB

@Robert-Hairgrove I get what you want, I just don't know how well that fits in with QTextEdit. Best of luck.

Pl45m4

@JonB

I don't know if it was you who started the topic, but I remember there was a discussion about the "variable/argument hints" in CodeBrowser (for Qt source code) and a guy who tried to implement something like this as text-based QWidget...

// function
void foo(int someInt, double someDouble);

// with call
foo(42, 13.37);

// with added "hint labels"
// (non-functional, only read-only as Robert describes)
foo(["someInt":] 42, ["someDouble":] 13.37);

So my thought: If this is possible, showing/adding control characters should also be doable.
But don't ask me where and when this was and how it ended... :)

SGaist

Did you already saw the QTextOption enum ?
It seems to set what you want with caveat that the font used needs to support the stuff you want to show ?
I just stumbled upon it while remembering there was something somewhere to at least show the line return.

Robert Hairgrove

@SGaist Thanks again!

I just did a little test ... here is the outcome. Unfortunately, the display shows CR and LF exactly the same way:

It was a good idea, though!

JonB

@Robert-Hairgrove
But does your #define line literally have the content shown in the visual, i.e. something like

#define SAMPLE_TEXT \
  "The quick brown fox\r\r\nA line ..."

i.e. you typed in a literal string including C++ \r\n "escape" sequences, or do you mean that the macro actually contains physical CR/LF characters in those positions? These are not the same as each other...!

Also while it is here, if you are intending to paste in C++ code I do not think QTextEdit in rich text/HTML mode is "safe" to get the correct content/output reliably. You should be using QPlainTextEdit, or maybe QTextEdit but forced to be in text/non-HTML mode.

Robert Hairgrove

@JonB I don't understand the question ... in C++, you must escape any control codes, otherwise the compiler would complain.

The macro is like this (copied and pasted from my code):

#define SAMPLE_TEXT \
  "The quick brown fox\r\r\nA line with\tTAB...\nSome more text here"

Then, in the constructor of the main window, I set the text like this:

  QString x = QString::fromLocal8Bit(SAMPLE_TEXT);
  ui->textEdit->setText(x);

As to QPlainTextEdit vs. QTextEdit, I don't know what I will eventually use. To achieve my goal, it looks like I will end up using inline pixmaps instead of actual characters, which would imply using HTML or Markdown text. The other alternative would be to create my own font, which I really don't want to do.

JonB

@Robert-Hairgrove
If you paste arbitrary C++ code into a QTextEdit in HTML mode, how do you know whether there might be character sequences which would be interpreted as HTML rather than literal, e.g. if the code happens to have <pre> or & or similar in it?

Robert Hairgrove

@JonB For my purposes here as a demo, it doesn't matter. The QTextEdit widget is always set to read-only, anyway.

JonB

@Robert-Hairgrove said in How to visualize LF, CR and other control pictures in QTextEdit?:

The QTextEdit widget is always set to read-only, anyway.

Nothing to do with widget read-only-ness. The issue will come when you take arbitrary C++ code and use QTextEdit::insertHtml() or similar from your code. Anyway I will leave that thought with you.

Robert Hairgrove

I am marking this as solved, since I developed a work-around for what I am doing. Basically, I am using the following characters which are supported by the most common fonts:

Description	Symbol	Unicode valiue
Carriage return (CR, 0x0D):	`¬`	U+00AC
Line feed (LF, 0x0A):	`¶`	U+0086
Space (SPC, 0x20):	`·`	U+0087
Tab (TAB, 0x09):	`→`	U+2192
Invalid code point:	`�`	U+FFFD
Miscellaneous control code:	`¤`	U+00A4

If Invalid code point or a Miscellaneous control code are detected, the user can hover over the character with the mouse and a tooltip will display the actual Unicode code point as a hexadecimal number.

Thanks for all the useful suggestions!

SGaist

How did you implement it in the end ?

Robert Hairgrove

@SGaist

How did you implement it in the end ?

I'm still working on it. :)

The app is kind of a toolkit for inspecting CSV files and correcting them to a limited extent, if necessary. These files can come from anywhere, and often the consumer of a CSV file doesn't have any control over the process which creates them. What it is supposed to do is the following:

Determine the text encoding used, either semi-automatically (if Unicode is used, this can be 100% automatic) or by letting the user choose, or for unknown 8-bit encodings iconv, uchardet or icu can be used if the support is installed;
If the encoding cannot be determined, the file is rejected, or else the user can try different encodings. This is where the control characters can be displayed;
As to how I am implementing this internally, a copy of the text from the file is made, converting it to QString and using QTextEdit to display it. All of the special control characters can either be displayed or hidden depending on the user's preference; also the highlighting (colors, etc.) is user-configurable;
The delimiter token and quote character can be automatically determined through heuristics, or the user can override these to something else;
There is a limited amount of support for comments and metadata at the beginning of a file. The user can add or remove these, and the file can be saved without the extra metadata, and/or in a different encoding, lie endings, etc.

Since I don't want to turn this into a full-fledged text editor, the app will just print out a report if the CSV file is not acceptably formatted, indicating the location of invalid characters, and the user can correct this in some other application. When importing CSV data into a database, for example, often the import will fail without giving enough diagnostic information as to where the failure occurred. This app should be helpful in that respect.