Special characters in XML files created with QDomDocument

ionwind

I'm using Qt to write a program that creates a XML files that can be opened in Excel. So far I have been progressing rather well but now I have this problem that I need to put newline characters to some of the cells containing some text. Excel XML sheets use 
 as the newline character.

So the problem is that with this:
@QString contents = "Foo&amp#10;Bar";
QDomText cellContents = doc->createTextNode(contents);
data.appendChild(cellContents);@
I get this:
@<Cell><Data ss:Type="String">Foo&ampamp;#10;Bar</Data></Cell>@
While this is what I need:
@<Cell><Data ss:Type="String">Foo
Bar</Data></Cell>@

I tried this:
@doc->implementation().setInvalidDataPolicy(QDomImplementation::AcceptInvalidChars);@

But either that doesn't help here or then there's something wrong on how I'm doing this. Anyone cares to help?

zester

Characters like "<" and "&" are illegal in XML elements.

Entity References <--- See this section
http://www.w3schools.com/xml/xml_syntax.asp

also see

http://www.w3schools.com/xml/xml_cdata.asp

ionwind

Yeah, I know the basics of xml and I know they're illegal :D
I was just wondering if it is possible to anyway break the rules or to create some kind of a workaround. At least Microsoft breaks the rules: If you create a spreadsheet document in Excel and save it as .xml and you have there a cell with newline it'll appear as in the xml document.

zester

The out put that your getting that you say you don't want use that as your string.

ionwind

If I remember correctly I tried that before, then the output was something like
@<Cell><Data ss:Type="String">Foo&ampamp;&ampamp;#10;Bar</Data></Cell>@
But I'll try I tomorrow just to make sure...

zester

Hmmmm I am not sure how it would be done on Windows.

But from reading I see that...

In Windows applications, a new line is normally stored as a pair of characters: carriage return (CR) and line feed (LF). In Unix applications, a new line is normally stored as an LF character. Macintosh applications also use an LF to store a new line.

XML stores a new line as LF.

goetz

This should do the trick:

@
QString contents = "Foo\nBar";
QDomText cellContents = doc->createTextNode(contents);
data.appendChild(cellContents);
@

Wether the newline is put as an numeric entity (
) or as a literal character is irrelevant. XML wise, both are equivalent.

[EDIT:] PS:
createTextNode() manipulates the input text in such a way, that when the document is parsed again the then retrieved string is exactly the same as your original one. So, every single ampersand (&) is transformed into a &. The same holds for "<" and ">", and in some cases the single and double quotation marks (' and ").

zester

I'm not sure that will work volker but then again I could be wrong.

What the OP was getting at was that excel needs to uses special characters that are illegal in xml and the parser is converting the characters in such away to make them legal, which end's up not being the proper format for excel.

In my previous post I was back tracking trying to figure out why that maybe.

Excel supports XML-based files, but the file must conform to some rules (excel XML sheet XSD schema).
http://en.wikipedia.org/wiki/Microsoft_Office_XML_formats#Excel_XML_Spreadsheet_example

Not sure if that will help ;)

Chris Kawa

Why not just put your text inside CDATA section i.e. instead of QDomText use QDomCDATASection?
&, < and newlines are all valid text inside that markup.

ionwind

Thanks for all the suggestions, but unfortunately none of them seems to work so far :(

zester's got the right idea there what I'm up to here ;)

I had tried before what Volker suggested. \n in the string does this to the output file:
@<Cell><Data ss:Type="String">Foo
Bar</Data></Cell>@
While that's right it's not what I'm after here, since in Excel that shows up as a space between Foo and Bar, not a newline.

I was also wondering if CDATASection suggested by Krzysztof would work. I was rather hopeful when I saw this to appear in the output file:
@<Cell><Data ss:Type="String"><![CDATA[Foo&amp#10;Bar]]></Data></Cell>@
But unfortunately, when opening the output file in Excel, the contents of the cell were still Foo
Bar instead of Foo and Bar being on separate lines.

If anyone got any more suggestions I'd be happy to try them, but I guess it's simpler if I just arrange the output file so that I don't need the damn newlines :P

zester

It's a case of your damed if you do and damed if you don't thank microsoft
for there non-standard ways lol ;) I will keep looking I have run into this problem
before I just can't for the life of me remember how I solved it. Maybe when volker
get's a free minute from the Qt Summit he can enlighten us.

jim_kaiser

So, sounds like you needs ASCII char 10 in your data. Did you try this?

@ QString contents = QString("Foo %1 Bar").arg((char)10); @

Remove the spaces around %1 .. seems code tag is eating it without the spaces..

[ Edit: No different from putting a \n ofcourse.. that doesn't work in excel?? ]

[ Edit2: Just use a string with \n while in Qt.. in the end when you save to an excel file... convert characters to excels format. That should work no? Okay my bad.. i see what you need.. the special chars from excel in Qt.. ]

Chris Kawa

When you use CDATA everything you put there will be used as is, without any parsing so no wonder it shows untransformed 

Maybe you tried it already, but I think the mix of those two solutions i.e. CDATA and \n should work.

[Edit.] Heh, just a wild idea just now if that doesn't work. You can combine CDATA and text like so:
@ <![CDATA[some text]]>
<![CDATA[some more text]]> @

goetz

Guys! Get you some book about basic XML!

XML wise all this is equivalent:

@

 == '\n' == QChar(10)
@

It is completely irrelevant if you put 
 or '\n' into an XML parser! If it makes a difference, the parser is not standards compliant.

And just to prove: Make a hand crafted xlsx file an just put a newline in it. At least in OpenOffice it is displayed with a line break. I didn't have a "genuine" Excel at hand to test.

ionwind

jim_kaiser, just to mention that \n doesn't work in Excel. Anyway the last solution you posted would work of course. I take it you mean something like this:
@QFile exportFile(fileName);
if(exportFile.open(QIODevice::WriteOnly))
{
QTextStream TextStream(&exportFile);
QString docString = doc->toString().replace("&ampamp;", "&");
TextStream << docString;
exportFile.close();
}@
I could do it that way, yes. In my case the exported XML files have thousands of lines and the largest ones have "only" 30-50 thousand lines, so it's not a problem, even though searching and replacing bits of that large string just isn't programmatically good-looking ;P

Krzysztof, thanks for the suggestions but so far any combination of CDATA and any newline sequence doesn't seem to do the trick :\

Chris Kawa

@Volker It doesn't make a difference if it's in a text node, but id does if it's in the CDATA node. 
 is a 4-letter string there and not a newline mark.
Other than that what you said is absolutely true - that's why I think \n (or QChar(10) which is not necessarily the same with windows 2-char newlines is it?) should work just fine inside CDATA node(used to have those & and <), but if ionwind says it doesn't than I don't really know..

zester

@Volker when ubuntu finishes there Qt Unity Interface and all those gtk developers turn into qt developers,
and you have 40,000 american ubuntu users swarming the qt forums wanting help learning qt.

There going to chew you up and spit you out my friend, lol If I was you I would work on my people skills.
and how do I know this because I did gtk development support for ubuntu at one time I had a single thread
about gtk programming. It received 142,000 hits and 3,000 comments and everyone wanted to be kindly spoon-feed.
If you want proof of my claims just ask I am sure I can dig that old thread up.

just saying ;)

Number of Ubuntu users = 12+ million :)
Ubuntu's current focus = Qt

ionwind

In case someone wants to try it out, here's a xml file that can be opened in Excel. On the first cell (that couldn't be made with Qt, so far) there's a newline that Excel understands, the other three are suggestions made here that can be produced by Qt but won't work in Excel.
Thanks for everyone anyway ;D
@<?xml version="1.0"?>
<?mso-application progid="Excel.Sheet"?>
<Workbook xmlns:ss="urn:schemas-microsoft-com:office:spreadsheet">
<Worksheet ss:Name="Newline">
<Table>
<Row>
<Cell><Data ss:Type="String">Foo&amp#10;Bar</Data></Cell>
<Cell><Data ss:Type="String">Foo&ampamp;#10;Bar</Data></Cell>
<Cell><Data ss:Type="String">Foo<![CDATA[&amp#10;]]>Bar</Data></Cell>
<Cell><Data ss:Type="String">Foo\nBar</Data></Cell>
</Row>
</Table>
</Worksheet>
</Workbook>@

[Edit] There should be xmlns="urn:schemas-microsoft-com:office:spreadsheet" in Workbook element too, but for some reason it won't appear in the code :|

mlong

You could always render your well-formed XML into a QString (or QByteArray, maybe?) and then use a QRegExp to malform it into something hideous that Excel can understand (i.e., replace all instances of 
 with )

It's a dirty hack, and not without it's limitations, but it might work...

goetz

Hi,

the following example works for me.

It creates an xlsx file with two cells. The first contains a text without line breaks, the second cell has a line break after each word.

It's displayed correctly in OpenOffice here. As I'm in a train now and have no access to a windows box, I cannot check on genuine Microsoft Office's Excel - I'll do that on Tuesday. I'd be happy to get feedback though.

As you can see, it's all Qt DOM methods in use, no fancy search and replace magic.

@
#include <QApplication>
#include <QDebug>
#include <QtXml>
#include <QFile>
#include <QTextStream>

QDomElement appendDomNode(QDomNode &parent, const QString &name, const QString &content = QString())
{
QDomDocument doc = parent.ownerDocument();
QDomElement elem = doc.createElement(name);
if(!content.isNull()) {
QDomCharacterData pcdata = doc.createTextNode(content);
elem.appendChild(pcdata);
}
parent.appendChild(elem);
return elem;
}

int main(int argc, char *argv[])
{
QApplication a(argc, argv);

QDomDocument doc;
QDomProcessingInstruction xmlVers = doc.createProcessingInstruction("xml", "version=\"1.0\" encoding='utf-8'");
doc.appendChild(xmlVers);

QDomElement Workbook = doc.createElementNS("urn:schemas-microsoft-com:office:spreadsheet", "ss:Workbook");
doc.appendChild(Workbook);

QDomElement Styles = appendDomNode(Workbook, "ss:Styles");
QDomElement Style = appendDomNode(Styles, "ss:Style");
Style.setAttribute("ss:ID", "1");

QDomElement Worksheet = appendDomNode(Workbook, "ss:Worksheet");
Worksheet.setAttribute("ss:Name", "Sheet1");

QDomElement Table = appendDomNode(Worksheet, "ss:Table");

QDomElement Column_1 = appendDomNode(Table, "ss:Column");
Column_1.setAttribute("ss:Width", "80");

QDomElement Column_2 = appendDomNode(Table, "ss:Column");
Column_2.setAttribute("ss:Width", "80");

QDomElement Row = appendDomNode(Table, "ss:Row");
Row.setAttribute("ss:StyleID", "1");

QDomElement Cell_1 = appendDomNode(Row, "ss:Cell");
QDomElement Data_1 = appendDomNode(Cell_1, "ss:Data", "A string without line breaks");
Data_1.setAttribute("ss:Type", "String");

QDomElement Cell_2 = appendDomNode(Row, "ss:Cell");
QDomElement Data_2 = appendDomNode(Cell_2, "ss:Data", "A\nstring\nwith\nline\nbreaks");
Data_2.setAttribute("ss:Type", "String");

qDebug() << doc.toString();

QFile xlsxFile&#40;"/tmp/qttest.xlsx"&#41;;
xlsxFile.open(QIODevice::WriteOnly);
QTextStream ts(&xlsxFile);
doc.save(ts, 4);
xlsxFile.close();

return 0;

}
@