I solved this.
I moved everything related to the serial port into a separate class SerialWorker and moved it to a separate thread.
QThread* serialthread = new QThread;
SerialWorker* serialworker = new SerialWorker();
serialworker->moveToThread(serialthread);
connect(serialthread, SIGNAL(started()), serialworker, SLOT(InitSerialPort()));
serialthread->start();
connect(serialworker, SIGNAL(NewProto(QByteArray)), this, SLOT(ProtocolInterpreter(QByteArray)));
connect(this, SIGNAL(SendRS(QByteArray)) , serialworker, SLOT(writeData(QByteArray)));
This works as supposed. Serial port is now more or less very responsive.
WebKit is still blocking the GUI thread sometimes but thats not as much an issue as the serial port not responding.