Performance issues with multiple visible Windows

SGaist

Sorry I wasn't clear, what I meant is to always code with minimal copy in mind Currently, AFAIK, you can't avoid this copy however that doesn't mean you shouldn't use e.g. const QString &myString as parameter signature there might be other optimisation going on that you might loose otherwise. The fact that Qt does a deep copy when using queued connections should rather be considered an implementation detail although you have to take it into account when using hight volume of data.

So by all mean, don't try to use direct connections by hand, you'll likely shoot yourself in the foot at some point without even noticing.

One other thing you can do since you are using a QTreeView is to set uniformRowHeights to true if it's the case, you might gain some milliseconds again.

I just thought of something: how much memory are you consuming ? With that many models being update so frequently you might also be hitting the swap at some point.

Zmote89

@SGaist

So I measured memory consumption over 1 minute with 20 Clients (Windowed), it starts about 50mb and htis ~ 400mb after 1 minute. There will be a logLimit, which per requirement, is to be 10'000 logs per Model, which should stabilize memory consumption at some point.

I took snapshots of the heap:
alt text

What exactly is that swaping issue? Could you elaborate?

Furthermore, I've already activated setUniformRowHeights, so I think I'm good there. I'm currently trying to optimize to const ref's where possible.

Is there any other way to increase view update speed? Like for threads, you can set QThread::currentThread()->setPriority(QThread::HighPriority) to prioritize a thread, is there a setting like that for views? Does Qt not provide ways to write your own Views or it's update cycle?

Thanks for all the help, I've achieved significant performance boost already, but this 20 Windows lagging thing is still present and it could be a deal breaker for my project, so I'm all ears for any further tipps, even unorthodox ones! :)

SGaist

The swap partition is used when application are starting to use up all amiable memory, the system then start to swap in/out memory pages between the RAM and the partition on the hard drive.

By the way, why a QTreeView ? Are you using any feature from it ?

Zmote89

@SGaist
Ah ok, now I get what you mean with swap.

I used QTreeView because 1) It seems to perform better, 2) the default look works in my favor (I don't want the borders of the table).

As my current system has active 12GB ram, that shouldn't be that much of a problem. Furthermore, I once ran the application to 100'000 log entries for each of the 20 Clients and it went up to 2GB~. With the logLimit of 10'000 logs per Model memory shouldn't become much of a problem.

SGaist

Wouldn't a QListView do what you want ?

Zmote89

@SGaist

As far as I know, QListView doesn't support columns and I need columns.

SGaist

For filtering or just for formatting ?

Zmote89

@SGaist

Filtering and formatting. Some columns need to be able to be turned off and on and the "log table" needs to be able to be filter according to columns.

SGaist

In your custom model, are you returning the minimal from your data method ?

Zmote89

@SGaist

Tha's how my current data function looks in my model:

QVariant CustomTableModel::data(const QModelIndex & index, int role) const
{
	if (!index.isValid())
		return QVariant();

	if (index.row() >= listOfEntries.size() || index.row() < 0)
		return QVariant();

	if (role == Qt::DisplayRole) {
		if (index.column() < 0 || index.column() > 6)
			return QVariant();
		if(index.column() == 0)
		{
			return QString::number(index.row());
		}
		return listOfEntries.at(index.row()).at(index.column()-1);
	}
	return QVariant();
}

SGaist

Looks good.

IIRC, you can avoid the QString::number call and just return index.row() or something along the lines of:

QVariant CustomTableModel::data(const QModelIndex & index, int role) const
{
    if (!index.isValid() || role != Qt::DisplayRole)
        return QVariant();

    const int column = index.column();
    const int row = index.row();
    if (column < 0 || column > 6)
        return QVariant();
    if(column == 0) {
        return row;
    } else {
        return listOfEntries.at(row).at(column-1);
    }
    return QVariant();
}

Zmote89

For those who are interested, I now how a working solution, thanks to the help of my advisors at the HSR.

As the main problem lay with the updating of multiple views, I first tried to change updates to bulk updates, with an updater thread signaling when to update the GUI.
Meaning:

A CaptureWorker in a Thread gathered logs until it was signalled by
A UpdateWorker, which in a set amount of time, sent an "update" signal to the CaptureWorker, which in turn signaled, with the logs,
RT_Main, ie., my MainWindow's handleProcessedLogs

This meant, that for example every 200ms multiple clients where updated at once.

The problem that issued from this solution, was that too many update view signals where created at the same time, which froze the GUI.

The new solution basically spreads the updates in the given interval on a per client basis, ie. there are still 20 Clients updated within the 200ms interval, but now they are progressively updated during the 200ms intervall and not at once every 200ms. The current solution looks like this:

Main Window:

void RT_Main::handleProcessedLogs(const QString & clientName, const QList<QList<QString>> & rows) const
{
	handleClientLog(modell.getConstClient(clientName), rows);
}

void RT_Main::handleClientLog(const RT_Client& client, const QList<QList<QString>>& logs) const
{
	if (client.live())
	{
		client.addLog(logs);
	}
	if (client.toFile() || client.toRingFile())
	{
		signalToFileLog(logs, client);
	}
	if (client.getClientId() != RT_Constants::GLOBAL_ID && client.toParent())
	{
		handleClientLog(modell.getConstGroupClient(client.getGroupName()), logs);
	}
	if (client.toGlobal())
	{
		modell.globalClient.addLog(logs);
	}
}

void RT_Main::adjustUpdateInterval()
{
	int windowCount = modell.getWindowedClientsCount();
	modell.updateInterval(windowCount > 1 ?
		RT_Constants::baseUpdateInterval + windowCount * (RT_Constants::baseUpdateInterval / 4)
		: RT_Constants::baseUpdateInterval);
}

The adjustUpdateIntervall is called whenever a window is created or closed, there is a base Interval, ie. the lowest update frequency, whcih is at 200ms when there are no additional log windows, and gets increased with every new window (ie. UpdateWorker's signal interval ranges from 200ms to 1250ms at 0 - 20 Windows). The modell.updateInterval is thread-safe, ie. Mutex-Locked for writing. Maybe I should do one for reading as well?

My CaptureWorker, where logs are processed and gathered until and update signal arrives:

void RT_CaptureWorker::signalLogsReady()
{
	if(!entries.isEmpty())
	{
		int sleepTime = modell.updateInterval() / entries.keys().count();
		for (const auto & key : entries.keys()) {
			emit logsProcessed(key, *entries.constFind(key));
			QThread::msleep(sleepTime);
		}
		entries.clear();
	}
}

void RT_CaptureWorker::handleLog(const QString & clientName, const QString & groupName, const REP::PROTO::LogMsg & msg)
{
	if (entries.contains(clientName))
	{
		entries[clientName].append(CommonLogUtils::prepareLogEntry(clientName, groupName, msg));
	}
	else
	{
		entries.insert(clientName, QList<QList<QString>>{CommonLogUtils::prepareLogEntry(clientName, groupName, msg)});
	}
}

The UpdateWorker, which in given intervals, signals the CaptureWorker to update:

inline void RT_UpdateWorker::startUpdater()
{
	while(true)
	{
		emit signalUpdate();
		QThread::msleep(modell.updateInterval());
	}
}

I hope this "approach" may be an inspiration for others who struggled with a similar problem. I'm open for any further improvements!

SGaist

One other thing you might want to try: replace your 2 dimensional QList with 2 dimensional QVector.