Exchange data between threads... typical question :D
-
Hello guys, :-)
I'm reading a few examples about threads and trying to understand the way data is exchanged between threads. Because everytime I exchange data between my threads my program crashes. The example I'm reading is the following:
http://doc.qt.nokia.com/4.7/threads-queuedcustomtype.html
So my questions are:
- From what I saw, any data one would like to exchange between threads must be meta-objects, and data has to be send through signals and slots. Is that true?
- Is this the only way to exchange data between main thread and subthreads?
- I'm creating a program that may read hundreds of MBs of data from files. So I would like to do the file reading operations in a thread. The data when loaded is saved in deque containers. Should I insert all these deque containers into a meta-object to be able to exchange it with the class?
- One last stupid question. Does calling "QThread::start()" execute the function QThread::run() just once? or does it keep looping the function run() till I call the QThread::quit()? I couldn't understand what they said in the class reference.
Thank you! :-)
[EDIT: list formatting, please use '*' instead of '-', Volker]
-
[quote author="TheDestroyer" date="1306352766"]
-From what I saw, any data one would like to exchange between threads must be meta-objects, and data has to be send through signals and slots. Is that true?
[/quote]Not entirely. You have a few options.- Signal/Slot connection ideal for notifications and small data sets imo.
- "QMetaObject::invokeMethod()":http://doc.trolltech.com/latest/qmetaobject.html#invokeMethod, does something like a signal/slot connection, but doesn't require you to trigger a signal.
- Locking (QMutex with QMutexLocker, or QReadWriteLock with QReadLocker/QWriteLocker)
Depending on the precise type of communication between threads either of the above may be best applicable. There may even be options that I didn't mention here.
[quote]-Is this the only way to exchange data between main thread and subthreads?
[/quote]That would be a "no" then.
[quote]-I'm creating a program that may read hundreds of MBs of data from files. So I would like to do the file reading operations in a thread. The data when loaded is saved in deque containers. Should I insert all these deque containers into a meta-object to be able to exchange it with the class?
[/quote]Since std::deque is not implicitly shared, you might want to consider locking. Even so, you might be able to pull this off using QtConcurrent instead of an explicit thread implementation, but again, that is entirely up to you and dependent on what you want to achieve (the bigger picture).[quote]-One last stupid question. Does calling "QThread::start()" execute the function QThread::run() just once? or does it keep looping the function run() till I call the QThread::quit()? I couldn't understand what they said in the class reference.[/quote]QThread::start() triggers exactly one execution of the run() function. If you use the default thread implementation, an event loop is started in the run() function, which makes it run forever. If you overload the run() function, it is dependent on your implementation whether you will be looping or running once.
-
Thank you so much! I'll try what you told me and tell you how it went :-)
Would you know if I post anything here?
-
If you're going to use threads and signals & slots (or just QObjects, for that matter), don't forget to study "this":http://developer.qt.nokia.com/wiki/ThreadsEventsQObjects wiki entry first. Also, spend time thinking how you might be able to limit the communication between threads to an absolute minimum. Threads are most effective if they can run independently. If you need lots of locking, you're not on the right track.
-
Thank you. I'll read it right now :)
-
It seems to be OK to emit an object from inside the thread out to the main thread. But how can I send data from the main thread to the thread? I tried copying a meta-object to the thread but didn't work!!
-
ah! yes I mean metatype!! :$
I'll put most of the code here, because I'm a beginner with this, and I'm going crazy not knowing how to fix this correctly! The program has worked! but I still have some problems.
The program is simply for downsampling. So it lowers the filesize within some ratio.
The problem now is, if the file size is huge, the program freezes!
and the other problem is, I want to signal the object "data" by reference! is this possible?here's the most of the code:
Thread code:
@
void downSamplingThread::run()
{
passedData data;
data = dataPassed;
downSample(data.inputFile,data.outputFile,data.dataUnitSize,data.downSamplingRate, data);}
qint16 downSamplingThread::downSample(QString inputFile, QString outputFile, quint64 dataUnitSize, quint64 downSamplingRate, passedData &data)
{
double progressValue = 0;
int progressLimit = 0;
qint64 fileSize;
std::fstream fileReader(inputFile.toStdString().c_str(),std::ios::in | std::ios::binary);
QFile file(inputFile.toStdString().c_str());
if(!fileReader.is_open())
{
return 1;
}
if(inputFile == outputFile && inputFile.length() != 0)
{
return 3;
}
std::fstream fileWriter(outputFile.toStdString().c_str(),std::ios::out | std::ios::binary);
if(!fileWriter.good())
{
return 2;
}
quint64 dataUnit = dataUnitSize;
quint64 downsamplingRate = downSamplingRate;
char* buffer = new char[dataUnitdownSamplingRate];
fileSize = file.size();
file.close();
while(!fileReader.eof())
{
fileReader.read(buffer,dataUnitdownsamplingRate);
if(!fileReader.eof())
fileWriter.write(buffer,dataUnit);
progressValue = (double)(100*fileReader.tellg())/fileSize;
if(progressLimit < floor(progressValue))
{
progressLimit++;
data.progress = progressLimit;
emit progressTimeout(data);
}
}
data.enableButton = 1;
data.progress = 100;
emit progressTimeout(data);
return 0;
}
@
in dialog for sending the command:@
...
connect(startButton,SIGNAL(clicked()),this,SLOT(downSampleCommand()));
connect(thread,SIGNAL(progressTimeout(passedData)),this,SLOT(updateProgress(passedData)));
...
void Dialog::downSampleCommand()
{
statusLabel->setText("Process started...");
startButton->setEnabled(0);
data.dataUnitSize = dataUnitSizeSpinbox->value();
data.inputFile = inputPathLineEdit->text();
data.outputFile = outputPathLineEdit->text();
data.downSamplingRate = downSamplingRateSpinbox->value();
data.enableButton = 0;
thread->dataPassed = this->data; //copy
thread->start();
qint16 outputReturn = 0;
//qint16 outputReturn = thread->downSample(inputPathLineEdit->text(),outputPathLineEdit->text(),dataUnitSizeSpinbox->value(),downSamplingRateSpinbox->value(),data);
if(outputReturn == 1)
{
statusLabel->setText("Error reading file!");
}
else if(outputReturn == 2)
{
statusLabel->setText("Error writing file!");
}
else if(outputReturn == 3)
{
statusLabel->setText("Input and output files are the same. Error!");
}
else if(outputReturn == 0)
{
statusLabel->setText("Done!");
}
}
void Dialog::setLastFilePath(QString path)
{
*lastFile = path;
inputFileDialog->setDirectory(path);
outputFileDialog->setDirectory(path);
}
void Dialog::updateProgress(passedData data)
{
progressBar->setValue(data.progress);
startButton->setEnabled(data.enableButton);
//QMessageBox *msg = new QMessageBox;
//msg->setText(QVariant(data.inputFile).toString());
//msg->show();
}
@metatype:
@
#include <QColor>
#include <QDebug>
#include <QMetaType>
#include <QRect>class passedData
{
public:
passedData();
~passedData();
passedData(const passedData &src);
passedData& operator=(const passedData &src);public:
QString inputFile;
QString outputFile;
quint64 dataUnitSize;
quint64 downSamplingRate;
bool enableButton;
int progress;
signals:
};Q_DECLARE_METATYPE(passedData);
#endif // PASSEDDATA_H
#include "passeddata.h"
passedData::passedData()
{
}
passedData::~passedData()
{}
passedData::passedData(const passedData &src)
{
this->dataUnitSize = src.dataUnitSize;
this->downSamplingRate = src.downSamplingRate;
this->enableButton = src.enableButton;
this->inputFile = src.inputFile;
this->outputFile = src.outputFile;
this->progress = src.progress;
}
passedData& passedData::operator =(const passedData &src)
{
this->dataUnitSize = src.dataUnitSize;
this->downSamplingRate = src.downSamplingRate;
this->enableButton = src.enableButton;
this->inputFile = src.inputFile;
this->outputFile = src.outputFile;
this->progress = src.progress;
return *this;
}
@and in main, I called
@
qRegisterMetaType<passedData>();
@The program's idea is very simple. It just reads a chunk of the file, and writes a part of the chunk again. It didn't work on a file with size 30 GB (which is a typical size we need).
Please tell me how to make this program as effective as possible. I know I'm being dependent somehow, but I have been reading about this for long time, and I'm not getting the idea!!! :(
Thank you for any efforts :-)
-
Hm, from reading this, I wonder why exactly you want to pass the data between threads? If the program will only be down sampling, your worker should be doing the down sampling and report some progress. The setup I'd then use is practically the same, but I would only be releasing the progress counter into the world, maybe a status string if it was really important.
-
Indeed, I agree with Franzk. Quoting myself:
[quote]Also, spend time thinking how you might be able to limit the communication between threads to an absolute minimum. Threads are most effective if they can run independently. If you need lots of locking, you’re not on the right track.[/quote]In this case, indeed, this would probably be only sending some progress information (and not too frequently either!)
-
Thanks for the answers guys :)... another problem is present after saving the first one :D
This program is just a sample. I'm learning, and practicing!! and actually I find it the same if I send progress or the whole data object for a very simple reason! I'm sending a reference (in the new implementation), not an object!!! I just learned how to do it!! so it's just a pointer! (and it was my purpose from the very beginning). Is it possible or legal to emit a simple variable like long int without it being a metatype rather than emitting a whole object?
The only reason I didn't send the progress value alone (which I do only 100 times during the process) is because I don't want to create another metatype class that contains the progress... so adding the progress to "data" was the fastest way. Don't you agree guys?
Actually, guys, I'm facing another problem! I'm unable to control the thread in means of starting it and closing it. For example, if the user inputs a wrong path, or does anything wrong, I want the thread to close and another one to start from the next "start" button clicked. I tried resolving this many ways. But the only way was creating a new thread on every "Start" button clicked, which implies ignoring the previous thread instantiations... because waiting, terminating, exiting or quitting it isn't working at all!!! so on every "Start" button clicked, a call "new downSamplingThread" is done, and new connections are done. But this wastes memory! how can I avoid this? how can I make sure the threads are to be deleted?
@
void Dialog::downSampleCommand()
{
thread = new downSamplingThread(this);
connect(thread,SIGNAL(progressTimeout(passedData)),this,SLOT(updateProgress(passedData)));
connect(thread,SIGNAL(sendOutputReturn(qint16)),this,SLOT(getReturnValue(qint16)));statusLabel->setText("Process started..."); startButton->setEnabled(0); data.dataUnitSize = dataUnitSizeSpinbox->value(); data.inputFile = inputPathLineEdit->text(); data.outputFile = outputPathLineEdit->text(); data.downSamplingRate = downSamplingRateSpinbox->value(); data.enableButton = 0; thread->dataPassed = this->data; //copy thread->start();}
void Dialog::updateProgress(const passedData &data)
{
progressBar->setValue(data.progress);
startButton->setEnabled(data.enableButton);
//QMessageBox *msg = new QMessageBox;
//msg->setText(QVariant(data.inputFile).toString());
//msg->show();
}
void Dialog::getReturnValue(qint16 outputReturn)
{
if(outputReturn == 1)
{
statusLabel->setText("Error reading file!");
}
else if(outputReturn == 2)
{
statusLabel->setText("Error writing file!");
}
else if(outputReturn == 3)
{
statusLabel->setText("Input and output files are the same. Error!");
}
else if(outputReturn == 0)
{
statusLabel->setText("Done!");
}
thread->terminate();
thread->quit();
thread->exit(1);
//thread->wait();
//delete thread;
startButton->setEnabled(1);
}
void downSamplingThread::run()
{
passedData data;
data = dataPassed;
downSample(data.inputFile,data.outputFile,data.dataUnitSize,data.downSamplingRate, data);}
qint16 downSamplingThread::downSample(QString inputFile, QString outputFile, quint64 dataUnitSize, quint64 downSamplingRate, passedData &data)
{
double progressValue = 0;
int progressLimit = 0;
quint64 fileSize;
QFile fileReader(inputFile);
fileReader.open(QFile::ReadOnly);
if(fileReader.error())
{
emit sendOutputReturn(1);
//this->quit();
//this->terminate();
//this->exit();
return 1;
}
QFile fileWriter(outputFile);
fileWriter.open(QFile::WriteOnly);
if(inputFile == outputFile && inputFile.length() != 0)
{
emit sendOutputReturn(3);
//this->quit();
//this->terminate();
//this->exit();
return 3;
}
if(fileWriter.error())
{
emit sendOutputReturn(2);
//this->quit();
//this->terminate();
//this->exit();
return 2;
}
quint64 dataUnit = dataUnitSize;
quint64 downsamplingRate = downSamplingRate;
char* buffer = new char[dataUnitdownSamplingRate];
fileSize = fileReader.size();
quint64 i = 0;
while(i < fileSize)
{
fileReader.read(buffer,dataUnitdownsamplingRate);
fileWriter.write(buffer,dataUnit);
progressValue = (double)(100*fileReader.pos())/fileSize;
if(progressLimit < floor(progressValue))
{
progressLimit++;
data.progress = progressLimit;
emit progressTimeout(data);
}
i += dataUnit * downsamplingRate;
}
data.enableButton = 1;
data.progress = 100;
emit progressTimeout(data);
sendOutputReturn(0);
return 0;
}@
so how can I stop this thread after failing to open the file or in general emit an error in a way that saves memory and doesn't block the main window. Cuz if I do anything like delete thread; or thread->wait()... the window freezes for some long time!!!!!
-
Btw, I used QFile in the new implementation because fstream was, for some reason, unable to read long files. Apparently "long int"'s precission is not enough to catch 30 gb... is this problem common? any idea? I'm using in the new implementation quint64, which works! Anything you think I should know about this?
-
come on guys!!! It's the last part of the problem :(
-
Sure, buddy :-)