Hashing ... !

Abbas Naghdi

Hi
I need the files to a folder using the code below Hash
But the speed is very low and it is my software crashes
How fast do go up and my program does not crash?
Thanks ...

Code :

@ QString strdir = "C://Scan";
QDirIterator dir(strdir, QDirIterator::Subdirectories);

while(dir.hasNext()) {
QCryptographicHash crypto(QCryptographicHash::Md5);
QFile file(dir.next());
file.open(QFile::ReadOnly);
while(!file.atEnd()){
crypto.addData(file.read(1));
}

QByteArray hash = crypto.result();
QString strsw = hash.toHex();@

Code_ReaQtor

I see that you have some heavy processes which makes the GUI "freeze" (slow down, not responding, etc.). Try to move those processes in a QThread. :)

Abbas Naghdi

[quote author="Code_ReaQtor" date="1364742265"]I see that you have some heavy processes which makes the GUI "freeze" (slow down, not responding, etc.). Try to move those processes in a QThread. :)[/quote]

Hello again and thank you
I do not know how to code with QThread to Hash
Please help me get more ...
I really need some help ... Thank you

SGaist

Hi,

What does the crash says ?

Problems i can see:
You do not check whether the next entry is a file or a directory or something else.
You are reading the files one byte at a time, that won't be very efficient.

MuldeR

[quote author="Abbas Naghdi" date="1364762846"]
[quote author="Code_ReaQtor" date="1364742265"]I see that you have some heavy processes which makes the GUI "freeze" (slow down, not responding, etc.). Try to move those processes in a QThread. :)[/quote]

Hello again and thank you
I do not know how to code with QThread to Hash
Please help me get more ...
I really need some help ... Thank you[/quote]

With <QtConcurrentRun> you can do something like:
@QString someLengthyCalculation(const QString &fileName)
{
QCryptographicHash crypto(QCryptographicHash::Md5);
QFile file(fileName)
if(file.open(QFile::ReadOnly))
{
crypto.addData(file.readAll());
}
QByteArray hash = crypto.result();
return hash.toHex();
}

main()
{
/begin calculation in background, does NOT block/
QFuture<QString> future = QtConcurrent::run(someLengthyCalculation, fileName);

/*do some other work here*/

/*get final result, will BLOCK until result is available*/
QString result = future.result();

}@

If you have multiple files you need to hash, it should be easy to build a loop around the calls so that always N hashes are calculated in parallel. Usually N will be chosen as QThread::idealThreadCount().

ChrisW67

Start by not reading the file byte-at-a-time. For a 1MB file you make 1 million calls to read() when a much smaller number of reads using a large buffer is called for.

In any case, you should just pass the opened file directly to QCryptographicHash::addData(QIODevice * device) and let it be efficient reading the file.

I am also unsure how a single hash of all the files and directories in a tree, in indeterminate order, will be useful.

Abbas Naghdi

Hi again and thanks to all
I need to make a Directory Hash files
but if the size of the files is large or the number of files is too much software crash.
What do I do to my software does not crash?
I'm using the following code ... Please fix this code for me ...
Thanks to all ...

@First Code :

QDirIterator dir("E://Unknown//Scan Files",QDirIterator::Subdirectories);
while(dir.hasNext()) {
QFile file(dir.next());
if(file.open(QIODevice::ReadOnly)) {
QByteArray fileData = file.readAll();
QByteArray hashData1 = QCryptographicHash::hash(fileData, QCryptographicHash::Md5);
QByteArray hashData2 = QCryptographicHash::hash(fileData, QCryptographicHash::Sha1);
ui->LV_Found->addItem(dir.next());
ui->LV_NFound->addItem(hashData1.toHex());
ui->LV_NFound->addItem(hashData2.toHex());
file.close();
}}}

Second Code (Speed) :

QDirIterator dir("E://Unknown//Scan Files",QDirIterator::Subdirectories);
QCryptographicHash crypto1(QCryptographicHash::Md5);
QCryptographicHash crypto2(QCryptographicHash::Sha1);
while(dir.hasNext()) {
QFile file(dir.next());
if(file.open(QFile::ReadOnly)) {
crypto1.addData(file.readAll());
crypto2.addData(file.readAll());
}
QByteArray hash1 = crypto1.result();
QByteArray hash2 = crypto2.result();
ui->LV_Found->addItem(dir.next());
ui->LV_NFound->addItem(hash1.toHex());
ui->LV_NFound->addItem(hash2.toHex());
file.close();
}}@

MuldeR

Do you really want to make a single hash for all files in the directory ???
If so (which is kind of unusual), then note that the hash value will depend on the order you process these files, i.e. the order in which you pass the file contents into addData(). But be aware that files in a directory do NOT have a specific order. I don't think that the QDirIterator will ensure a specific ordering!
readAll() obviously is a bad idea, for big files, because it reads the whole file into memory. Instead, read the file iteratively. But not byte-by-byte! Use something like 1 MB chunks, for example.
Two times readAll() on the same QFile object probably gives EMPTY result in the second call. This probably means that "crypto2" in your code will be meaningless, as it will always give the hash for an empty input!
If you really want to make a single hash for all files, then parallelizing the calculations will not be possible.

Abbas Naghdi

[quote author="MuldeR" date="1364841718"]1. Do you really want to make a single hash for all files in the directory ???

If so (which is kind of unusual), then note that the hash value will depend on the order you process these files, i.e. the order in which you pass the file contents into addData(). But be aware that files in a directory do NOT have a specific order. I don't think that the QDirIterator will ensure a specific ordering!
readAll() obviously is a bad idea, for big files, because it reads the whole file into memory. Instead, read the file iteratively. But not byte-by-byte! Use something like 1 MB chunks, for example.
Two times readAll() on the same QFile object probably gives EMPTY result in the second call. This probably means that "crypto2" in your code will be meaningless, as it will always give the hash for an empty input!
If you really want to make a single hash for all files, then parallelizing the calculations will not be possible.[/quote]

Thank you
I want all files of a Directory Hash do
What do you think?

I changed the code :

@ crypto1.addData(file.read(1));
crypto2.addData(file.read(1));@

MuldeR

Now you read one byte and feed it into "crypto1" only, then you read the next byte and feed it into "crypto2" only. And so on. That's probably not what you want, as one hash only hashes the odd bytes and the other hash only hashes the even bytes! It's very slow too, as you read only one byte at a time! Better try something like:
@static const qint64 CHUNK_SIZE = 1024*1024;

QFile file(dir.next());
if(file.open(QFile::ReadOnly))
{
forever
{
QByte chunk = file.read(CHUNK_SIZE); //reads up to 1 MB in a chunk
if(!chunk.isEmpty())
{
crypto1.addData(chunk);
crypto2.addData(chunk);
continue;
}
break;
}
}@

And again: Keep in mind that the hash value will depend on the order in which you process the files !!!

Abbas Naghdi

Hi ... Again my software crash.
I have 3,000 files with size less than 10 KB.
When I do this the file Hash 3000 software I crash. ... !
Maybe this is the problem of the roof inast 3000 File Hash. ... !
Maybe with a Hash of the file in a specified time problem solved. ... !
You can help Me?

MuldeR

Hash functions can easily process VERY LARGE amount of data (e.g. multiple gigabytes), so that is not the problem. Also the hash function doesn't care whether the data came from 3000 files of 10 KB size or from a single file a single 30.000 KB file. So there must be some bug (e.g. memory leak or access violation) in your code.

So, where exactly does it crash? And what is the crash ??? (or at least post some more code!)