QProcess for a Sudo, dd Command and a Pipe for md5sum

BitFlipper

Hello,
I am trying to do a block copy and md5sum for only what was copied in a QProcess.
I need to execute something like this:
sudo dd if=/dev/sda2 of=/dev/sdb2 bs=4096 conv=noerror | md5sum
in a Qprocess. The sda2 and sdb2 are passed in as Qstrings.
I have this working:
"
auto myCopy = [&](QString source_path, QString destination_path, int blk_size, int end_block)
{
QString blk_size_par = QString::number(blk_size);
QString end_blk_par = QString::number(end_block);
blk_size_par.prepend("bs=");
end_blk_par.prepend("count=");
source_path.prepend("if=");
destination_path.prepend("of=");
QProcess *process = new QProcess;
QStringList arguments;
//sudo dd if=/dev/sda2 of=/dev/sdb2 bs=4096 conv=noerror | md5sum
arguments << "dd" << source_path << destination_path << end_blk_par << blk_size_par << "conv=noerror" << "status=progress";
// Appending this does not work<< "|" << "| md5sum";
process->startDetached("sudo",arguments);
"
Please excuse the poor coding convention I am at the end of rope on this one. Any help is appreciated Thanks! Karma++ for help!

jsulm

@BitFlipper said in QProcess for a Sudo, dd Command and a Pipe for md5sum:

I have this working

I'm not sure what the question is then?
sudo will ask for password, you will need to do a bit more here: read the stdout from the process, detect the password prompt, and send the password to the process via its stdin.

JonB

@BitFlipper
You have several problems here.

Why have you gone for QProcess::startDetached() rather than QProcess::start()? I keep seeing people do this here without them understanding why/what the consequences are. If you have a good reason that may be fine, but you are going to have a problem with startDetached() if the sudo needs to prompt for a password, which leads us to...
If your sudo needs a password it's going to want to prompt for it. With startDetached() if you're lucky it will recognise you can't be prompted for one and will gracefully error, if you're unlucky it will block the command waiting for a password till Hell freezes over....
If you use start() instead and this is a Qt console application you may be able to be prompted for the password. Do a man sudo. For example, perhaps -A or -S will help in your situation.
Assuming you get past the sudo password issue. Your problem is that the OS does not recognise the | symbol in a command, it is the shell which deals with piping when it parses and executes the line. You have two choices:

a. Send the complete line including the pipe to the shell. This means it is the shell which will be the executable, not the sudo. You will have to run something like:

/bin/sh -c "<complete command line>"

so:

QString shellCommandLine = "sudo dd if=/dev/sda2 of=/dev/sdb2 bs=4096 conv=noerror | md5sum";
arguments << "-c" << shellCommandLine;
process->start("/bin/sh", arguments);

Put all your arguments into that shellCommandLine as desired. Note that it's a single string. Be careful about what needs quoting, you need to understand shell quoting rules if you have any "funny" characters in it. Practice it at the terminal (via /bin/sh -c "..." ) till you get it right.

Or b.

Create separate QProcess instances for each side of the | and do the pipe connection yourself using http://doc.qt.io/qt-5/qprocess.html#setStandardOutputProcess, as per the example there.

Finally, on a separate matter, what's the whole thing of

dd if=... of=... | md5sum

all about? You've specified an output file for the dd, where does dd write anything on its standard output which is for input to md5sum?? Your dd writes nothing to stdout (it all goes to /dev/sdb2), at best it writes a message to stderr (which you're not capturing), is that what you want to pass to md5sum??

BitFlipper

@jsulm This works refers to arguments of:
arguments << "dd" << source_path << destination_path << end_blk_par << blk_size_par << "conv=noerror" << "status=progress";
I can not get the additional pipe for md5sum to work, hence:
/ Appending this does not work<< "|" << "| md5sum";
Just to clarify I am not having issues with sudo, no password is prompted.
Thanks!

BitFlipper

@JonB
Thanks for the response

I am moving big data, I fire these off in their own threads the start detached is so I can start a lot of processes when I want and just leave them spin until they are done. I have set up certain commands to have rights sudo in just this specific instance does not need a password. This is used in a standalone embedded system with no networking.
No password needed, sudo is not the issue.
Again same topic
kicking dead horses
a) Ok now we are getting to the issue at hand. So my command changes in this function, should I build a custom string every time and pass it in as one statement? I always practice in terminal first, and do dry runs that print out what will run in dd because if you don't you will sooner or latter wipe the hdd with the OS with dd cmds lol. I will try this and report back.

b) I tried that already with no luck

Haha! the final note. I am block copying big data and I am doing it fast. I need a sum token on what I copy so later after I flush buffers I can read everything back and verify the hardware wrote and can read just those blocks specifically. First call is:
dd from source to destination of these blocks | md5.. I get a token from this during the copy, later I do the same thing but:
dd source(old destination) of these blocks that basically just reads what I just wrote in a stream and I pipe it through md5sum to get a token then I check them against each other. There is a whole thread management going on in the back I did not show. But What I am doing is asynchronously reading writing and verifying multiple data sources to multiple other data destinations in the GBs per second regime.
QT was used just for the pretty GUI, I question if this was the correct framework, it's been painful doing low level and process stuff on.
Thanks for your response Karma++;

Christian Ehrlicher

dd if=... of=... | md5sum

this will not calculate any useful md5sum. It will just calculate the md5sum of the output of dd's stdout which is for sure not what you want...

BitFlipper

@Christian-Ehrlicher I think I see what you are saying. This will not give me a md5sum for the blocks I copy with dd, but instead a sum for the stdout for the dd. So in my case the result information from the dd will be getting summed. How do I get a sum for just specific blocks? Good catch!!! Karma++!!

JonB

@BitFlipper
b) I tried that already with no luck

Then you did not try right! There is an example of just what you need at the link I posted. The outline of this approach would be:

QProcess ddProcess, md5sumProcess;
ddProcess.setStandardOutputProcess(&md5sumProcess);
ddProcess.start("sudo dd if=/dev/sda2 of=/dev/sdb2 bs=4096 conv=noerror");
md5sumProcess.start("md5sum");
md5sumProcess.waitForFinished();

However, it's probably more suitable for you to issue the whole lot as a single string passed to /bin/sh or /bin/bash with the -c argument, and let it figure the | for you., as per my example earlier.

So my command changes in this function, should I build a custom string every time and pass it in as one statement?

Yes.

I always practice in terminal first

For this purpose make yourself use /bin/sh -c "sudo dd ... | md5sum" as that is what you will need. Be careful if anything in your command requires its own quoting (your current example does not), as the whole command itself is now inside quotes.

As I wrote earlier, your md5sum is not going to see the contents from the dd. Your example does not lend itself to checksumming as it uses a single dd if=... of=... which does the input & output in one go. What you want is for the output from the dd to go both to the output file and to md5sum. Here are two possibilities for you to play with:

sudo dd if=/dev/sda2 bs=4096 conv=noerror | sudo tee /dev/sdb2 | md5sum

sudo dd if=/dev/sda2 bs=4096 conv=noerror | tee >(md5sum 1>&2) | sudo dd of=/dev/sdb2 bs=4096

In the first case we tee the output off to /dev/sdb2 as well as letting it through to md5sum. Simpl-ish, but you lose the ability to specify the bs= for the output to /dev/sdb2. I don't know if that matters to you.

In the second case you use "shell magic" (you'll probably have to use /bin/bash not /bin/sh, I think) to send tee's output to md5sum process as well as passing it onto a second dd to do the output. I have made it so md5sum's output goes to standard error instead of standard output.

Finally: is all this dd and checksum stuff worth it? Probably not. Have a read of, say, https://unix.stackexchange.com/a/45854/104736 for alternative suggestions.