Loop trough files and replace variable values between quotes

WD30EFRX

Hi all,

i'm strugling for a few days now to solve this issue;

I need to loop trough several files and find fixed strings, while their value is variable.
For example:

'this is a fixed string and my value is ("0x0000000") forever;'

So, i need a way or function to read the 0x0000000 and replace it with another (fixed) value depending on the first part of the string
.
One file can contain up to 50 strings i need to replace, so i'm looking for the most optimized solution to handle this.

So far, i didn't get past simple find/replace and some regex, but it's really bad coding and low performance.

I hope you're expertise can help me out!

Thank you

mrjj

Hi and welcome
If people are to suggest a good way to do it,
we might need some extra info

How big are are files?

Can you always load them into memory and replace ?

Also, are all values like "0x0000000" with different lengths
"0x00", "0x00000" ?
also
"depending on the first part of the string"
In what ways?
can you list 3 different cases and what trigger the correct replacement ?

WD30EFRX

@mrjj

Hi!

3 different examples of the text to replace between quotes:

static const uint256 randomBlock("0x000006ad592d2fde995b3bdc89affa684bcd3310febd09e245cb0b41092e35a3");
const char* pszTimestamp = "This will become some random scraped news headline";
uint256("0xa980d2ab0cee949d3ac21f7af8d99ea47e13699992ad7be8320f077e906c7507"));

So i really look for some way to identify the start of the string, for example "static const uint256 randomBlock" and replace the variables.

The files are sourcecodes, and max out around 5000 lines.

mrjj

@WD30EFRX
ahh, it's inside source files.
Well for the 0x0 ones, then
"0x
should be enough.
pszTimestamp is a bit worse.
Can we see what you have tried so far as not to suggest the same?

WD30EFRX

@mrjj

So far i've used the following code

QStack<QString> dirs;
    dirs.push(QString(argv[3]));

    // continue if there are sub directory
    while(dirs.size()){
        QDir currentDir(dirs.pop());
        QStringList child =  currentDir.entryList(filters);

        QStringList::const_iterator itbegin = child.begin();
        QStringList::const_iterator itend = child.end();
        for(; itbegin != itend ; ++itbegin){
            const QString nextFile(currentDir.path() + "/" + (*itbegin));
            //std::cout << "Open " << (const char*)nextFile.toLatin1() << std::endl;

            // Open het bestand en plaats inhoud in een string
            QString oldContent;
            {
                QFile inFile(nextFile);
                if(!inFile.open(QIODevice::ReadOnly)){
                    std::cout << "Cannot open " << (const char*)nextFile.toLatin1() << "\n";

                }
                QTextStream inStream(&inFile);
                oldContent = inStream.readAll();
                inFile.close();
            }

            // Is there anything to change? - Geen match
            if( oldContent.indexOf(oldText) == -1){
                   // std::cout << "Nothing to do in " << (const char*)nextFile.toLatin1() << "\n";
            }
            else{
                QString newContent = oldContent.replace(oldText,newText);
                {
                    QFile inFile(nextFile);
                    if(!inFile.open(QIODevice::WriteOnly)){
                        std::cout << "Cannot open to write " << (const char*)nextFile.toLatin1() << "\n";

                    }

                    QTextStream outStream(&inFile);
                    outStream << newContent; // schrijf naar bestand
                    inFile.close();
                    std::cout << "Source changed in " << (const char*)nextFile.toLatin1() << "\n";

                }
            }
        }


        // Get subdirectories and files
        QStringList childdir =  currentDir.entryList(QDir::NoDotAndDotDot | QDir::Dirs);
        QStringList::const_iterator itbegindir = childdir.begin();
        QStringList::const_iterator itenddir = childdir.end();
        for(; itbegindir != itenddir ; ++itbegindir){
            dirs.push(currentDir.path() + "/" + (*itbegindir));
            //std::cout << "Add " << (const char*)(currentDir.path() + "/" + (*itbegindir)).toLatin1() << std::endl;
        }
    }

While this does the job, i really need to re-loop every file for every line i have to replace.
So i'm looking for something cleaner and more performant.

mrjj

Hmm, dont seem really bad and u use readall so
relooping the in memory file should be pretty fast?

How do you recognize pszTimestamp cases?

WD30EFRX

@mrjj

Not yet. Still looking for a way to extract and replace the values that are inside quotes after the first known part of the string.
If i had such function, it would be easy to exchange all variables from the partially known line

Hence my question :)

mrjj

@WD30EFRX said:
Well, its a nice problem :)
Is this something you must do many times since u care about the performance?

will it always start with "const char* " for those pszTimestamp cases?

SGaist

Hi and welcome to devnet,

You provided only input examples. Can you also provide a complete before/after example ?

WD30EFRX

@SGaist

Hi!

The output variables are generated by an external algortithm much like Bitcoin.
So new hash values and variables need to get 'computed' first and are all related to each other.

So the output is truly random, the only part that i know for sure are the leading string that identify them in the sources.

So to shorten my request, what i need is a nice, optimized function to :

Open the file
Match a whole line by the leading string
Split is somewhere after the leading string and replace the variables OR replace with a regex between the quotes
Write back to file

My current code opens every file multiple times for each new string i need to replace.
So optimized, it would open 1 file and look for ALL the predefined strings i need to replace.

Hope this helps :)

Thank you

SGaist

Do you know all the strings you have to search before opening the file ?

WD30EFRX

@SGaist

Yes i do.

Thank you

SGaist

Then I'd build something like a vector containing the string/regexp to search along with their replacement and loop on that over a buffer containing your file data. Then you wouldn't need to open every file several times.

WD30EFRX

@SGaist

Thank you.

Right now i'm looking to replace the fixed lines using their known line numbers, split the strings, extract the values and join them back.

This seems to work. However, i have several multiline codeblocks that needs replacement between {}, and now i'm stucked with those. :)

mrjj

Hi
several multiline codeblocks ?
That is a new type it seems?

WD30EFRX

@mrjj

As example:

int64_t DoSomeFunctionHere(int64_t param1, int64_t param2)
{
    Some line of code here
    Some line of code here
    return Param1 + Param2;
}

So i need to look for DoSomeFunctionHere and replace everything that follows inside the { and }

mrjj

Ok, so that is still "0x0" type
for "Some line of code here"
or new case?

WD30EFRX

@mrjj

It's combined. But i think i'll look for someone to hire to complete my project since my deadline is very near :)

mrjj

@WD30EFRX
Ok that could be an option too,
If u use something like
https://www.freelancer.com

Make sure to describe the problem very carefully with
actual samples of what to find and what to replace with and the
different cases. Also what " first known part of the string." is. and so on.
Also, if the files are in subfolder etc.

Else you wont get a good estimate of the cost.