Loop trough files and replace variable values between quotes



  • Hi all,

    i'm strugling for a few days now to solve this issue;

    I need to loop trough several files and find fixed strings, while their value is variable.
    For example:

    'this is a fixed string and my value is ("0x0000000") forever;'

    So, i need a way or function to read the 0x0000000 and replace it with another (fixed) value depending on the first part of the string
    .
    One file can contain up to 50 strings i need to replace, so i'm looking for the most optimized solution to handle this.

    So far, i didn't get past simple find/replace and some regex, but it's really bad coding and low performance.

    I hope you're expertise can help me out!

    Thank you


  • Qt Champions 2016

    Hi and welcome
    If people are to suggest a good way to do it,
    we might need some extra info

    How big are are files?

    Can you always load them into memory and replace ?

    Also, are all values like "0x0000000" with different lengths
    "0x00", "0x00000" ?
    also
    "depending on the first part of the string"
    In what ways?
    can you list 3 different cases and what trigger the correct replacement ?



  • @mrjj

    Hi!

    3 different examples of the text to replace between quotes:

    static const uint256 randomBlock("0x000006ad592d2fde995b3bdc89affa684bcd3310febd09e245cb0b41092e35a3");
    const char* pszTimestamp = "This will become some random scraped news headline";
    uint256("0xa980d2ab0cee949d3ac21f7af8d99ea47e13699992ad7be8320f077e906c7507"));

    So i really look for some way to identify the start of the string, for example "static const uint256 randomBlock" and replace the variables.

    The files are sourcecodes, and max out around 5000 lines.


  • Qt Champions 2016

    @WD30EFRX
    ahh, it's inside source files.
    Well for the 0x0 ones, then
    "0x
    should be enough.
    pszTimestamp is a bit worse.
    Can we see what you have tried so far as not to suggest the same?



  • @mrjj

    So far i've used the following code

    QStack<QString> dirs;
        dirs.push(QString(argv[3]));
    
        // continue if there are sub directory
        while(dirs.size()){
            QDir currentDir(dirs.pop());
            QStringList child =  currentDir.entryList(filters);
    
            QStringList::const_iterator itbegin = child.begin();
            QStringList::const_iterator itend = child.end();
            for(; itbegin != itend ; ++itbegin){
                const QString nextFile(currentDir.path() + "/" + (*itbegin));
                //std::cout << "Open " << (const char*)nextFile.toLatin1() << std::endl;
    
                // Open het bestand en plaats inhoud in een string
                QString oldContent;
                {
                    QFile inFile(nextFile);
                    if(!inFile.open(QIODevice::ReadOnly)){
                        std::cout << "Cannot open " << (const char*)nextFile.toLatin1() << "\n";
    
                    }
                    QTextStream inStream(&inFile);
                    oldContent = inStream.readAll();
                    inFile.close();
                }
    
                // Is there anything to change? - Geen match
                if( oldContent.indexOf(oldText) == -1){
                       // std::cout << "Nothing to do in " << (const char*)nextFile.toLatin1() << "\n";
                }
                else{
                    QString newContent = oldContent.replace(oldText,newText);
                    {
                        QFile inFile(nextFile);
                        if(!inFile.open(QIODevice::WriteOnly)){
                            std::cout << "Cannot open to write " << (const char*)nextFile.toLatin1() << "\n";
    
                        }
    
                        QTextStream outStream(&inFile);
                        outStream << newContent; // schrijf naar bestand
                        inFile.close();
                        std::cout << "Source changed in " << (const char*)nextFile.toLatin1() << "\n";
    
                    }
                }
            }
    
    
            // Get subdirectories and files
            QStringList childdir =  currentDir.entryList(QDir::NoDotAndDotDot | QDir::Dirs);
            QStringList::const_iterator itbegindir = childdir.begin();
            QStringList::const_iterator itenddir = childdir.end();
            for(; itbegindir != itenddir ; ++itbegindir){
                dirs.push(currentDir.path() + "/" + (*itbegindir));
                //std::cout << "Add " << (const char*)(currentDir.path() + "/" + (*itbegindir)).toLatin1() << std::endl;
            }
        }
    

    While this does the job, i really need to re-loop every file for every line i have to replace.
    So i'm looking for something cleaner and more performant.


  • Qt Champions 2016

    Hmm, dont seem really bad and u use readall so
    relooping the in memory file should be pretty fast?

    How do you recognize pszTimestamp cases?



  • @mrjj

    Not yet. Still looking for a way to extract and replace the values that are inside quotes after the first known part of the string.
    If i had such function, it would be easy to exchange all variables from the partially known line

    Hence my question :)


  • Qt Champions 2016

    @WD30EFRX said:
    Well, its a nice problem :)
    Is this something you must do many times since u care about the performance?

    will it always start with "const char* " for those pszTimestamp cases?


  • Lifetime Qt Champion

    Hi and welcome to devnet,

    You provided only input examples. Can you also provide a complete before/after example ?



  • @SGaist

    Hi!

    The output variables are generated by an external algortithm much like Bitcoin.
    So new hash values and variables need to get 'computed' first and are all related to each other.

    So the output is truly random, the only part that i know for sure are the leading string that identify them in the sources.

    So to shorten my request, what i need is a nice, optimized function to :

    • Open the file
    • Match a whole line by the leading string
    • Split is somewhere after the leading string and replace the variables OR replace with a regex between the quotes
    • Write back to file

    My current code opens every file multiple times for each new string i need to replace.
    So optimized, it would open 1 file and look for ALL the predefined strings i need to replace.

    Hope this helps :)

    Thank you


  • Lifetime Qt Champion

    Do you know all the strings you have to search before opening the file ?



  • @SGaist

    Yes i do.

    Thank you


  • Lifetime Qt Champion

    Then I'd build something like a vector containing the string/regexp to search along with their replacement and loop on that over a buffer containing your file data. Then you wouldn't need to open every file several times.



  • @SGaist

    Thank you.

    Right now i'm looking to replace the fixed lines using their known line numbers, split the strings, extract the values and join them back.

    This seems to work. However, i have several multiline codeblocks that needs replacement between {}, and now i'm stucked with those. :)


  • Qt Champions 2016

    Hi
    several multiline codeblocks ?
    That is a new type it seems?



  • @mrjj

    As example:

    int64_t DoSomeFunctionHere(int64_t param1, int64_t param2)
    {
        Some line of code here
        Some line of code here
        return Param1 + Param2;
    }
    

    So i need to look for DoSomeFunctionHere and replace everything that follows inside the { and }


  • Qt Champions 2016

    Ok, so that is still "0x0" type
    for "Some line of code here"
    or new case?



  • @mrjj

    It's combined. But i think i'll look for someone to hire to complete my project since my deadline is very near :)


  • Qt Champions 2016

    @WD30EFRX
    Ok that could be an option too,
    If u use something like
    https://www.freelancer.com

    Make sure to describe the problem very carefully with
    actual samples of what to find and what to replace with and the
    different cases. Also what " first known part of the string." is. and so on.
    Also, if the files are in subfolder etc.

    Else you wont get a good estimate of the cost.


Log in to reply
 

Looks like your connection to Qt Forum was lost, please wait while we try to reconnect.