Python list of floats to Qt c++ List how to ?

RahibeMeryem

Hi;

I am generating list of floats in python and try to save it sqlite ; and want to read that list as float list from Qt c++ ?

How can do this wth PyQt5 ?
I just want to read as float list from c++ , dont want to save string and convert back to float which is reallly time consuming.

Python:
list = [0.003002388635650277, 0.038279131054878235, -0.001260823686607182, -0.011782282963395119, -0.014349400997161865, 0.05877083167433739, -0.03566373884677887, -0.006537315901368856, 0.03893105685710907]

Want to save above to the Sqlite as ie. blob . and read from c++ as float list etc.

JonB

@RahibeMeryem
To be clear: you have one Qt program written in Python/PyQt5, from which you want to save to database, and then you have another Qt(?) program written in C++, from which you want to read back? They are not both Qt/Python/PyQt5 programs?

Anyway. If you really "Want to save above to the Sqlite as ie. blob" you had better find out how this "blob" gets saved to SQLite, because a "blob" usually means you will lose the fact that this started as a list of floats and perhaps save some binary byte sequence. So it won't be saved as a "float list" and won't come back as that. Unless you know better about SQLite blobs than I (not a SQLite user).

Separately, those "floats" you show seem to me to have quite a lot of digits even for a C++ double. Maybe I'm wrong, I can only assume that Python "floats" have the same representation/precision as C++ doubles.

RahibeMeryem

@JonB yes one program in Python with PyQt5 . other program is in Qt5 C++.

These floats is from deep-learning matrix . Constantly creating in Python and need to be transferred to the C++ as fast as possible because there is millions of float matrix (numpy) each has 1024 float numbers as above.

I am currently saving as json string to the sqlite and reading from C++ , splitting , converting to float and finally it is end up with EigenMatrix. It is really slow. 1million rows took 10min in C++ (split "," & convert each string to float and create Eigen.

when I proifle the %95 part is string split and float from string.

So I need more native method to transfer these floats to the C++ (want to use Sqlite as storage)

Best

JonB

@RahibeMeryem
Yes, converting floats to string will be slow, and even slower converting string back to float. Not to mention, you are in danger of losing precision and getting not quite the same value float back after the conversion float->string->float.

I don't know if you have to do this via an external database/file. If the two programs run at the same time, you would be better sending the output from the Python program direct to the C++ program. But I will assume that's how it has to work.

Then if speed is critical I would skip any JSON and any strings. Save the floats into a consecutive memory array/byte-buffer --- I don't know how you do that from Python, but I'm sure it's doable --- and save that as the "blob". It will be laid out as something like 1024 x 8 bytes for each number. Then do the inverse from C++, reading the whole blob into a byte buffer and treat it as an array of 8-byte floats (doubles) to read back. No conversion, a million times faster --- minus whatever time to write to/read from the database.

RahibeMeryem

Hİ @JonB

I can write from python:

        array = (ctypes.c_float * len(feature_list))(*feature_list)

feature_list is a python list holds floats. list has 1024 floats.

reading from the sqlite in c++:

    auto dat = query->value (1) ;
    double out[512];
    QByteArray outByteArray = dat.toByteArray();
    auto outch = outByteArray.data ();
    memcpy (&out , outch ,sizeof(out));

is not give the same result :(

JonB

@RahibeMeryem
You cannot mix your types like this. What you write from the Python size determines what you can read at the C++ side.

It looks like your Python code (correctly?) creates an array of bytes. Make sure it's of the right size. I think for a start you have picked the wrong type, but it's for you to examine. Your Python type is ctypes.c_float. That is C++ type float, 4 bytes. I have already told you that will need C++ type double, 8 bytes, to maintain precision.

What you seem to do is send 1024 * 4-byte-floats and receive as 512 * 8-byte-doubles. The sizes might match, but the contents will be wrong.

Start by sorting that out. It might help to send just one number from Python and get the code right for receiving that one number correctly at the C++ side.

SGaist

Hi,

Are you only using numpy in your Python application or other stuff as well ?

RahibeMeryem

@JonB @SGaist mainly using numpy. really pissed f this python float to C++ float :)

i have in some points:
converting numpy to list and than :

array = (ctypes.c_double * len(feature_list))(*feature_list)
 data_array = array('d', feature_list)

saving to sqlite () for data_array converting to the bytes())

first hundered looks good conversion but than c++ rounding or misinterpret.

here c++

    auto dat = query->value (1) ; (BLOB saved floats/doubles)
    QByteArray outByteArray = dat.toByteArray();
    double out_double[512];

    char *outch = outByteArray.data ();
    memcpy (&out_double , outch ,512);

and for next to assign floats to the eigen float.

if I use float in out_double its rounding numbers . I want exact copy of the python floats which is a numpy matrix.

JonB

@RahibeMeryem said in Python list of floats to Qt c++ List how to ?:

memcpy (&out_double , outch ,512);

Oh dear! Look at definition of memcpy(), that only copies 512 bytes == 64 8-byte doubles! If you are going to use these routines, please take the time to read their documentation carefully!

RahibeMeryem

How did I missed thiss....

its ruined my 2 hours...

memcpy (&out_double , outch ,512);

this is working for first 60s double.. and the rest is problem... so when debugging looks fine :)

so correct one is :

memcpy (&out_double , outch ,4096);

Again. again.. you saved my day.

if you looking the wrong place for problem never find the solution.

thanks again.

JonB

@RahibeMeryem
Yep :) Use at least sizeof(double) * 512 instead of 4096, so that you can pair that with the Python sender's ctypes.c_double * ... :)

If the sender does not always send 512 doubles of actual data (e.g. maybe it's short at the end?), you may need to protect yourself against reading too much data. E.g. int num_doubles = outByteArray.size() / sizeof(double).

SGaist

Out of curiosity, isn't there a C/C++ API that you can leverage to use numpy directly from C++ ?