Qt containers inadequate to x64



  • Hello folks!

    As everybody knows Qt containers(QVector, QList etc.) use int as underlying type for indexes and memory allocation. It's good enough if you're working in 32 bit. But you may have some troubles with working 64 bit application because you won't have an opportunity to store elements beyond the INT_MAX constant. Maybe your application need no such amount of memory but other may need.
    Another point: C++ encourages its developers to use size_t(or ptrdiff_t maybe) as primary type for indexes and righteous developers indulge it. But if you mix size_t with Qt containers you will have a lot of warnings(or static_casts). It makes code or build outputs ugly. Moreover you take a chance to have a subtle and annoying bugs in your future x64 release.

    According to above explanatory I have a question: Does Qt team have a plan to improve Qt containers by making them truly cross-architectural?



  • Indeed, this is a good question!

    Problem is, I don't see how you could change this while preserving binary compatibility in this version. I fear it might be something for Qt 5.0...



  • Hmm, what binary compatibility are you meaning? If you are talking about dll compatibility then I see no issue because there is no compatibility even between different two version of one vendor's compiler&linker. If you are meaning binary compatibility of serializated containers then I see no issue too, because standard Qt container have no serialize\deserialize functions at all :)
    Could you explain your point of view more detail, please?



  • Well, the Trolls guarantee binary compatibility for their libs. That means, that I should be able to replace for instance version 4.5.2 with 4.7.0 and expect my software compiled against it to work still, without having to recompile the software. Of course, this does not work across arches and is dependent on using the same compiler and linker, but those are external factors the Trolls can not influence. The question if you find it an issue, is a different one than if it would be possible to do while sticking to the current BC policy. But who knows, I am not an expert at this; perhaps I am wrong.

    Qt containers do have serialization functions, through QDataStream. And yes, there there is an issue too, but for those the version could be bumped without problems. The format changes or is extended regulary anyway.

    One way that it could be done, is to introduce duplicate versions of the containers. It is not unheard of, QRect and QPoint for instance both have an ...F version. Adding a QList64, QVector64, etc. should be possible I think.



  • Good point, thank you.
    It would be good if some insider will answer.



  • While i agree on a techincal level...

    When are you going to need INT_MAX items in the context these containers were designed for (use within the Qt toolkit).

    I think that while its an issue, it will probably wait until version 5 AT LEAST.



  • When? Well, as soon as I need to store more than 2.147.483.647 items.

    To be honest, that is not unheard of at all. I am working on a scientific application that works on big matrices and lists. Big as in perhaps bigger than 500.000 items. At one point, I need to calculate a correlation between all the items in such a list, resulting in a 500.000 x 500.000 matrix. The number of cells in that matrix is quite a bit bigger than INT_MAX. And yes, I can store those results in different data structures, but there are valid reason why people need 64bits OS-es and the address space they provide. No reason for Qt to limit these people, is there?



  • [quote author="Andre" date="1278331006"]but there are valid reason why people need 64bits OS-es and the address space they provide.[/quote]

    Can you give one that is required inside of Qt? As that is what these containers were written for, working with and in the interface of Qt. There isn't a reason for Qt to limit these people, but there was also no need for them to waste extra memory on 64 bits way back when the containers were designed for Qt 4.0.

    I'm not saying that it shouldnt happen at some point, but there isn't a rush for the Qt developers, because Qt doesn't need it.



  • You are not claiming that the Qt containers were designed to be used internally by Qt itself only, are you?

    The container classes in Qt are, AFAIK, mend to be used by developers all over the globe who use Qt to produce real-live applications. They are part of the public API, and have been since Qt 4.0 (and before). It is great that the Qt library itself does not have huge storage requirements, but it would be silly to extend that to mean that no application build using Qt has them, right?

    Qt 4.0 was introduced in 2005. While that is a long time ago, it was after the introduction of 64 bit machines to the consumer market. Granted, they were not as wide spread back then, but IMHO, it would have made sense back then to use 64 bits on 64 bit systems. It makes perfect sense now that 64 bit OSses are finally mainstream.



  • Jorj, If you have no application with such a memory demands it's your lucky but some people have. Furthermore, if you have an application with dynamical data structure you are already possible victim of x32 constraints in x64 machine. Qt is posed as versatile framework but without x64 containers it wouldn't be. Moreover using int for index is a wrong in C++.



  • I've noticed that this is still a problem in 5.4. Is there any trend at all toward changing this shortcoming? It's becoming a problem in my application, forcing me to migrate to std::vector.



  • I've noticed that this is still a problem in 5.4. Is there any trend at all toward changing this shortcoming? It's becoming a problem in my application, forcing me to migrate to std::vector.



  • No, there is no inclination to change this. If you are using data structures that big, you are better of with non-Qt data structures anyway, so the opinus communis is that it is not worth changing. What might change is that the containers may start using an index type that is defined to be 64 bit on 64 bit machines, and 32 bits otherwise. That, however, would be Qt 6 material, as it would break binary compatibility.

    So, by all means choose std::vector. It will most likely be faster than QVector due to the lack of overhead of the COW.



  • No, there is no inclination to change this. If you are using data structures that big, you are better of with non-Qt data structures anyway, so the opinus communis is that it is not worth changing. What might change is that the containers may start using an index type that is defined to be 64 bit on 64 bit machines, and 32 bits otherwise. That, however, would be Qt 6 material, as it would break binary compatibility.

    So, by all means choose std::vector. It will most likely be faster than QVector due to the lack of overhead of the COW.



  • Using std:: data structures makes interconnection with Qt world inconvenient so it's really disappointing to hear the Qt community sticks with wrong index type instead of acknowledging the C++ way. This is not only about big-size data structures it's about how to use Qt with other C++ libraries. In this situation Qt is absolutely wrong using int as the container size/indices



  • Using std:: data structures makes interconnection with Qt world inconvenient so it's really disappointing to hear the Qt community sticks with wrong index type instead of acknowledging the C++ way. This is not only about big-size data structures it's about how to use Qt with other C++ libraries. In this situation Qt is absolutely wrong using int as the container size/indices



  • I know. But you probably would not do much interconnection with Qt on huge datasets anyway.

    Note that there is quite a bit of API on the Qt containers that relies on the signed integers. Changing that would mean a huge break in source compatibility, making it much harder to port your code from a current Qt version to whatever Qt version would introduce such a break.

    Also note that Qt containers offer an std compatible interface. So, you can use all your std and compatible algorithms on the Qt containers if you so wish.



  • I know. But you probably would not do much interconnection with Qt on huge datasets anyway.

    Note that there is quite a bit of API on the Qt containers that relies on the signed integers. Changing that would mean a huge break in source compatibility, making it much harder to port your code from a current Qt version to whatever Qt version would introduce such a break.

    Also note that Qt containers offer an std compatible interface. So, you can use all your std and compatible algorithms on the Qt containers if you so wish.


Log in to reply
 

Looks like your connection to Qt Forum was lost, please wait while we try to reconnect.