Unit testing in Qt seems too hard
-
When trying to unit-test code written with Qt classes (mainly core), I often find it too hard to unit-test because it's not easy to replace production code with test code (test doubles/fakes/mocks).
Canonically, behavior replacement and sensing is done via interfaces. In his famous book Working Effectively with Legacy Code, Michael Feathers writes (pp. 280-281):
In other cases, classes are concrete and declared final or sealed, or they have key functions that are non-virtual, leaving no way to fake them out under test. In these cases, sometimes the best thing you can do is write a thin wrapper over the classes that you need to separate out. Make sure that you write your vendor and give them grief about making your development work difficult.Take e.g. a QFile, it has no interface to depend on, and many of its methods aren't virtual. So we're left with lot of work to get it faked or mocked, and it's a burden to maintain. We'd sometimes use duck typing with templates, but again it's not a great experience, especially without C++ Concepts at hand yet. Also, we end up doing more coarse-grained testing which isn't a great source of feedback.
So, now that the Qt Company seems to increase investment into easier testing, can we expect any improvements at this point? We're paying customers, by the way.
-
When trying to unit-test code written with Qt classes (mainly core), I often find it too hard to unit-test because it's not easy to replace production code with test code (test doubles/fakes/mocks).
Canonically, behavior replacement and sensing is done via interfaces. In his famous book Working Effectively with Legacy Code, Michael Feathers writes (pp. 280-281):
In other cases, classes are concrete and declared final or sealed, or they have key functions that are non-virtual, leaving no way to fake them out under test. In these cases, sometimes the best thing you can do is write a thin wrapper over the classes that you need to separate out. Make sure that you write your vendor and give them grief about making your development work difficult.Take e.g. a QFile, it has no interface to depend on, and many of its methods aren't virtual. So we're left with lot of work to get it faked or mocked, and it's a burden to maintain. We'd sometimes use duck typing with templates, but again it's not a great experience, especially without C++ Concepts at hand yet. Also, we end up doing more coarse-grained testing which isn't a great source of feedback.
So, now that the Qt Company seems to increase investment into easier testing, can we expect any improvements at this point? We're paying customers, by the way.
@rico-chet
Hi. This is a user forum, users just like you.can we expect any improvements at this point? We're paying customers, by the way.
If you are paying you should take this up with The Qt Company!
-
Hi,
QFile is a good example, if you want to mock a file then your interface should handle a QIODevice rather than specifically a QFile. Doing so you can replace it with a mock class that implements whatever you want for the tests.
-
@rico-chet
Hi. This is a user forum, users just like you.can we expect any improvements at this point? We're paying customers, by the way.
If you are paying you should take this up with The Qt Company!
@JonB so the Qt Company doesn't care about the user voices here? Not the greatest attitude to me... I will try to bring this up through our support channel.
@SGaist Let's take
QFile, it has its own methods likeremove()which cannot be replaced via its base class. It cannot be replaced by a derived class either, as it's not virtual. Same goes forQIODevice::read()which isn't virtual and cannot be replaced easily. -
@JonB so the Qt Company doesn't care about the user voices here? Not the greatest attitude to me... I will try to bring this up through our support channel.
@SGaist Let's take
QFile, it has its own methods likeremove()which cannot be replaced via its base class. It cannot be replaced by a derived class either, as it's not virtual. Same goes forQIODevice::read()which isn't virtual and cannot be replaced easily.@rico-chet said in Unit testing in Qt seems too hard:
so the Qt Company doesn't care about the user voices here?
As paying user you can ask QtCompany directly. This here is user forum, most of the people here do not work for QtCompany.
Also, if you have any improvement suggestions then file a change request in Qt bug tracker -
@rico-chet said in Unit testing in Qt seems too hard:
@SGaist Let's take QFile, it has its own methods like remove() which cannot be replaced via its base class. It cannot be replaced by a derived class either, as it's not virtual. Same goes for QIODevice::read() which isn't virtual and cannot be replaced easily.
For QIODevice, you are looking at the wrong function, there's QIODevice::readData for you to re-implement.
For QFile, if you need to replace the remove function with something else, then I would argue that you are not using the correct abstraction level. Not knowing anything about what you are currently implementing, it's impossible to discuss what would be best. In any case, making every method of a class virtual is usually not the solution. In the remove example, needing to re-implement it to perform some other task than removing would both feel counter-intuitive and dangerous. If you need to do something different, you might rather want to add a new method to do what you want or implement a wrapper class that provides the API you need on top of QFile.
-
@rico-chet said in Unit testing in Qt seems too hard:
@SGaist Let's take QFile, it has its own methods like remove() which cannot be replaced via its base class. It cannot be replaced by a derived class either, as it's not virtual. Same goes for QIODevice::read() which isn't virtual and cannot be replaced easily.
For QIODevice, you are looking at the wrong function, there's QIODevice::readData for you to re-implement.
For QFile, if you need to replace the remove function with something else, then I would argue that you are not using the correct abstraction level. Not knowing anything about what you are currently implementing, it's impossible to discuss what would be best. In any case, making every method of a class virtual is usually not the solution. In the remove example, needing to re-implement it to perform some other task than removing would both feel counter-intuitive and dangerous. If you need to do something different, you might rather want to add a new method to do what you want or implement a wrapper class that provides the API you need on top of QFile.
@SGaist said in Unit testing in Qt seems too hard:
For QIODevice, you are looking at the wrong function, there's QIODevice::readData for you to re-implement.
There's no guarantee that
QIODevice::readAllcallsQIODevice::readData, it's not part of its contract. Also, testing through theQFileimplementation isn't unit testing, as a red test can have multiple reasons to fail (the unit under test or theQFileimplementation)For QFile, if you need to replace the remove function with something else, then I would argue that you are not using the correct abstraction level.
I need to replace the function for sensing during unit testing, e.g. to check that the code under test removes a file under certain conditions only.
[...] implement a wrapper class that provides the API you need on top of QFile.
I believe this work should be performed at the Qt Company, not by their users or customers. I asked our internal contact to file a request.
-
@SGaist said in Unit testing in Qt seems too hard:
For QIODevice, you are looking at the wrong function, there's QIODevice::readData for you to re-implement.
There's no guarantee that
QIODevice::readAllcallsQIODevice::readData, it's not part of its contract. Also, testing through theQFileimplementation isn't unit testing, as a red test can have multiple reasons to fail (the unit under test or theQFileimplementation)For QFile, if you need to replace the remove function with something else, then I would argue that you are not using the correct abstraction level.
I need to replace the function for sensing during unit testing, e.g. to check that the code under test removes a file under certain conditions only.
[...] implement a wrapper class that provides the API you need on top of QFile.
I believe this work should be performed at the Qt Company, not by their users or customers. I asked our internal contact to file a request.
@rico-chet said in Unit testing in Qt seems too hard:
I believe this work should be performed at the Qt Company, not by their users or customers. I asked our internal contact to file a request.
This forum is first line of defence for users, the mailing list has a lot more participation from TQC and other key external figures in the Qt development ecosystem.
-
@SGaist said in Unit testing in Qt seems too hard:
For QIODevice, you are looking at the wrong function, there's QIODevice::readData for you to re-implement.
There's no guarantee that
QIODevice::readAllcallsQIODevice::readData, it's not part of its contract. Also, testing through theQFileimplementation isn't unit testing, as a red test can have multiple reasons to fail (the unit under test or theQFileimplementation)For QFile, if you need to replace the remove function with something else, then I would argue that you are not using the correct abstraction level.
I need to replace the function for sensing during unit testing, e.g. to check that the code under test removes a file under certain conditions only.
[...] implement a wrapper class that provides the API you need on top of QFile.
I believe this work should be performed at the Qt Company, not by their users or customers. I asked our internal contact to file a request.
@rico-chet said in Unit testing in Qt seems too hard:
There's no guarantee that QIODevice::readAll calls QIODevice::readData, it's not part of its contract.
Take a look at the implementation, the contract of the class is that all methods that do some sort of reading will call in the end readData. It would not make sense to provide this protected method and its write counterPart to ignore them.
@rico-chet said in Unit testing in Qt seems too hard:
I need to replace the function for sensing during unit testing, e.g. to check that the code under test removes a file under certain conditions only.
Then, I'll stand on my point: you are doing something off with your tests. There's no reasons to modify QFile in order to check whether the removal succeeded under the right conditions. You would need tests for both normal cases and what is expected to know normal failure cases.
-
@rico-chet said in Unit testing in Qt seems too hard:
I believe this work should be performed at the Qt Company, not by their users or customers. I asked our internal contact to file a request.
This forum is first line of defence for users, the mailing list has a lot more participation from TQC and other key external figures in the Qt development ecosystem.
-
@rico-chet said in Unit testing in Qt seems too hard:
There's no guarantee that QIODevice::readAll calls QIODevice::readData, it's not part of its contract.
Take a look at the implementation, the contract of the class is that all methods that do some sort of reading will call in the end readData. It would not make sense to provide this protected method and its write counterPart to ignore them.
@rico-chet said in Unit testing in Qt seems too hard:
I need to replace the function for sensing during unit testing, e.g. to check that the code under test removes a file under certain conditions only.
Then, I'll stand on my point: you are doing something off with your tests. There's no reasons to modify QFile in order to check whether the removal succeeded under the right conditions. You would need tests for both normal cases and what is expected to know normal failure cases.
@SGaist said in Unit testing in Qt seems too hard:
Take a look at the implementation
Software development rule #1: don't depend on unstable things. Implementation details are among the least stable things. I don't have time to go over Qt sources every time we update to a new version to check if the old assumptions still hold, sorry.
@rico-chet said in Unit testing in Qt seems too hard:
I need to replace the function for sensing during unit testing, e.g. to check that the code under test removes a file under certain conditions only.
Then, I'll stand on my point: you are doing something off with your tests. There's no reasons to modify QFile in order to check whether the removal succeeded under the right conditions.
To be more precise, I need to replace the function for sensing during unit testing, e.g. to check that the code under test attempts to remove a file under certain conditions only. I don't care so much about failure cases, yet.
Replacing behavior of collaborating objects (in our case a
QFile) with test doubles is exactly the precondition for small-grained and quick unit testing.
