I'm not sure anyone cares about this anymore, but I believe I know what the problem is.
When the plugin class is created, it is actually created outside the currently executing QApplication (via dlopen) and the code is running in a different thread. By reading up a bit on dlopen, it seems that the memory where the library is created comes from a different pool and that all libraries (used like this at least), should make sure to implement a function to destroy themselves.
Not sure if this makes any sense for someone, but it does for me.
Now, I just need to figure a way to get the libraries thread to join with the man QApplication that is already running.