Looking for engin.io replacement -joining our effort initiative

[Charby]

To help synthesize the results of our evaluation of the different BaaS services, I propose to work on a commun document where we could all insert/edit the solutions we know (and also correct each other). I have started this google sheet for this purpose : https://docs.google.com/spreadsheets/d/1NoSQq2CtxDZEPTwe_1f2bmrB4BT4BRNeEj4qGdXa9S8/edit?usp=sharing

feel free to modify/update !

[gadlim]

@Charby said:

@gadlim I think the lock-in could be acceptable as long as we would use the backend services using a commun Qt API that would have a backup and reload function, don't you think ?

Yes, but a real, production-ready, common Qt API is a hard task. We're not there yet.

Regarding the push services, what would you advise as possible candidate ?

I haven't used any (using Pushbullet but that's different), so can't really advise.

[gadlim]

@Charby said:

To help synthesize the results of our evaluation of the different BaaS services, I propose to work on a commun document where we could all insert/edit the solutions we know (and also correct each other).
feel free to modify/update !

Good idea but wrong link

[Charby]

@gadlim Hopefully you have seen it quickly ! I have updated the link (could you please remove the link in your reply ?) - thanks !

[gadlim]

@gadlim said:

Follow-up on my Cloudant evaluation: I ran into problems trying to figure out how to encrypt user passwords.
I'll have a conversation on that subject with IBMers tomorrow.
I hope to also discuss the broader subject of whether Cloudant is a good fit for 2-tiers apps (no server code, or almost no server code).

Follow-up to the follow-up:

• user passwords have to be manually-encoded (not a big deal but annoying)
• 2-tiers apps; not a perfect fit, for the above reason and others

Coming from Enginio (MongoDB), the migration can be painful. The main problem is that it's not as much "forget-there's-even-a-database" as MongoDB is. For instance you don't have the concept of the collection, so if you treat each collection as a separate DB in CouchDB, you have to manage different DBs, and for each DB you have to think ahead about the indexes you need (in MongoDB indexes are are not mandatory for queries).
In contrast, the Parse REST API is eerily similar to the Enginio API. There's not so many ways to implement a REST API over MongoDB, but still you have to wonder if Enginio used Parse.com as an inspiration.

In my case, I'm running out of time and have a fair amount of code to port, so my choice is made: I'll go with a Parse Server hosting. There's several options, I think I'll go with IBM Bluemix. I've followed the instructions to setup a Parse server, it involved a whole day of trial-and-error but it's working, and I've ported our main app in less than a day - it's just a matter of tweaking some parameters. It's not fully working yet because of the fact that Parse only sends back the ID of new objects instead of the whole objects, but it's still ten times easier than other options.

Porting the Enginio SDK to Parse Server or anything similar should be easy too.

[Charby]

@gadlim regarding the user passwords, If I am correct, there are 2 different methods, one can either base64 encode username:password to be passed as header (with "Basic " as a prefix) or you can post with something like this :

QUrlQuery postData;
postData.addQueryItem("name", user);
postData.addQueryItem("password", password);
QNetworkAccessManager NAM;
NAM.post(request, postData.toString(QUrl::FullyEncoded).toUtf8());

I don't really understand the 2-tiers issue as you can directly use Cloudant i.o Cloudant through BlueMix, what would be your concern in this case ?

I have spend the last days playing with BlueMix (as I need to set up Push Notification) but regarding Cloudant I came to same conclusions that this lack of collection principle might be an issue for porting enginio apps...Firebase seems to be a much better candidate in this field (and Firebase has a powerfull mecanism to notify asynchronously when new data comes in - in replacement of enginion web socket - and I haven't found something similar with Cloudant...that would be another issue when polling is not an option).

I am very interested in the Parse Server hosting solution...I thought Parse was a no go as they are ramping down the services this year...do FB plans to release Parse to opensource (or did they already do ?). BTW, do you know what are their reasons for stopping the service ?
-Edit - replying to myself ;-) - Parse.com did release to opensource the datastore which is great - but the not the complete Parse solution (social login, PN, auth...), anyhow Parse server is promising ! I think I will have a try...

[gadlim]

@Charby said:

@gadlim regarding the user passwords, If I am correct, there are 2 different methods, one can either base64 encode username:password to be passed as header (with "Basic " as a prefix) or

I'm refering to the way the password is stored in the DB, not how it's sent to the server. In Enginio it's hidden (and probably encrypted), in Parse it's encrypted, etc. For Cloudant you have to encrypt yourself in the client or in an "update function" you're setting on the server.

I don't really understand the 2-tiers issue as you can directly use Cloudant i.o Cloudant through BlueMix, what would be your concern in this case ?

Nothing in particular, except passwords, but there's a feeling you get from the sample apps + the opinion from others that used it that you're expected to write some server code for "serious" applications.

I have spend the last days playing with BlueMix (as I need to set up Push Notification) but regarding Cloudant I came to same conclusions that this lack of collection principle might be an issue for porting enginio apps...Firebase seems to be a much better candidate in this field (and Firebase has a powerfull mecanism to notify asynchronously when new data comes in - in replacement of enginion web socket - and I haven't found something similar with Cloudant...that would be another issue when polling is not an option).

Maybe PouchDB can manage async data updates through the offline / online mechanism, but I haven't explored it.

I am very interested in the Parse Server hosting solution...I thought Parse was a no go as they are ramping down the services this year...do FB plans to release Parse to opensource (or did they already do ?). BTW, do you know what are their reasons for stopping the service ?
-Edit - replying to myself ;-) - Parse.com did release to opensource the datastore which is great - but the not the complete Parse solution (social login, PN, auth...), anyhow Parse server is promising ! I think I will have a try...

It's too early to know how the OSS Parse project will evolve, but it starts with a good momentum (around 600 000 apps), so it think it should carry on. For now, missing services are provided by the hosters, but for instance Push notifications have already been added.

What's interesting about Bluemix is that they have everything. But that's also a problem if you don't have a guide, because you're quickly lost in a sea of various tools and services. See for instance this , and that's just one single service.

[Charby]

Better late than never, I finally published a first draft of my work to design a Baas plugin.
The project can be found here : https://github.com/a-team-fr/QtQML-BaaS
It is really a Work In Progress stuff but things are slowly getting up...

For now, it is only supporting Parse Server, I used to start working on Firebase then Bluemix but I stopped and finally even dropped the code for now. I find Parse to be a much better candidate and I prefer first reach an operative state with Parse server only and then complete the plugin with others cloud services.

So far, the BaaS element can only manage users (told you it was an early phase!) and the BaaSModel can be used to get data and work as usual with a ListView for instance...I have implemented some of the feature for query but haven't tested yet.
The best (I think) is to build the whole project and see the sample project to see how to add new feature.
All comments are welcome ! I will try to improve the commenting and documenting...

I have an issue when deploying the plugin, the qmldir file is not copied to the plugin location - will investigate this later.

[gadlim]

@Charby thanks !

I think you're right, and that Parse Server is a good starting point. It may be even more than a starting point because it's evolving to be more modular and allow to plug many services, including other DBs.

I've (mostly) finished porting my app, it's working nicely. I had to patch the Parse Server code for a feature existing in Engin.io but missing in Parse (atomic updates of subdocuments), it wasn't that hard in spite of my lack of Node.js knowledge. I was pleasantly suprised by the size of the code (it's rather small, I was expecting a huge pile of exotic JS).
I've made a pull request for my patch, let's see how the discussion with the maintainers goes.

About your QML plugin, did you consider just forking/patching the Enginio C++/QML SDK instead ? If not, why ?

[Charby]

@gadlim I felt the same with Parse server, the project is rather small, simple and well written. Furthermore the project team seems very productive and I bet the project will become very popular.
I didn't fork Enginio for a couple of reasons :

• I looked at the code and found it rather difficult to master - I still haven't understand the private class principle and I haven't found a design documentation. But maybe I should have spent more time to study and hopefully understand it...
• I think forking the project would have been more difficult in the long term to main backward compatibility with existing API.
• even if my project is not a fork from enginio, it is inspired of some enginio design principles - for instance I liked the enginio model, but rewriting a similar model was not a complex task (and it was interesting from the learning point of view)
• I wanted the plugin to be as much as possible backend agnostic, that the reason why the main class is pure virtual and will be refined to ease integration of additional backend services.

[gadlim]

@Charby said:

I didn't fork Enginio for a couple of reasons :

• I looked at the code and found it rather difficult to master - I still haven't understand the private class principle and I haven't found a design documentation.

Ditto, I don't fully understand it either (the websocket stuff in particular). The private classes are mainly for binary compatibility that Qt has to maintain for all the Qt5 line, it's much simpler without that burden.

[SGaist]

Hi,

Don't start by dropping the PIMPL idiom from that module, especially if the goal is to get it integrated with the Qt distribution some day.

For more details about private implementation you can look here

[Charby]

@SGaist Thanks for this information, I wasn't aware of this...
Nevertheless, for now I would prefer YAGNI over PIMPL ;-)

[kshegunov]

@gadlim

The private classes are mainly for binary compatibility that Qt has to maintain for all the Qt5 line, it's much simpler without that burden.

Actually, as far as I know, binary compatibility is guaranteed only between major versions, but not for the whole Qt 5 line. Meaning 5.X.Y are binary compatible across all .Y versions, but not between the .X versions. Or in other words, Qt 5.5.1 would be compatible with 5.5.2, 5.5.3 and so on, but 5.4 is not guaranteed to be binary compatible with 5.5.

@Charby

Nevertheless, for now I would prefer YAGNI of PIMPL ;-)

I believe @SGaist's point is that you're in fact going to need it if you ever hope to have your module as part of Qt. :)
Otherwise you'd be asking everyone that uses your library to do a full rebuild of their code on any change in the private implementation of your module (like adding a member of a class for internal purposes) ... and this ain't a good way to design a library ...

Kind regards.

[gadlim]

@kshegunov said:

Actually, as far as I know, binary compatibility is guaranteed only between major versions, but not for the whole Qt 5 line. Meaning 5.X.Y are binary compatible across all .Y versions, but not between the .X versions. Or in other words, Qt 5.5.1 would be compatible with 5.5.2, 5.5.3 and so on, but 5.4 is not guaranteed to be binary compatible with 5.5.

No, there's binary compatibility for .X (minor) versions too
About the build dependencies, you're right, PIMPL is also better for that.
But I'll side this @Charby on this, it's a bit YAGNI at that point. The important thing is to get that project off the ground, and the code he's submitted is rather clean and simple, so easier to contribute to and build upon that the official Qt Enginio SDK. If someone wants to contribute a PIMPLification, I think he won't object (?), but I wouldn't be fair to require him to do that himself, as it would eat some of his time that can be better spent adding features.

[kshegunov]

@gadlim

No, there's binary compatibility for .X (minor) versions too

Right, either I have remembered wrongly, or the versioning system has changed somewhat from Qt 4. But in any case, using PIMPL is a good way to ensure all the aforementioned features.

But I'll side this @Charby on this, it's a bit YAGNI at that point.

I don't presume to tell you how to design or code your library, just throwing my 2 cents. ;)

but I wouldn't be fair to require him to do that himself, as it would eat some of his time that can be better spent adding features.

The only problem I see is that at a later point implementing PIMPL from existing code might be more involved, so I'd go with it from the start, but as I noted, it's up to the actual designer/programmer how to proceed. As a side note, the idiom doesn't actually require more work, only some care to separate the data from the interface. (If you wish you could take a peek at a library (wrap) I'm developing for OpenMPI, to have a baseline for what implementing PIMPL might involve).

Kind regards.

[Charby]

@kshegunov Just to clarify slightly my words : when I mentioned my preference to YAGNI over PIMPL, I was not meaning that PIMPL implementation would not be needed : I have a better understanding of PIMPL now - thanks to you guys! - and I am convinced that I would definitely go for it.

Actually, I meant that my plugin design is not mature enough at this stage, I have only implemented one backend service (Parse) and it is not even complete...so it is likely that when adding new features (or even worst, when I will integrate new backend services) , I would need to change the design, modifying declaration, visibility and so on...and at this stage, it would be much faster if I only modify one class i.o two.
As far as I understood the PIMPL implementation, adding the private class when my plugin design would be mature enough should be a straightforward process. But if you think, I am missing something please tell me, so I could integrate PIMPL right away.

Thanks for your feedback !

[Charby]

@gadlim said:

and the code he's submitted is rather clean and simple

That's always nice to hear, thank you !!!

[Charby]

Another but related topic : I have seen on the wiki, that it is encouraged to request for Qt playground project creation even if the project is in early stage. I think, that's would be great in this case so we could have a common repository to collaborate on a great plugin to give Qt App backend services.

Do you know how to proceed for creating playground projects ?
So far, I only sent a request on the development mailing list but I don't know what are the next steps...

[kshegunov]

@Charby
Hello,

As far as I understood the PIMPL implementation, adding the private class when my plugin design would be mature enough should be a straightforward process. But if you think, I am missing something please tell me, so I could integrate PIMPL right away.

Mostly it's very straightforward process, however there are few "tricks" that are usually closely associated with the PIMPL idiom. I've used two of them in my project (sourced in my previous post), i.e. private slots and private object constructor. The private slots can be substituted with lamda functions if you're willing to enforce C++11. The second one can be implemented at a later stage, but I personally would detest refactoring a full working piece of code only to ensure that the user won't be able to create an object of given class. These two help with the design - not exposing anything besides the barebone interface, but you could certainly skip them (in most cases), or at least defer the[ir] implementation. I hope this is helpful.

Kind regards.