XMLHtpRequest problems with SSL sites on Mac OS

peteispo

I have a problem with connecting to SSL sites from Mac OS when using HTTPS.
I'm using MacOS 10.12.5 on a late 2012 iMac. Qt Version 5.9.1.
The following code is a test program which hopefully reproduces the problem

import QtQuick 2.6
import QtQuick.Window 2.2
import QtQuick.Controls 2.1
import QtQuick.Layouts 1.1
Window {
    visible: true
    height: 300
    width: 200
    ColumnLayout {
        id: httpstest
        anchors.fill: parent
        Button {
            text: "Test Google"
            // Google (should return 200)
            onClicked: getRequest("https://www.google.com",this)
        }
        Button {
            text: "Test Justcroft Intranet2"
            // Apache HTTPS server on Ubuntu 16.04 (should return 200)
            onClicked: getRequest("https://intranet2.justcroft.com",this)
        }        
        Button {
            text: "Test StartEx"
            // Apache HTTPS server on Ubuntu 16.04 (should return 200)
            onClicked: getRequest("https://startexchange.justcroft.com/",this)
        }
        Button {
            text: "Test StartExApp"
            // Apache HTTPS server on Ubuntu 16.04 - app access (should return 401)
            onClicked: getRequest("https://startexchange.justcroft.com/app.php/investor/test",this)
        }
    }

    /*
     * url    is the location to get
     * button is the button pressed so we can update the button text
     */
    function getRequest(url,button)
    {
        var req = new XMLHttpRequest()
        var timer = Qt.createQmlObject("import QtQuick 2.3; Timer {interval: 5000; repeat: false; running: false;}",httpstest,"MyTimer");
        timer.triggered.connect(function(){
            req.abort();
            console.log("timed out");
        });
        console.log("\n"+button.text+": "+url);
        req.onreadystatechange = function (event)
        {
            button.text = button.text.replace(/ \d+$/,'') + ' ' + req.readyState;
            switch (req.readyState)
            {
            case XMLHttpRequest.DONE:
                console.log(" Done : status = " + req.status +", "+ req.statusText)
                button.text = button.text.replace(/ \d+$/,'') + ' ' + req.status
                timer.running = false;
                break;
            case XMLHttpRequest.LOADING:
                console.log(" Loading: status = " + req.status +", "+ req.statusText)
                break;
            case XMLHttpRequest.HEADERS_RECEIVED:
                console.log(" Headers: status = " + req.status +", "+ req.statusText)
                console.log("\n" + req.getAllResponseHeaders())
                break;
            case XMLHttpRequest.OPENED:
                console.log(" Opened Request: readyState="+req.readyState)
                break;
            case XMLHttpRequest.UNSENT:
                console.log(" Unsent Request: readyState="+req.readyState)
                break;
            }
        }
        // Clear the button text
        button.text = button.text.replace(/ \d+$/,'')
        // Make the request
        req.open("GET", url, true)
        req.send()
        timer.start()
    }
}

So clicking the appropriate buttons should make the call to the various URLs to test.

The first one tries Google: that usually works (returning a 200 HTTP status).
The second also usually works: it's a test server I have set up to compare with the one that doesn't work.
The third is the problem: it doesn't appear to try to connect, only getting to readyState 1 (XMLHttpRequest.OPENED) and then giving up.
The fourth also doesn't work in the same way, but that's not surprising given that the third fails.

On the servers, intranet2.justcroft.com logs the connection and response just fine. startexchange.justcroft.com logs nothing at all until the test program is closed, at which point it logs 403 errors for each connection attempt.
The two justcroft.com servers are VMs on the same host running Ubuntu 16.04.2 on kernel 4.4.0-83, with Apache/2.4.18 handling the HTTPS and OpenSSL version 1.0.2g. Both have LetsEncrypt certificates which check out fine on browsers.

All of these URLs work fine from Safari (the fourth should ask for a username and password), and also using curl from a command line. In these cases the servers log the appropriate responses and there are no 403 errors.
They also all work fine in the same test code on Linux, and Android.

The addresses are all public-visible servers, with DNS set up and appropriate firewall rules to let HTTPS in (port 443), so anyone should be able to test them (please be reasonable!)

So I need clues about where to look, please.

peteispo

@peteispo
Some more checking reveals that actually both servers running Ubuntu 16.04 are failing under Mac OS: the successful server (intranet2.justcroft.com) was running Ubuntu 14.04, which I then upgraded to 16.04 where it then started to fail.

So it looks like the problem is in how XMLHttpRequest in Qt on Max OS interacts with an Apache HTTPS server and the differences between Ubuntu 14.04 and Ubuntu 16.04.
The obvious difference is that the default Apache mod-ssl configuration in 14.04 allows all SSL protocols (which I believe is SSLv3, TLSv1, TLSv1.1, TLSv1.2) whereas 16.04 mod-ssl config explicitly excludes SSLv3 (due to the Poodle attack). However, enabling SSLv3 on that server doesn't change the outcome.

If you want to compare the difference between a 14.04 server and a 16.04 server, then you can use https://www.justcroft.com instead of https://intranet2.justcroft.com in the second test.

What is really puzzling is that the XMLHttpRequest never seems to complete in the failing scenario: the connection is only logged on the server when the connection is dropped by the client program closing. That suggests that the two ends of the connection are waiting for something to happen: maybe the client thinks it has sent everything but the server is not receiving it all?

I'm looking at testing the sequence with Wireshark to see what is being sent, but I'm not an expert on TCP/IP packet analysis so that may not bear much fruit...

peteispo

@peteispo

Wireshark capture files:
First for https://www.justcroft.com (Ubuntu 14.04.5 GNU/Linux 4.4.0-45-generic x86_64, Apache 2.4.7, OpenSSL 1.0.1f)
https://www.dropbox.com/s/k7gqakwg8b0o5w9/test-www.pcapng?dl=0
Second for https://startexchange.justcroft.com (Ubuntu 16.04.2 GNU/Linux 4.4.0-83-generic x86_64, Apache 2.4.18, OpenSSL 1.0.2g)
https://www.dropbox.com/s/9h4q47fcbl3mq9f/test-startex.pcapng?dl=0

From what I can make of these captures, after about 0.3 a second (frame 20) the first one has completed it's handshake and the server is now sending Application Data packets in response to the client's Application Data packet (presumably the GET request), and all is well.
The second one shows the client sending an Application Data packet at frame 15 (the request) but then gets more Encrypted Handshake Message packets (and no Application Data) from the server. After about a minute (66 seconds - frame 23 in the capture) I close the program and a couple of packets get exchanged to close it off.