How to handle a threading socket server

[DieterV]

Hi,

I'm trying to achieve something which I think would be fairly basic:

• Have a main window with a QPlainTextEdit widget for feedback during inital prototyping
• Start a thread that waits for a raw socket connection
• If a new connection comes in, create a new thread that handles input from that connection, likely emitting a signal that is connected in the main window to update the gui and be processed.

My code (I've tried to limit to what matters):

class MainWindow(qtw.QMainWindow, Ui_wdw_main_window):

    def __init__(self, *args, obj=None, **kwargs):
        super().__init__(*args, **kwargs)
        self.setupUi(self)

        self.txt_edit = qtw.QPlainTextEdit(self)
        self.server = Server()
        self.threads = []
        self._connected_socks = []
        self.server_thread = qtc.QThread()
        self.server.moveToThread(self.server_thread)
        self.server.sgn_new_connection.connect(self._new_connection)
        log.debug('Now starting server...')
        self.server_thread.started.connect(self.server.run)
        self.server_thread.start()
        self.threads.append(self.server_thread)

    @qtc.pyqtSlot(str, int, socket.socket)
    def _new_connection(self, ip: str, port: int, sock: socket.socket) -> None:
        if sock in self._connected_socks:
            log.warning(f'{ip} already connected.')
            return

        self._connected_socks.append(sock)

        self.txt_edit.appendPlainText(f'Connection from {ip}.')
        self._conn = ClientHandler(ip=ip, port=port, sock=sock)
        self._thread = qtc.QThread()  # CRASH HAPPENS HERE ON 2nd CONNECTION
        self._conn.moveToThread(self._thread)
        self._conn.sgn_new_message.connect(self._new_message)
        self._thread.started.connect(self._conn.run)
        self._thread.start()

    def _new_message(self, ip: str, msg: str) -> None:
        log.debug(f'message: {msg}')
        self.txt_edit.appendPlainText(f'[{ip}] -> {msg}')

In _new_connection I check if the socket is already connected. I don't know why yet, but this slot is called twice each time a new client connects. I'm not proud about this hack but until I figure out why it's called twiced, this seems to work.

The two QObject subclasses handling the sockets:

class Server(qtc.QObject):
    sgn_new_connection = qtc.pyqtSignal(str, int, socket.socket)
    sgn_new_message = qtc.pyqtSignal(str, str)
    sgn_connection_closed = qtc.pyqtSignal(str)

    def __init__(self, ip: str = '0.0.0.0', port: int = 8080):  # nosec B104
        # Thread.__init__(self)
        super().__init__()
        self.ip = ip
        self.port = port

    def run(self):
        log.debug(f'Starting server on {self.ip}:{self.port}')
        ip = self.ip
        port = self.port
        tcpServer = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        tcpServer.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
        tcpServer.bind((ip, port))
        threads = []

        tcpServer.listen(10)
        log.debug(f'Server listening on {self.ip}:{self.port}.')

        while True:
            (sock, (ip, port)) = tcpServer.accept()
            self.sgn_new_connection.emit(ip, port, sock)
            log.debug(f'New connection from {ip}:{port}')
            self.sgn_new_connection.emit(ip, port, sock)

    def _new_message(self, ip: str, msg: str) -> None:
        log.debug('triggered in server')
        self.sgn_new_message.emit(ip, msg)


class ClientHandler(qtc.QObject):
    sgn_new_message = qtc.pyqtSignal(str, str)
    sgn_connection_closed = qtc.pyqtSignal(str)

    def __init__(self, ip, port, sock):
        super().__init__()
        self.ip = ip
        self.port = port
        self.sock = sock
        log.debug(f'Socket thread started for {ip}:{port}')

    def run(self):
        while True:
            data = self.sock.recv(2048)
            if len(data) == 0:
                log.debug('Socket closed from remote side.')
                self.sgn_connection_closed.emit(self.ip)
                self.terminate()
            data_str = data.decode('utf-8').strip()
            self.sgn_new_message.emit(self.ip, data_str)
            log.debug(f'Data Rx from {self.ip}: {data_str}')

This works for one connection. However, when i open a second connection, the application crashes in when the thread get created in _new_connection in the main window.

I've tried a number of things to get this working, but since I don't have a good understanding of how to work with threading yet (and it seems to be a minefield) I'm afraid of doing things the wrong way and having them work but not reliable.
Originally, I started a new thread for each connection from within the Server class. This seemed to work, but starting a thread for each socket from within the server thread started in the main window, I couldn't get signals to work correctly.and

I looked into a QRunnable and QThreadPool but from what I read, a QRunnable does not support signals and slots, which would make it useless.
I also looked into subclassing QThread, but most sources I found adviced against this and I see no benefit in my case.

So, a couple of questions:

1. Is my approach correct? (clearly not, it crashes :) )
2. If not, why?
3. I don't get any feedback when the app crashes, just "QThread: Destroyed while thread is still running". I still need to figure out how to close running threads when the app crashes and all the other scenarios. Is there a way to have PyQT5 use a child logger of my root logging object?

Thanks in advance...

[SGaist]

Hi,

First thing: since you are using Qt, why not make use of QTcpServer and QTcpSocket ?
Second thing: since you are using Qt, if you make use of the classes mentioned above, you can avoid the use of threads as Qt being asynchronous, you likely won't need any thread at all which will simplify your code.

[DieterV]

@SGaist

I'm afraid the answer to both questions would be "because I didn't know any better" :). I looked for a way to create a socket server. Since qt has it's own implementation of threads, I used those.

I'll look for example code and work from there. This seemed like a good time to bite the bullet and learn how to work with threads, but if I can avoid it, I'll gladly take that chance :).

Thanks!

[DieterV]

@SGaist

"Unfortunately", this is not the first time that the solution to a problem of mine is "that is actually included in Qt".

Next time I'll check if there isn't anything already included in Qt. Promise.

[jeremy_k]

@DieterV said in How to handle a threading socket server:

• Start a thread that waits for a raw socket connection
  tcpServer = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

Quick terminology point: This is a stream or TCP socket, rather than a raw socket (socket.SOCK_RAW).

[DieterV]

@jeremy_k
Another thing I'll read up on. Thanks.

[DieterV]

@jeremy_k

Ok, I indeed definitely mean SOCK_STREAM. Thanks for clearing that up.