How to use Qt to keep a class open and emit a signal

localthink

I am trying to send a .hdf5 file to a method in the main python file in the class here:

class DesignerMainWindow(QtGui.QMainWindow, Ui_MainWindow):
    """Customization for Qt Designer created window"""

    signal_output_log = QtCore.Signal("QString")
    sig_clear_log = QtCore.Signal()

    def __init__(self, parent=None):
        super(DesignerMainWindow, self).__init__(parent)
        self.setupUi(self)
        self.image_analyzer = ImageAnalyzer(self)
        self.listener = watchdog_search.ObserverWrapper("/home/Test_Data/")
        self.on_finished_run(self.listener.wait_for_file())
    def on_finished_run(self, tuple: ()):
        self.image_analyzer.load_image(str(tuple[0]), str(tuple[1]), from_remote=True)

The .hdf5 file comes from this `watchdog_search.py:

import time
import traceback
import os

import h5py
import queue
from typing import Union

from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler, DirCreatedEvent, FileCreatedEvent


class NewFileHandler(FileSystemEventHandler):
    """h5 file creation handler for Watchdog"""

    def __init__(self):
        self.file_queue = queue.Queue()

    # callback for File/Directory created event, called by Observer.
    def on_created(self, event: Union[DirCreatedEvent, FileCreatedEvent]):
        if event.src_path[-4:] == "hdf5":
            # run callback with path string
            self.file_queue.put(event.src_path)


class ObserverWrapper:
    """Encapsulated Observer boilerplate"""

    def __init__(self, path: str, recursive=True):
        self.path = path
        self.recursive = recursive

        self.observer = Observer()
        self.handler = NewFileHandler()

        self.observer.schedule(self.handler, path=path, recursive=recursive)

        self.start()

    def start(self):
        """
        Starts observing for filesystem events. Runs self.routine() every 1 second.

        :param blocking: If true, blocks main thread until keyboard interrupt.
        """

        self.observer.start()
    def stop(self):
        """
        Stops the observer. When running self.start(blocking=True) then you don't need to call this.
        """

        self.observer.stop()
        self.observer.join()

    def wait_for_file(self):
        """
        Wait and Process newly created files
        """

        max_retry_count = 3500 # for test purposes now but want to set an upper bound on verifying a file is finished.
        # will try h5 file for a max of 35 seconds (upper bound) to see if the file is finished.
        # Files are usually finished within 20-30 seconds
        #
        retry_interval_seconds = .01 # every hundreth it will try the file to see if it finished writing

        # wait for file to be added
        #print(self.handler.file_queue.get(block=True))
        file_path = self.handler.file_queue.get(block=True)
        file_name = os.path.basename(file_path)

        # try to open the file
        retry_count = 0
        while True:
            try:
                file = h5py.File(file_path, "r")
                file.close()
                return file_path, file_name
            except OSError:
                if retry_count < max_retry_count:
                    retry_count += 1
                    print(f"h5 file <{file_path}> is locked, retrying {retry_count}/{max_retry_count}")
                    time.sleep(retry_interval_seconds)
                else:
                    print(f"h5 file <{file_path}> reached max retry count, skipping")

            except Exception as err:
                print(f"Got unexpected Error <{type(err).__name__}> while opening <{file_path}> ")
                traceback.print_exc()

Currently I call this file by

self.listener = watchdog_search.ObserverWrapper("/path/to/folder/of/interest")

in the main.py file but this only sends one hdf5 file and doesn't send any more. Watchdog needs to stay open and send content to main.py every time there is a new hdf5 file available. Does anyone know how to use Qt to do this? This is not a question of watchdog but rather a question of asynchronous programming and Qt. I need at the end to send the file_path and file_name of the .hdf5 to on_finished_run() or perhaps have on_finished_run() be the subscriber for the signal but I am unsure how to do this when the try returns not the emitted signal but rather the file path and name. Any insight is greatly appreciated.

JonB

@localthink
I don't know much about this because all of your code in watchdog_search.py is Python/watchdog, whatever that is, and has nothing to do with Qt. But I believe you have 3 possible approaches:

As long as you have while True: or lots of retry_count or time.sleep() you will need to put the watchdog stuff in a thread to avoid blocking/allow run the Qt main UI event loop.
If this is something you can do once regularly on a timer --- i.e polling --- you can get rid of your blocking loop and use Qt's QTimer to do the polls. No threads, no blocking.
If your watchdog does something about looking to see whether a file exists intermittently Qt has its own QFileSystemWatcher instead which again does not block.

when the try returns not the emitted signal but rather the file path and name

If you do end up using this code: call it from class and use the return result as parameters to a Qt signal. I think the Python syntax would be something like:

file_path, file_name = something.wait_for_file()
self.some_signal_name.emit(file_path, file_name)

The signal emitting class must inherit from QObject (either solely, or you could use Python multiple inheritance).

localthink

@JonB Thanks for the reply.

I think putting watchdog in a thread would be nice. I've been reading up on this but have failed to get it to work.
No it won't be done once but rather a lot of times so I don't think this is a feasible option.
Watchdog just checks if a new file is populated in the folder. When a new file is populated, the NewFileHandler.on_created() checks if it is an hdf5 file. So I wouldn't be able to check what the name of the file is or any of that but rather do post processing in this sense.

If you do end up using this code: call it from class and use the return result as parameters to a Qt signal. I think the Python syntax would be something like:

Call what from class exactly?

JonB

@localthink said in How to use Qt to keep a class open and emit a signal:

I think putting watchdog in a thread would be nice. I've been reading up on this but have failed to get it to work.

This would be my last choice. Especially if I were new I would advise trying to keep away from threads if possible.

I think either of the other two would be preferable.

Call what from class exactly?

The two lines of code I wrote, which use self.some_signal_name.emit(). The class which is emitting the signal.

localthink

@JonB So would the signal in this case be file_name and file_path and they get sent to the method on_finished_run()?

I've been trying to play around with this the last couple of weeks (emit and signal) and this is what I have so far but I'm just not making progress on it:

class DesignerMainWindow(QtGui.QMainWindow, Ui_MainWindow):
    """Customization for Qt Designer created window"""

    signal_output_log = QtCore.Signal("QString")
    sig_clear_log = QtCore.Signal()

    def __init__(self, parent=None):
        super(DesignerMainWindow, self).__init__(parent)
        self.setupUi(self)
        self.image_analyzer = ImageAnalyzer(self)
        self.listener = watchdog_search.ObserverWrapper("/home/Test_Data/")
        self.handler.bridge.created.connect(self.on_finished_run)
    def on_finished_run(self, tuple: ()):
        self.image_analyzer.load_image(str(tuple[0]), str(tuple[1]), from_remote=True)

and from watchdog_search.py

import time
import traceback
import os

import h5py
import queue
from typing import Union

from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler, DirCreatedEvent, FileCreatedEvent

from .tools.qt import QtCore


class Bridge(QtCore.QObject):
    created = QtCore.Signal(FileCreatedEvent)

class NewFileHandler(FileSystemEventHandler):
    """h5 file creation handler for Watchdog"""

    def __init__(self):
        super().__init__()
        #self.file_queue = queue.Queue()
        self.bridge = Bridge()

    # callback for File/Directory created event, called by Observer.
    def on_created(self, event: Union[DirCreatedEvent, FileCreatedEvent]):
        if event.src_path[-4:] == "hdf5":
            # run callback with path string
            #self.file_queue.put(event.src_path)
            self.bridge.created.emit(event.src_path)


class ObserverWrapper:
    """Encapsulated Observer boilerplate"""

    def __init__(self, path: str):#, recursive=True):
        self.path = path
        #self.recursive = recursive

        self.observer = Observer()
        self.handler = NewFileHandler()

        self.observer.schedule(self.handler, path=path, recursive=True)

        self.start()


    def start(self):
        """
        Starts observing for filesystem events. Runs self.routine() every 1 second.

        :param blocking: If true, blocks main thread until keyboard interrupt.
        """

        self.observer.start()
    def stop(self):
        """
        Stops the observer. When running self.start(blocking=True) then you don't need to call this.
        """

        self.observer.stop()
        self.observer.join()

    def wait_for_file(self):
        """
        Wait and Process newly created files
        """

        max_retry_count = 3500 # for test purposes now but want to set an upper bound on verifying a file is finished.
        # will try h5 file for a max of 35 seconds (upper bound) to see if the file is finished.
        # Files are usually finished within 20-30 seconds
        #
        retry_interval_seconds = .01 # every hundreth it will try the file to see if it finished writing

        # wait for file to be added
        #print(self.handler.file_queue.get(block=True))

        #print(self.handler.on_created(self.path))
        #file_path =
        #print(self.handler.file_queue.get(block=True))
        #file_name = os.path.basename(file_path)

        # try to open the file
        retry_count = 0
        while True:
            try:
                file = h5py.File(file_path, "r")
                file.close()
                return file_path, file_name
            except OSError:
                if retry_count < max_retry_count:
                    retry_count += 1
                    print(f"h5 file <{file_path}> is locked, retrying {retry_count}/{max_retry_count}")
                    time.sleep(retry_interval_seconds)
                else:
                    print(f"h5 file <{file_path}> reached max retry count, skipping")

            except Exception as err:
                print(f"Got unexpected Error <{type(err).__name__}> while opening <{file_path}> ")
                traceback.print_exc()

JonB

@localthink said in How to use Qt to keep a class open and emit a signal:

and this is what I have so far but I'm just not making progress on it:

I'm sorry but I don't know what this means. Do you have a specific issue/question?

self.handler.bridge.created.connect(): looks quite involved! Maybe your code would be easier if you did class NewFileHandler(FileSystemEventHandler, QtCore.QObject): so you can do the signals inside that method instead of needing the class Bridge?

You have chosen to go down your current route, with blocking calls to wait_for_file(). I said that would be my least favourite approach. Rewriting it to do one try each time off a QTimer would seems preferable to me. However as I said if you maintain this way you will need to use threads, which I have said I would avoid, yet I do not see any threads in your code.

localthink

@JonB Thanks for the reply. Yea I would prefer to do the easy route. I've been trying the third option with QFileSystemWatcher and I don't really believe it can do what watchdog is doing unfortunately. I am unsure how you propose using Qtimer exactly. Could you help me understand what you mean with option two?

Edit: couldn't I just make block=False in watchdog and there will then be no blocking?

JonB

@localthink said in How to use Qt to keep a class open and emit a signal:

Edit: couldn't I just make block=False in watchdog and there will then be no blocking?

How do I know if I know nothing about watchdog, as I said? But if that avoids blocking and therefore the need for a thread it sounds preferable.

I am unsure how you propose using Qtimer exactly.

So far as i understand your code, wait_for_file() does/can sit and try 3,500 times with sleep() for 1/100th of a second between retries. If you only make it do one (or possibly a very small number, like 10 for 1/10th of a second delay) call at a time, and return either the file whatever or failure, then you could just call that repeatedly from a QTimer with a corresponding interval, doing your own member variable for counting up the total retries till you are fed up of waiting. By using QTimer like this you allow the main UI thread to keep running responsively without the need to introduce any threading.

localthink

@JonB The 3500 times is just arbitrary and will be changed to something smaller. I took your advice and I am trying to get it up and running with QTimer but am really confused on how it can solve the issue. Some questions that come up are: How is the signal emitted? How do I connect it to the __init__() method in DesignerMainWindow?

Here is the code I am working with now and I am not sure how QTimer could help from here:

class NewFileHandler(FileSystemEventHandler):
    """h5 file creation handler for Watchdog"""

    def __init__(self):
        super().__init__()
        self.file_queue = queue.Queue()
        # self.bridge = Bridge()

    # callback for File/Directory created event, called by Observer.
    def on_created(self, event: Union[DirCreatedEvent, FileCreatedEvent]):
        if event.src_path[-4:] == "hdf5":
            # run callback with path string
            self.file_queue.put(event.src_path)
            # self.bridge.created.emit(event.src_path)


class ObserverWrapper(QMainWindow):
    """Encapsulated Observer boilerplate"""

    def __init__(self, path: str):#, recursive=True):
        super().__init__()
        self.path = path
        #self.recursive = recursive

        self.observer = Observer()
        self.handler = NewFileHandler()

        self.observer.schedule(self.handler, path=path, recursive=True)

        self.start()

        self.counter = 0
        timer = QTimer(self)
        # Add a method with the timer
        timer.timeout.connect(self.wait_for_file())


    def start(self):
        """
        Starts observing for filesystem events. Runs self.routine() every 1 second.

        :param blocking: If true, blocks main thread until keyboard interrupt.
        """

        self.observer.start()
    def stop(self):
        """
        Stops the observer. When running self.start(blocking=True) then you don't need to call this.
        """

        self.observer.stop()
        self.observer.join()

    def wait_for_file(self):
        """
        Wait and Process newly created files
        """

        max_retry_count = 3500 # for test purposes now but want to set an upper bound on verifying a file is finished.
        # will try h5 file for a max of 35 seconds (upper bound) to see if the file is finished.
        # Files are usually finished within 20-30 seconds
        #
        retry_interval_seconds = .01 # every hundreth it will try the file to see if it finished writing

        # wait for file to be added
        #print(self.handler.file_queue.get(block=True))

        #print(self.handler.on_created(self.path))
        #file_path =
        #print(self.handler.file_queue.get(block=True))
        #file_name = os.path.basename(file_path)
        file_path = self.handler.file_queue.get(block=True)
        file_name = os.path.basename(file_path)

        # try to open the file
        # retry_count = 0
        while True:
            try:
                file = h5py.File(file_path, "r")
                file.close()
                return file_path, file_name
            except OSError:
                if self.counter < max_retry_count:
                    self.counter += 1
                    #retry_count += 1
                    print(f"h5 file <{file_path}> is locked, retrying {self.counter}/{max_retry_count}")
                    time.sleep(retry_interval_seconds)
                else:
                    print(f"h5 file <{file_path}> reached max retry count, skipping")

            except Exception as err:
                print(f"Got unexpected Error <{type(err).__name__}> while opening <{file_path}> ")
                traceback.print_exc()

JonB

@localthink
Sorry but I don't know what you are having trouble with.

I took your advice and I am trying to get it up and running with QTimer but am really confused on how it can solve the issue.

The timer will tick. Each time you make one attempt to read or whatever the file. If that succeeds you do your processing, if it fails you increment a member counter and it will try again on next tick. The point is that doing one try each time on a QTimer allows the UI to remain responsive; while going while True: sleep() blocks the UI.

How is the signal emitted?

How is what signal emitted? The QTimer? It just emits it on each interval expiry.

How do I connect it to the init() method in main.py?

What? Only classes have __init__() method, main.py is not a class. Create a QTimer in, say, ObserverWrapper, or perhaps in NewFileHandler. Outline:

self.retries = 0
self.timer = QTimer()
self.timer.timeout.connect(self.timerExpired)

def timerExpired(self):
    result = file_handler.checkOnceForFile()
    if result:
        # do whatever with successful result
        # for you this might be `emit` your own signal with the file information as parameter
        return
    # unsuccessful result
    self.retries++
    if self.retries > rery_limit:
        raise("Too many retries exception")
    # if we get here do nothing other than return
    # this is a failed retry, let it retry again next time the time expires

Notes from your code:

        timer = QTimer(self)
        # Add a method with the timer
        timer.timeout.connect(self.wait_for_file())

timer is a local variable. You should understand enough Python to realise this will go out of scope and be destroyed as soon as __init__() exits.

class ObserverWrapper(QMainWindow):
    """Encapsulated Observer boilerplate"""

This seems a completely wrong conceptual architecture. ObserverWrapper is not an instance of a QMainWindow, it's a utility class for doing some file observation.

localthink

@JonB

@JonB said in How to use Qt to keep a class open and emit a signal:

What? Only classes have __init__() method, main.py is not a class. Create a QTimer in, say, ObserverWrapper, or perhaps in NewFileHandler.

Whoops I meant DesignerMainWindow which is in a file called main.py. I updated the post.

This seems a completely wrong conceptual architecture. ObserverWrapper is not an instance of a QMainWindow, it's a utility class for doing some file observation.

Right but I thought I needed QTimer directly for replacing the 3500 and .01 parts of the try.

So in your example would timeExpired() be there to replace wait_for_file()?

JonB

@localthink
My timerExpired() calls file_handler.checkOnceForFile() or whatever. As the name suggests, that will (presumably) have the essential file = h5py.File(file_path, "r") from your wait_for_file(). But it's no longer a wait for a file: no loop, no retries, no sleep, just check once and return the result.

localthink

@JonB Ah. I understand. I am not sure this would work but perhaps it might. Essentially the whole point of the try part is due to the face that the hdf5 files take roughly 20 seconds to finish writing. Once one is done writing, the hdf5 file is ready to be sent to on_finished_run(). How would I be able to continue checking if a hdf5 file is done writing?

Edit: Also, the hdf5 files come in one-by-one every 30 seconds or so but at the end there will be thousands in the folder. Once an hdf5 file is ready to write and send to on_finished_run it won't be used again. Not sure if this is helpful info but I'm just trying to paint a picture of why I used the current setup.

JonB

@localthink said in How to use Qt to keep a class open and emit a signal:

due to the face that the hdf5 files take roughly 20 seconds to finish writing

You have never said anything about this. I am not going to keep answering and asking. Only you know how it works. Is there a statement in the code of yours you have shown which takes 20 seconds? I don't think so. Maybe the other side which writes it takes 20 seconds, I don't know. How is that relevant? What has it got to do with try?

How would I be able to continue checking if a hdf5 file is done writing?

I don't understand this. Plus I have said multiple times I don't know about a Python a watchdog, and I really don't care,

So far as I can the following 3 lines of code

                file = h5py.File(file_path, "r")
                file.close()
                return file_path, file_name

are what checks to see whatever you want to test. Either they hit their return, for success, or they do not, and go round the loop again, presumably following the except OSError route to retry and sleep. But only you know. If I am right I do not see any relevance to your question/last post, and you're just confusing me further.....

Does just the following code:

            try:
                file = h5py.File(file_path, "r")
                file.close()
                return file_path, file_name
            except OSError:
                return "", ""

do a single test of whether the file is currently available, yes or no? And does this test return immediately (not 30 seconds), yes or no? Assuming yes to both, if you use a timer to call it 100 times per second or whatever and it keeps returning "", "", does that indeed do 100 retries for you, yes or no? And if the caller counts the retries and abandons after 3,500 failure-calls, does that behave as you want and as your old, blocking code did, yes or no?

And if by any chance you mean file = h5py.File(file_path, "r") --- which you know I know nothing about --- sits and takes 30 seconds before returning (when something else is writing to the file, presumably), then you really, really should have said so.....