How to use Qt to keep a class open and emit a signal
-
I am trying to send a
.hdf5
file to a method in the main python file in the class here:class DesignerMainWindow(QtGui.QMainWindow, Ui_MainWindow): """Customization for Qt Designer created window""" signal_output_log = QtCore.Signal("QString") sig_clear_log = QtCore.Signal() def __init__(self, parent=None): super(DesignerMainWindow, self).__init__(parent) self.setupUi(self) self.image_analyzer = ImageAnalyzer(self) self.listener = watchdog_search.ObserverWrapper("/home/Test_Data/") self.on_finished_run(self.listener.wait_for_file()) def on_finished_run(self, tuple: ()): self.image_analyzer.load_image(str(tuple[0]), str(tuple[1]), from_remote=True)
The
.hdf5
file comes from this `watchdog_search.py:import time import traceback import os import h5py import queue from typing import Union from watchdog.observers import Observer from watchdog.events import FileSystemEventHandler, DirCreatedEvent, FileCreatedEvent class NewFileHandler(FileSystemEventHandler): """h5 file creation handler for Watchdog""" def __init__(self): self.file_queue = queue.Queue() # callback for File/Directory created event, called by Observer. def on_created(self, event: Union[DirCreatedEvent, FileCreatedEvent]): if event.src_path[-4:] == "hdf5": # run callback with path string self.file_queue.put(event.src_path) class ObserverWrapper: """Encapsulated Observer boilerplate""" def __init__(self, path: str, recursive=True): self.path = path self.recursive = recursive self.observer = Observer() self.handler = NewFileHandler() self.observer.schedule(self.handler, path=path, recursive=recursive) self.start() def start(self): """ Starts observing for filesystem events. Runs self.routine() every 1 second. :param blocking: If true, blocks main thread until keyboard interrupt. """ self.observer.start() def stop(self): """ Stops the observer. When running self.start(blocking=True) then you don't need to call this. """ self.observer.stop() self.observer.join() def wait_for_file(self): """ Wait and Process newly created files """ max_retry_count = 3500 # for test purposes now but want to set an upper bound on verifying a file is finished. # will try h5 file for a max of 35 seconds (upper bound) to see if the file is finished. # Files are usually finished within 20-30 seconds # retry_interval_seconds = .01 # every hundreth it will try the file to see if it finished writing # wait for file to be added #print(self.handler.file_queue.get(block=True)) file_path = self.handler.file_queue.get(block=True) file_name = os.path.basename(file_path) # try to open the file retry_count = 0 while True: try: file = h5py.File(file_path, "r") file.close() return file_path, file_name except OSError: if retry_count < max_retry_count: retry_count += 1 print(f"h5 file <{file_path}> is locked, retrying {retry_count}/{max_retry_count}") time.sleep(retry_interval_seconds) else: print(f"h5 file <{file_path}> reached max retry count, skipping") except Exception as err: print(f"Got unexpected Error <{type(err).__name__}> while opening <{file_path}> ") traceback.print_exc()
Currently I call this file by
self.listener = watchdog_search.ObserverWrapper("/path/to/folder/of/interest")
in the
main.py
file but this only sends one hdf5 file and doesn't send any more. Watchdog needs to stay open and send content tomain.py
every time there is a new hdf5 file available. Does anyone know how to use Qt to do this? This is not a question of watchdog but rather a question of asynchronous programming and Qt. I need at the end to send thefile_path
andfile_name
of the.hdf5
toon_finished_run()
or perhaps haveon_finished_run()
be the subscriber for the signal but I am unsure how to do this when thetry
returns not the emitted signal but rather the file path and name. Any insight is greatly appreciated. -
@localthink
I don't know much about this because all of your code inwatchdog_search.py
is Python/watchdog, whatever that is, and has nothing to do with Qt. But I believe you have 3 possible approaches:-
As long as you have
while True:
or lots ofretry_count
ortime.sleep()
you will need to put the watchdog stuff in a thread to avoid blocking/allow run the Qt main UI event loop. -
If this is something you can do once regularly on a timer --- i.e polling --- you can get rid of your blocking loop and use Qt's
QTimer
to do the polls. No threads, no blocking. -
If your watchdog does something about looking to see whether a file exists intermittently Qt has its own
QFileSystemWatcher
instead which again does not block.
when the
try
returns not the emitted signal but rather the file path and nameIf you do end up using this code: call it from class and use the return result as parameters to a Qt signal. I think the Python syntax would be something like:
file_path, file_name = something.wait_for_file() self.some_signal_name.emit(file_path, file_name)
The signal emitting class must inherit from
QObject
(either solely, or you could use Python multiple inheritance). -
-
@JonB Thanks for the reply.
- I think putting watchdog in a thread would be nice. I've been reading up on this but have failed to get it to work.
- No it won't be done once but rather a lot of times so I don't think this is a feasible option.
- Watchdog just checks if a new file is populated in the folder. When a new file is populated, the
NewFileHandler.on_created()
checks if it is an hdf5 file. So I wouldn't be able to check what the name of the file is or any of that but rather do post processing in this sense.
If you do end up using this code: call it from class and use the return result as parameters to a Qt signal. I think the Python syntax would be something like:
Call what from class exactly?
-
@localthink said in How to use Qt to keep a class open and emit a signal:
I think putting watchdog in a thread would be nice. I've been reading up on this but have failed to get it to work.
This would be my last choice. Especially if I were new I would advise trying to keep away from threads if possible.
I think either of the other two would be preferable.
Call what from class exactly?
The two lines of code I wrote, which use
self.some_signal_name.emit()
. The class which is emitting the signal. -
@JonB So would the signal in this case be
file_name
andfile_path
and they get sent to the methodon_finished_run()
?I've been trying to play around with this the last couple of weeks (emit and signal) and this is what I have so far but I'm just not making progress on it:
class DesignerMainWindow(QtGui.QMainWindow, Ui_MainWindow): """Customization for Qt Designer created window""" signal_output_log = QtCore.Signal("QString") sig_clear_log = QtCore.Signal() def __init__(self, parent=None): super(DesignerMainWindow, self).__init__(parent) self.setupUi(self) self.image_analyzer = ImageAnalyzer(self) self.listener = watchdog_search.ObserverWrapper("/home/Test_Data/") self.handler.bridge.created.connect(self.on_finished_run) def on_finished_run(self, tuple: ()): self.image_analyzer.load_image(str(tuple[0]), str(tuple[1]), from_remote=True)
and from
watchdog_search.py
import time import traceback import os import h5py import queue from typing import Union from watchdog.observers import Observer from watchdog.events import FileSystemEventHandler, DirCreatedEvent, FileCreatedEvent from .tools.qt import QtCore class Bridge(QtCore.QObject): created = QtCore.Signal(FileCreatedEvent) class NewFileHandler(FileSystemEventHandler): """h5 file creation handler for Watchdog""" def __init__(self): super().__init__() #self.file_queue = queue.Queue() self.bridge = Bridge() # callback for File/Directory created event, called by Observer. def on_created(self, event: Union[DirCreatedEvent, FileCreatedEvent]): if event.src_path[-4:] == "hdf5": # run callback with path string #self.file_queue.put(event.src_path) self.bridge.created.emit(event.src_path) class ObserverWrapper: """Encapsulated Observer boilerplate""" def __init__(self, path: str):#, recursive=True): self.path = path #self.recursive = recursive self.observer = Observer() self.handler = NewFileHandler() self.observer.schedule(self.handler, path=path, recursive=True) self.start() def start(self): """ Starts observing for filesystem events. Runs self.routine() every 1 second. :param blocking: If true, blocks main thread until keyboard interrupt. """ self.observer.start() def stop(self): """ Stops the observer. When running self.start(blocking=True) then you don't need to call this. """ self.observer.stop() self.observer.join() def wait_for_file(self): """ Wait and Process newly created files """ max_retry_count = 3500 # for test purposes now but want to set an upper bound on verifying a file is finished. # will try h5 file for a max of 35 seconds (upper bound) to see if the file is finished. # Files are usually finished within 20-30 seconds # retry_interval_seconds = .01 # every hundreth it will try the file to see if it finished writing # wait for file to be added #print(self.handler.file_queue.get(block=True)) #print(self.handler.on_created(self.path)) #file_path = #print(self.handler.file_queue.get(block=True)) #file_name = os.path.basename(file_path) # try to open the file retry_count = 0 while True: try: file = h5py.File(file_path, "r") file.close() return file_path, file_name except OSError: if retry_count < max_retry_count: retry_count += 1 print(f"h5 file <{file_path}> is locked, retrying {retry_count}/{max_retry_count}") time.sleep(retry_interval_seconds) else: print(f"h5 file <{file_path}> reached max retry count, skipping") except Exception as err: print(f"Got unexpected Error <{type(err).__name__}> while opening <{file_path}> ") traceback.print_exc()
-
@localthink said in How to use Qt to keep a class open and emit a signal:
and this is what I have so far but I'm just not making progress on it:
I'm sorry but I don't know what this means. Do you have a specific issue/question?
self.handler.bridge.created.connect()
: looks quite involved! Maybe your code would be easier if you didclass NewFileHandler(FileSystemEventHandler, QtCore.QObject):
so you can do the signals inside that method instead of needing theclass Bridge
?You have chosen to go down your current route, with blocking calls to
wait_for_file()
. I said that would be my least favourite approach. Rewriting it to do one try each time off aQTimer
would seems preferable to me. However as I said if you maintain this way you will need to use threads, which I have said I would avoid, yet I do not see any threads in your code. -
@JonB Thanks for the reply. Yea I would prefer to do the easy route. I've been trying the third option with
QFileSystemWatcher
and I don't really believe it can do what watchdog is doing unfortunately. I am unsure how you propose usingQtimer
exactly. Could you help me understand what you mean with option two?Edit: couldn't I just make
block=False
in watchdog and there will then be no blocking? -
@localthink said in How to use Qt to keep a class open and emit a signal:
Edit: couldn't I just make block=False in watchdog and there will then be no blocking?
How do I know if I know nothing about watchdog, as I said? But if that avoids blocking and therefore the need for a thread it sounds preferable.
I am unsure how you propose using
Qtimer
exactly.So far as i understand your code,
wait_for_file()
does/can sit and try 3,500 times withsleep()
for 1/100th of a second between retries. If you only make it do one (or possibly a very small number, like 10 for 1/10th of a second delay) call at a time, and return either the file whatever or failure, then you could just call that repeatedly from aQTimer
with a corresponding interval, doing your own member variable for counting up the total retries till you are fed up of waiting. By usingQTimer
like this you allow the main UI thread to keep running responsively without the need to introduce any threading. -
@JonB The 3500 times is just arbitrary and will be changed to something smaller. I took your advice and I am trying to get it up and running with
QTimer
but am really confused on how it can solve the issue. Some questions that come up are: How is the signal emitted? How do I connect it to the__init__()
method inDesignerMainWindow
?Here is the code I am working with now and I am not sure how
QTimer
could help from here:class NewFileHandler(FileSystemEventHandler): """h5 file creation handler for Watchdog""" def __init__(self): super().__init__() self.file_queue = queue.Queue() # self.bridge = Bridge() # callback for File/Directory created event, called by Observer. def on_created(self, event: Union[DirCreatedEvent, FileCreatedEvent]): if event.src_path[-4:] == "hdf5": # run callback with path string self.file_queue.put(event.src_path) # self.bridge.created.emit(event.src_path) class ObserverWrapper(QMainWindow): """Encapsulated Observer boilerplate""" def __init__(self, path: str):#, recursive=True): super().__init__() self.path = path #self.recursive = recursive self.observer = Observer() self.handler = NewFileHandler() self.observer.schedule(self.handler, path=path, recursive=True) self.start() self.counter = 0 timer = QTimer(self) # Add a method with the timer timer.timeout.connect(self.wait_for_file()) def start(self): """ Starts observing for filesystem events. Runs self.routine() every 1 second. :param blocking: If true, blocks main thread until keyboard interrupt. """ self.observer.start() def stop(self): """ Stops the observer. When running self.start(blocking=True) then you don't need to call this. """ self.observer.stop() self.observer.join() def wait_for_file(self): """ Wait and Process newly created files """ max_retry_count = 3500 # for test purposes now but want to set an upper bound on verifying a file is finished. # will try h5 file for a max of 35 seconds (upper bound) to see if the file is finished. # Files are usually finished within 20-30 seconds # retry_interval_seconds = .01 # every hundreth it will try the file to see if it finished writing # wait for file to be added #print(self.handler.file_queue.get(block=True)) #print(self.handler.on_created(self.path)) #file_path = #print(self.handler.file_queue.get(block=True)) #file_name = os.path.basename(file_path) file_path = self.handler.file_queue.get(block=True) file_name = os.path.basename(file_path) # try to open the file # retry_count = 0 while True: try: file = h5py.File(file_path, "r") file.close() return file_path, file_name except OSError: if self.counter < max_retry_count: self.counter += 1 #retry_count += 1 print(f"h5 file <{file_path}> is locked, retrying {self.counter}/{max_retry_count}") time.sleep(retry_interval_seconds) else: print(f"h5 file <{file_path}> reached max retry count, skipping") except Exception as err: print(f"Got unexpected Error <{type(err).__name__}> while opening <{file_path}> ") traceback.print_exc()
-
@localthink
Sorry but I don't know what you are having trouble with.I took your advice and I am trying to get it up and running with QTimer but am really confused on how it can solve the issue.
The timer will tick. Each time you make one attempt to read or whatever the file. If that succeeds you do your processing, if it fails you increment a member counter and it will try again on next tick. The point is that doing one try each time on a
QTimer
allows the UI to remain responsive; while goingwhile True: sleep()
blocks the UI.How is the signal emitted?
How is what signal emitted? The
QTimer
? It just emits it on each interval expiry.How do I connect it to the init() method in main.py?
What? Only classes have
__init__()
method,main.py
is not a class. Create aQTimer
in, say,ObserverWrapper
, or perhaps inNewFileHandler
. Outline:self.retries = 0 self.timer = QTimer() self.timer.timeout.connect(self.timerExpired) def timerExpired(self): result = file_handler.checkOnceForFile() if result: # do whatever with successful result # for you this might be `emit` your own signal with the file information as parameter return # unsuccessful result self.retries++ if self.retries > rery_limit: raise("Too many retries exception") # if we get here do nothing other than return # this is a failed retry, let it retry again next time the time expires
Notes from your code:
timer = QTimer(self) # Add a method with the timer timer.timeout.connect(self.wait_for_file())
timer
is a local variable. You should understand enough Python to realise this will go out of scope and be destroyed as soon as__init__()
exits.class ObserverWrapper(QMainWindow): """Encapsulated Observer boilerplate"""
This seems a completely wrong conceptual architecture.
ObserverWrapper
is not an instance of aQMainWindow
, it's a utility class for doing some file observation. -
@JonB said in How to use Qt to keep a class open and emit a signal:
What? Only classes have
__init__()
method,main.py
is not a class. Create aQTimer
in, say,ObserverWrapper
, or perhaps inNewFileHandler
.Whoops I meant
DesignerMainWindow
which is in a file called main.py. I updated the post.This seems a completely wrong conceptual architecture. ObserverWrapper is not an instance of a QMainWindow, it's a utility class for doing some file observation.
Right but I thought I needed
QTimer
directly for replacing the 3500 and .01 parts of thetry
.So in your example would
timeExpired()
be there to replacewait_for_file()
? -
@localthink
MytimerExpired()
callsfile_handler.checkOnceForFile()
or whatever. As the name suggests, that will (presumably) have the essentialfile = h5py.File(file_path, "r")
from yourwait_for_file()
. But it's no longer a wait for a file: no loop, no retries, no sleep, just check once and return the result. -
@JonB Ah. I understand. I am not sure this would work but perhaps it might. Essentially the whole point of the
try
part is due to the face that the hdf5 files take roughly 20 seconds to finish writing. Once one is done writing, the hdf5 file is ready to be sent toon_finished_run()
. How would I be able to continue checking if a hdf5 file is done writing?Edit: Also, the hdf5 files come in one-by-one every 30 seconds or so but at the end there will be thousands in the folder. Once an hdf5 file is ready to write and send to
on_finished_run
it won't be used again. Not sure if this is helpful info but I'm just trying to paint a picture of why I used the current setup. -
@localthink said in How to use Qt to keep a class open and emit a signal:
due to the face that the hdf5 files take roughly 20 seconds to finish writing
You have never said anything about this. I am not going to keep answering and asking. Only you know how it works. Is there a statement in the code of yours you have shown which takes 20 seconds? I don't think so. Maybe the other side which writes it takes 20 seconds, I don't know. How is that relevant? What has it got to do with
try
?How would I be able to continue checking if a hdf5 file is done writing?
I don't understand this. Plus I have said multiple times I don't know about a Python a watchdog, and I really don't care,
So far as I can the following 3 lines of code
file = h5py.File(file_path, "r") file.close() return file_path, file_name
are what checks to see whatever you want to test. Either they hit their
return
, for success, or they do not, and go round the loop again, presumably following theexcept OSError
route to retry and sleep. But only you know. If I am right I do not see any relevance to your question/last post, and you're just confusing me further.....Does just the following code:
try: file = h5py.File(file_path, "r") file.close() return file_path, file_name except OSError: return "", ""
do a single test of whether the file is currently available, yes or no? And does this test return immediately (not 30 seconds), yes or no? Assuming yes to both, if you use a timer to call it 100 times per second or whatever and it keeps returning
"", ""
, does that indeed do 100 retries for you, yes or no? And if the caller counts the retries and abandons after 3,500 failure-calls, does that behave as you want and as your old, blocking code did, yes or no?And if by any chance you mean
file = h5py.File(file_path, "r")
--- which you know I know nothing about --- sits and takes 30 seconds before returning (when something else is writing to the file, presumably), then you really, really should have said so.....