Hmm so the Task.init should be called on the main process, this way the subprocess knows the Task is already created (you can call Task.init twice to get the task object). I wonder if we somehow can communicate between the sub processes without initializing in the main one...
I think I found something relating to the issue on the subprocess not logging. Let me check if we can share something quickly
and one experiment takes 40 hours to run, so i let them run in parallel
No need, it should auto close it if you started it with Task.init (or the agent executed it)
now it has log, but only the initial one
So the subprocesses are not logged ?
btw, i’m running the code with multiprocessing, so the log is duplicated 3 times
one quick question is that do i need to do some task.close() at the end of each process?
in my case, we need to evaluate the result across many random seeds, so each task needs to log the result independently.
when i run the code i send you the other day, logging is perfect, but at the end of the code there is a semaphore warning, not sure if it’s related
Not sure on the cause but if you do:
mp.set_start_method('fork', force=True)
There is no semaphore leakage
there is a semaphore warning, not sure if it’s related
Can you resend it?
Is the Task marked as closed when the process ends ?
works most of time, this occurs only few times