btw, i’m running the code with multiprocessing, so the log is duplicated 3 times
works most of time, this occurs only few times
Hmm so the Task.init should be called on the main process, this way the subprocess knows the Task is already created (you can call Task.init twice to get the task object). I wonder if we somehow can communicate between the sub processes without initializing in the main one...
one quick question is that do i need to do some task.close() at the end of each process?
Not sure on the cause but if you do:
mp.set_start_method('fork', force=True)
There is no semaphore leakage
Wouldn't it make sense to use a single one ?
So what if i want three tasks running in parallel, should i Task.init in the main process and change the task name in the sub process?
check on the iteration on the right side,
i tried to start the experiment few times, and sometimes, 1 or 2 of the experiment seems just won’t start
Let me check, see what can be learned ...
How do I reproduce it ? (all the processes are on the same machine?)
now it has log, but only the initial one
So the subprocesses are not logged ?
we need to evaluate the result across many random seeds, so each task needs to log the result independently.
Ohh that kind of makes sense to me 🙂
Yes I'm also getting:
/usr/local/lib/python3.6/multiprocessing/semaphore_tracker.py:143: UserWarning: semaphore_tracker: There appear to be 74 leaked semaphores to clean up at shutdown
len(cache))
Not sure about that ...
Actually, before i update the clearml pkg, the 0.17 version can log the console even i init it in the subprocess, but the problem is that it will only won log all 3 subprocesses
I think I found something relating to the issue on the subprocess not logging. Let me check if we can share something quickly