Actually, before i update the clearml pkg, the 0.17 version can log the console even i init it in the subprocess, but the problem is that it will only won log all 3 subprocesses
one quick question is that do i need to do some task.close() at the end of each process?
So what if i want three tasks running in parallel, should i Task.init in the main process and change the task name in the sub process?
and one experiment takes 40 hours to run, so i let them run in parallel
Let me check, see what can be learned ...
check on the iteration on the right side,
i tried to start the experiment few times, and sometimes, 1 or 2 of the experiment seems just won’t start
Wouldn't it make sense to use a single one ?
in my case, we need to evaluate the result across many random seeds, so each task needs to log the result independently.
we need to evaluate the result across many random seeds, so each task needs to log the result independently.
Ohh that kind of makes sense to me 🙂
Yes I'm also getting:
/usr/local/lib/python3.6/multiprocessing/semaphore_tracker.py:143: UserWarning: semaphore_tracker: There appear to be 74 leaked semaphores to clean up at shutdown
len(cache))
Not sure about that ...
Hi @<1523710701275713536:profile|PompousHawk82> , can you try with the latest RC?
No need, it should auto close it if you started it with Task.init (or the agent executed it)
I think I found something relating to the issue on the subprocess not logging. Let me check if we can share something quickly
btw, i’m running the code with multiprocessing, so the log is duplicated 3 times
when i run the code i send you the other day, logging is perfect, but at the end of the code there is a semaphore warning, not sure if it’s related
