we need to evaluate the result across many random seeds, so each task needs to log the result independently.

Ohh that kind of makes sense to me 🙂
Yes I'm also getting:

/usr/local/lib/python3.6/multiprocessing/semaphore_tracker.py:143: UserWarning: semaphore_tracker: There appear to be 74 leaked semaphores to clean up at shutdown
  len(cache))

Not sure about that ...

Hi @<1523710701275713536:profile|PompousHawk82> , can you try with the latest RC?

Thanks!

i can only see the init log

there is a semaphore warning, not sure if it’s related

Can you resend it?
Is the Task marked as closed when the process ends ?

works most of time, this occurs only few times

Nice job

Hmm so the Task.init should be called on the main process, this way the subprocess knows the Task is already created (you can call Task.init twice to get the task object). I wonder if we somehow can communicate between the sub processes without initializing in the main one...

🤞

So what if i want three tasks running in parallel, should i Task.init in the main process and change the task name in the sub process?

seems the not logging problem is back

Let me know if you need some tests

pip install clearml==1.0.3rc1

one quick question is that do i need to do some task.close() at the end of each process?

No need, it should auto close it if you started it with Task.init (or the agent executed it)

No sure, here is the code

Let me know if it solved it, if it did I'll make sure we push the RC

Wouldn't it make sense to use a single one ?

now it has log, but only the initial one

So the subprocesses are not logged ?

Releasing an RC

yeah sure

Let me check, see what can be learned ...

Hmm let me check something

Not sure on the cause but if you do:

mp.set_start_method('fork', force=True)

There is no semaphore leakage

and one experiment takes 40 hours to run, so i let them run in parallel

And only the main one ?

in my case, we need to evaluate the result across many random seeds, so each task needs to log the result independently.

Can I send you a wheel to test ?