No sure, here is the code

Actually, before i update the clearml pkg, the 0.17 version can log the console even i init it in the subprocess, but the problem is that it will only won log all 3 subprocesses

there is a semaphore warning, not sure if it’s related

Can you resend it?
Is the Task marked as closed when the process ends ?

No need, it should auto close it if you started it with Task.init (or the agent executed it)

So what if i want three tasks running in parallel, should i Task.init in the main process and change the task name in the sub process?

now it has log, but only the initial one

So the subprocesses are not logged ?

Hmm so the Task.init should be called on the main process, this way the subprocess knows the Task is already created (you can call Task.init twice to get the task object). I wonder if we somehow can communicate between the sub processes without initializing in the main one...

Hmm let me check something

Releasing an RC

btw, i’m running the code with multiprocessing, so the log is duplicated 3 times

works most of time, this occurs only few times

Why are you initializing 3 diff Tasks ?

Thanks!

now it has log, but only the initial one

You can use this to reproduce

working very well actually

we need to evaluate the result across many random seeds, so each task needs to log the result independently.

Ohh that kind of makes sense to me 🙂
Yes I'm also getting:

/usr/local/lib/python3.6/multiprocessing/semaphore_tracker.py:143: UserWarning: semaphore_tracker: There appear to be 74 leaked semaphores to clean up at shutdown
  len(cache))

Not sure about that ...

How do I reproduce it ? (all the processes are on the same machine?)

i can only see the init log

Can I send you a wheel to test ?

Not sure on the cause but if you do:

mp.set_start_method('fork', force=True)

There is no semaphore leakage

not sure if it’s the problem with map

Wouldn't it make sense to use a single one ?

yes, that’s the error i got

And only the main one ?

Nice job

Let me know if it solved it, if it did I'll make sure we push the RC

Hi @<1523710701275713536:profile|PompousHawk82> , can you try with the latest RC?