Actually, before i update the clearml pkg, the 0.17 version can log the console even i init it in the subprocess, but the problem is that it will only won log all 3 subprocesses

Hmm let me check something

Can I send you a wheel to test ?

yeah sure

Let me check that for a sec

and one experiment takes 40 hours to run, so i let them run in parallel

Let me check, see what can be learned ...

check on the iteration on the right side,
i tried to start the experiment few times, and sometimes, 1 or 2 of the experiment seems just won’t start

in my case, we need to evaluate the result across many random seeds, so each task needs to log the result independently.

we need to evaluate the result across many random seeds, so each task needs to log the result independently.

Ohh that kind of makes sense to me 🙂
Yes I'm also getting:

/usr/local/lib/python3.6/multiprocessing/semaphore_tracker.py:143: UserWarning: semaphore_tracker: There appear to be 74 leaked semaphores to clean up at shutdown
  len(cache))

Not sure about that ...

Hi @<1523710701275713536:profile|PompousHawk82> , can you try with the latest RC?

I think I found something relating to the issue on the subprocess not logging. Let me check if we can share something quickly

Releasing an RC

not sure if it’s the problem with map

btw, i’m running the code with multiprocessing, so the log is duplicated 3 times

working very well actually

yes, that’s the error i got

when i run the code i send you the other day, logging is perfect, but at the end of the code there is a semaphore warning, not sure if it’s related

No sure what O'm seeing here

Why are you initializing 3 diff Tasks ?

Thanks!

You can use this to reproduce

now it has log, but only the initial one

So the subprocesses are not logged ?

now it has log, but only the initial one

How do I reproduce it ? (all the processes are on the same machine?)

And only the main one ?

Not sure on the cause but if you do:

mp.set_start_method('fork', force=True)

There is no semaphore leakage

YEY