yes, that’s the error i got

there is a semaphore warning, not sure if it’s related

Can you resend it?
Is the Task marked as closed when the process ends ?

pip install clearml==1.0.3rc1

in my case, we need to evaluate the result across many random seeds, so each task needs to log the result independently.

Hi @<1523710701275713536:profile|PompousHawk82> , can you try with the latest RC?

Can I send you a wheel to test ?

You can use this to reproduce

How do I reproduce it ? (all the processes are on the same machine?)

🤞

I think I found something relating to the issue on the subprocess not logging. Let me check if we can share something quickly

Actually, before i update the clearml pkg, the 0.17 version can log the console even i init it in the subprocess, but the problem is that it will only won log all 3 subprocesses

check on the iteration on the right side,
i tried to start the experiment few times, and sometimes, 1 or 2 of the experiment seems just won’t start

we need to evaluate the result across many random seeds, so each task needs to log the result independently.

Ohh that kind of makes sense to me 🙂
Yes I'm also getting:

/usr/local/lib/python3.6/multiprocessing/semaphore_tracker.py:143: UserWarning: semaphore_tracker: There appear to be 74 leaked semaphores to clean up at shutdown
  len(cache))

Not sure about that ...

Not sure on the cause but if you do:

mp.set_start_method('fork', force=True)

There is no semaphore leakage

yeah sure

now it has log, but only the initial one

So the subprocesses are not logged ?

Let me check, see what can be learned ...

Hmm let me check something

i can only see the init log

Let me know if it solved it, if it did I'll make sure we push the RC

Why are you initializing 3 diff Tasks ?

and one experiment takes 40 hours to run, so i let them run in parallel

seems the not logging problem is back

No sure, here is the code

Wouldn't it make sense to use a single one ?

No sure what O'm seeing here

Nice job

YEY