Actually, before i update the clearml pkg, the 0.17 version can log the console even i init it in the subprocess, but the problem is that it will only won log all 3 subprocesses

not sure if it’s the problem with map

Can get the result now but failed with this

yes, that’s the error i got

in my case, we need to evaluate the result across many random seeds, so each task needs to log the result independently.

now it has log, but only the initial one

i’m just curious about how does trains server on different nodes communicate about the task queue

works most of time, this occurs only few times

btw, i’m running the code with multiprocessing, so the log is duplicated 3 times

You can use this to reproduce

I'm checking it out today and see if I can put up something

unfortunately, no, when i try to click the link, there is nothing there https://demoapp.demo.clear.ml/projects/0d49bffcdaa441c2aa3224054737d0bd/experiments/26dd46ec11fd4f95ba522955820a8444/output/log

and one experiment takes 40 hours to run, so i let them run in parallel

i can only see the init log

but the solution in the answer doesn’t help cause when i do reverse with -R the server couldn’t be brought up

Let me know if you need some tests

working very well actually

when i run the code i send you the other day, logging is perfect, but at the end of the code there is a semaphore warning, not sure if it’s related

one quick question is that do i need to do some task.close() at the end of each process?

i tired to run trains-compose without -d to say the log,
trains-agent-services | trains_agent: ERROR: Connection Error: it seems api_server is misconfigured. Is this the TRAINS API server http://apiserver:8008 ?
trains-agent-services | http://192.5.53.86:8081 http://192.5.53.86:8080 http://apiserver:8008
I didn’t assign anything to TRAINS_HOST_IP, not sure if the apiserver:8008 caused the problem

i’ll finish my breakfast real quick and get in couple minuets

Appreciated it!

for example i have this experiments of 99 tasks, i’d like to pull all scaler data. How can i achieve this? Thank you!

I have two laptop, one is running ubuntu 20.04 and one is macos, both are running in my local network. I installed the server on ubuntu and ssh from mac to it to bring up the server then build up a tunnel using ssh -L

Here is the screenshot

Then access the 8008 through the tunnel

i’m trying to install it my lab server, but the same problem happen, when i try to create credentials, it say error but this time it give more info:
Error 301 : Invalid user id: id=f46262bde88b4928997351a657901d8b, company=d1bd92a3b039400cbafc60a7a5b1e52b