Yeaaa I got it working!

Specific step in the pipeline. The main step (the experiment) is currently just a file with a Task.init ` and then the experiment code. I am wondering how to modify this code such that it can be run in the pipeline or as standalone.

Thank you. I am still having the issue. I verified that output_uri of Task.init works and also clearml-data with MinIO storage works, but the logger still throws errors

Installed packages:

` # Python 3.7.10 (default, Feb 26 2021, 18:47:35) [GCC 7.3.0]

absl-py==0.12.0
aiostream==0.4.2
attrs==20.3.0
cached-property==1.5.2
cffi==1.14.5
chardet==4.0.0
clearml==0.17.5
cython==0.29.22
dm-control==0.0.364896371
dm-env==1.4
dm-tree==0.1.5
fasteners==0.16
furl==2.1.0
future==0.18.2
glfw==2.1.0
gym==0.18.0
h5py==3.2.1
humanfriendly==9.1
idna==2.10
imageio-ffmpeg==0.4.3
importlib-metadata==3.7.3
jsonschema==3.2.0
labmaze==1.0.4
lxml==4.6.3
moviepy==1.0.3
mujoco-py==...

Both, actually. So what I personally would find intuitive is something like this:
` class Task:
def load_statedict(self, state_dict):
pass

async def synchronize(self):
    ...

async def task_execute_remotely(self):
    await self.synchronize()
    ...

def add_requirement(self, requirement):
    ...

@classmethod
async def init(task_name):
    task = Task()
    task.load_statedict(await Task.load_or_create(task_name))
    await tas...

AlertBlackbird30 Thanks for asking. Just take everything with I grain of salt I say, because I am also not sure whether I do machine learning the correct way 😄

I think you got the right idea. I actually do reinforcement learning (RL), so I have multiple RL-environments and RL-agents. However, while the code for the agents differs between the agents, the glue code is the same. So what I do is I call python run_experiment.py --agent http://myproject.agents.my ` _agent --environm...

I have no idea whether it is a user error or because of the clearml-server update...

Too late for what?

I mean if I do CLEARML_DOCKER_IMAGE=my_image clearml-task something something it will not work, right?

My clearml-server server crashed for some reason, so I won't be able to verify until tomorrow.

Oh you are right. I did not think this through... To implement this properly it gets to enterprisy for me, so I ll just leave it for now :D

I was wondering whether some solution is builtin in clearml, so I do not have to configure each server manually. However, from your answer I take that this is not the case.

Yea, something like this seems to be the best solution.

I see. Thank you!

What do you mean?

Latest version for everything. I will message you again, if I encounter this problem again.

It is not explained there, but do you mean
CLEARML_API_ACCESS_KEY: ${CLEARML_API_ACCESS_KEY:-} CLEARML_API_SECRET_KEY: ${CLEARML_API_SECRET_KEY:-}?

If you compare the two outputs it put at the top of this thread, the one being the output if executed locally and the other one being the output if executed remotely, it seems like command is different and wrong on remote.

` ocker-compose ps
Name Command State Ports

clearml-agent-services /usr/agent/entrypoint.sh Restarting
clearml-apiserver /opt/clearml/wrapper.sh ap ... Up 0.0.0.0:8008->8008/tcp, 8080/tcp, 8081/tcp ...

AgitatedDove14 Thank you, that explains it.

Here is the minimal example: https://github.com/mctigger/test_clearml

I mean, could my hard drive not become full at some point? Can clearml-agent currently detect this?

Full log

For me this does not work (at least with nested tqdm bars, did not try single ones yet).

When I passed specific arguments (for example --steps) it ignored them...

I am not sure what you mean by this. It should not ignore anything.

No. Here is a better example. I have two types of workstations: Type X can execute tasks of type A and B. Type Y can execute tasks of type B. This could be the case if type X workstations have for example more VRAM, newer drivers, etc...
I have two queues. Queue A and Queue B. I submit tasks of type A to queue A and tasks of type B to queue B.

Here is what can happen:
Enqueue the first task of type B. Workstations of type X will run this task. Enqueue the second task of type A. Workstation ...

I got some warnings about broken packages. I cleaned the conda cache with conda clean -a ` and now it installed fine!

Is ther a way to see the contents of /tmp/conda_envaz1ne897.yml ? Seems to be deleted after the task is finihsed

Perfect, thanks a lot!

Sure, I ll try this