Sounds like a good hack, but not like a good solution 😄 But thank you anyways! 🙂

Another example on what I would expect:
` ### start_carla.py
def get_task():
task = Task.init(project_name="examples", task_name="start-carla", task_type="application")
# experiment is not run here. The experiment is only run when this is executed as standalone or on a clearml-agent.
return task

def run_experiment(task):
...

This task can also be run as standalone or run by a clearml-agent

if name == "main":
task = get_task()
run_experiment(task)

run_pi...

In the UI

Btw: I think Task.init is more confusing than Task.create and I would rather rename the former.

Or alternatively I just saw that Task.create takes a requirements.txt as an argument. This would also be fine for me, however I am not sure whether I should use Task.create ?

However, this seems like a pretty edge-case to me - why would you do that on a regular basis?

For me this is how I use ClearML as tensorboard replacement. To start some debug runs before adding it to a clearml-agent queue. For me this seems like the most common usage case or am I missunderstanding ClearML?

Just tested it again. Here is my config:
https://gist.github.com/mctigger/086c5f8071a604605e9f7a172800b51d
In the Web UI under Configuration -> Hyper Parameters -> Environment I can see the following:
MUJOCO_GL osmesa

Maybe the problem is that I do not start my docker containers from the root user, so 1001 is a mapping inside the docker to my actual user. Could it be that on the host the owner if your .ssh files is called root ?

Sure, let me try.

It is only a single agent that is sending a single artifact. server-->agent is fast, but agent-->server is slow.

What exactly do you mean by that?

Unfortunately, not. Quick question: Is there caching happening somewhere besides .clearml ? Does the boto3 driver create cache?

Ok, nice. Makes sense!

Or there should be an early error for trying to run conda based tasks on pip agents

Afaik, clearml-agent will use existing installed packages if they fit the requirements.txt. E.g. pytorch >= 1.7 will only install PyTorch if the environment does not already provide some version of PyTorch greater or equal to 1.7.

Thanks! I am fascinated by what you guys offer with clearml 🙂

Not for the logger, I just tested it.

I installed as told on pytorch.org : pip3 install --pre torch torchvision torchaudio --index-url None

Latter

Thank you, gonna try that!

But you can manually add them with Task.add_requirements, no?

In my opinion an ugly solution. I would have to keep track of which requirements are missing. Then I would rather just add all requirements manually.

I will create a minimal example.

Can you maybe also tell me which docker image you used? For me this is all not working unfortunately

Thank you. Seems like someone implemented a type check Error: Dataset id=8d7355655830427f9243671c8cf0a6b0 is not of type Dataset :)

I am also wondering how I integrate my (preexisting) main task in the pipeline. I start my main task like this: python my_script.py --myarg "myargs" . How are the arguments captured? I am very confused, how one integrates this correctly...

Could you elaborate on that:
"So the agent failed to actually restore it from the git (files that are not added are not considered part of the git diff, this is usually git behavior)."

Or I am using this method wrong.

I am still trying to solve the add_requirements + importlib combo. If I use detect_with_freeze I can not use add_requirements and if I use automatic code analysis it will not find all packages because of importlib .
For now I come to the conclusion, that keeping a requirements.txt and making clearml parse the requirements from there should be the most robust solution. Unfortunately, there seems to be no way to do this with Task.init .

What I get for args when I print it locally is not the same as what ClearML extracts from args .

It is server version 1.0 and everything that came with it.