Ah, very cool! Then I will try this, too.

Okay, thanks for the info! I am currently not using k8s, but may be good to know for the future.

Or alternatively I just saw that Task.create takes a requirements.txt as an argument. This would also be fine for me, however I am not sure whether I should use Task.create ?

I guess the supported storage mediums (e.g. S3, ceph, etc...) dont have this issue, right?

Here is some context on what I am currently trying to do (pseudocode):

`
def run_experiment(args):
...

def get_task_experiment():
task = Task.init(...)
task.bind_run(run_experiment)

return task

def run_with_pipeline(task):
pipe = PipelineController(...)
pipe.add_step(prepare_something...)
pipe.add_step(task)
pipe.add_step(postprocess_something...)

return pipe 

if name == "main":
task = get_task_experiment()
# Run without Pipeline
if ...

Can you tell me how I create tasks correctly? The PipelineController.add_step takes the task-id/task-name, but I would rather just define a function that returns the task directly, since the base-task may not be already on the clearml-server.

I have to correct myself, I do not even have CUDA installed. Only the driver and everything CUDA-related is provided by the docker container. This works with a container that has CUDA 11.4, but now I have one with 11.6 (latest nvidia pytorch docker).

However, even after changing the clearml.conf and overriding with CUDA_VERSION, the clearml-agent prints on the docker container agent.cuda_version = 114 ! (Other changes to the clearml.conf on the agent are reflected in the docker, so only...

The default behavior mimics Python’s assert statement: validation is on by default, but is disabled if Python is run in optimized mode (via python -O). Validation may be expensive, so you may want to disable it once a model is working.

But would storing then work?

AgitatedDove14 I have to problem that "debug samples" are not shown anymore after running many iterations. What's appropriate to use here: A colleague told me increasing task_log_buffer_capacity worked. Is this the right way? What is the difference to file_history_size ?

Any idea why deletion of artifacts on my second fileserver does not work?

fileserver_datasets: networks: - backend - frontend command: - fileserver container_name: clearml-fileserver-datasets image: allegroai/clearml:latest restart: unless-stopped volumes: - /opt/clearml/logs:/var/log/clearml - /opt/clearml/data/fileserver-datasets:/mnt/fileserver - /opt/clearml/config:/opt/clearml/config ports: - "8082:8081"
ClearML successfu...

Latest version for everything. I will message you again, if I encounter this problem again.

Okay, thank you 🙂

Yea, that I knew 😄 But somehow I didn't think about the clearml.conf

One question: Does clearml resolve the CUDA Version from driver or conda?

I just tested with remote_execution and the problem seems to exist there, too. It is just that when the task switches from local to remote execution (i.e. exists the local script) the local scalars will appear, but no scalar of remote execution will show up. So also the iteration will not update. However, at least for remote execution I get live console output.

When I change the owner and the group of the files to root it works.

Same error 😕

Yea, I don't do anything manually.

So clearml 1.0.1 clearml-agent 1.0.0 and clearml-server from master

Thanks for the answer. So currently the cleanup is done based number of experiments that are cached? If I have a few big experiments, this could make my agents cache overflow?

agent-forwarding is working just like your described here: https://github.com/allegroai/clearml-agent/issues/45 Looking forward to not having to use the absolute path in the future 🙂

Thank you. The reports feature is super cool! Greetings to the team. One of the best features for educational use!

By preexisting task I meant I have existing code that already uses Task.init . I would like to use this code as my main task in my pipeline, i.e. after carla started.

@<1523701205467926528:profile|AgitatedDove14> Thank you very much for your guidance. Setting these manually works for me!

Latter

I have set default_output_uri to s3://my_minio_instance:9000/clearml

If I set files_server to s3://my_minio_instance:9000 /bucket_that_does_not_exist it fails at uploading metrics, but model upload still works:

WARNING - Failed uploading to s3://my_minio_instance:9000/ bucket_that_does_not_exist ('NoneType' object has no attribute 'upload')

clearml.Task - INFO - Completed model upload to s3://my_minio_instance:9000/clearml

What is ` default_out...

Ah, thank you very much

Installed packages:

` # Python 3.7.10 (default, Feb 26 2021, 18:47:35) [GCC 7.3.0]

absl-py==0.12.0
aiostream==0.4.2
attrs==20.3.0
cached-property==1.5.2
cffi==1.14.5
chardet==4.0.0
clearml==0.17.5
cython==0.29.22
dm-control==0.0.364896371
dm-env==1.4
dm-tree==0.1.5
fasteners==0.16
furl==2.1.0
future==0.18.2
glfw==2.1.0
gym==0.18.0
h5py==3.2.1
humanfriendly==9.1
idna==2.10
imageio-ffmpeg==0.4.3
importlib-metadata==3.7.3
jsonschema==3.2.0
labmaze==1.0.4
lxml==4.6.3
moviepy==1.0.3
mujoco-py==...

It seems to work when I enable conda_freeze .