Sorry, just revisiting this as I'm only getting around to implementation now. How do you pass the ECR container ID to the defined task?

ECR access should be enabled as part of the role the agent instance assumes when it runs a task

Yes it does 🙂 I suspected this was the process. Thanks Jake. One last question, more so about the architecture design - is it advised to have the clearml server instance and a 'worker' instance listening to the queue as separate remote machines, or can I use the same instance for the web UI and and as a worker? I understand that processing pipelines may be compute intense enough to consume all resources and break the web UI, but I was wondering whether using a single large instance is a po...

Reason I am asking is because we have servers with large RAM capacity, but minimal storage capacity, meaning that objects held in memory can sometimes surpass storage capacity if export is required

Did the shell script route work? I have a similar question.

It's a little more complicated because the index URL is not fixed; it contains the token which is only valid for a max of 12 hours. That means the ~/.config/pip/pip.conf file will also need to be updated every 12 hours. Fortunately, this editing is done automatically by authenticating AWS codeartefact in the command line by logging in.

My current thinking is as follows:

Install the awscli - pip install awscli (c...

Awesome, thank you. I will give that a try later this week and update if it worked as expected! May finally solve my private dependencies issue 😂

By script, you mean entering these two lines separately as a list for that extra_docker_shell_scripts arugment?

One question - you can also set the agent.package_manager.extra_index_url , but since this is dynamic, will pip install still add the extra index URL from the pip config file? Or does it have to be set in this agent config variable?

Sounds good to me. Thanks Martin 🙂

Yeah, it's not urgent. I will change the labels around to avoid this error 🙂 thanks for checking!

No worries, happy to help with the bug hunt 😄

Any news on this bug?

So how do I ensure that artefacts are uploaded in the correct bucket from within clearml?

Thanks maestro. Will give this a go

No rush, thanks for looking into it and sorry for tagging everyone here 😂

I don't even know if I have a valid concern for this. Just a little worried as airflow is accessible by more departments than just DS, which could result in some disasters

I dont think its that. its a 20kb file upload. This was the last message just printed
ClearML Monitor: Could not detect iteration reporting, falling back to iterations as seconds-from-star

I think there is more complexity to what I am trying to achieve, but this will be a good start. Thanks!

I will need to log data set ID, transformer (not the NN architecture, just a data transformer), the model (with all hyperparameters & metadata) etc. and how all things link

/home/ubuntu/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/clearml/utilities/plotlympl/mpltools.py:371: MatplotlibDeprecationWarning: The is_frame_like function was deprecated in Matplotlib 3.1 and will be removed in 3.3.This is the last print statement before it hangs

How we resolved this issue is by developing a package to deal with the connecting and querying to databases. This package is then used inside the task for pulling the data from the data warehouse. There is a devops component here for authorising access to the relevant secret (we used SecretsManager on AWS). The clearml-agent instances are launched with role permissions which allow access to the relevant secrets. Hope that is helpful to you

great, thanks! will give that a go

SuccessfulKoala55 thanks for your help as always. I will try to create a DAG on airflow using the SDK to implement some form of retention policy which removes things that are not necessary. We independently store metadata on artefacts we produce, and mostly use clearml as the experiment manager, so a lot of the events data can be cleared.

Just upgraded matplotlib, going to test now

ohhh ok. so I can actually remove this if those workers are no longer in use

yes it does, but that requires me to manually create a new agent every time I want to run a different env no?

The task is dependent on a few artefacts from another task. Is there anything else I can do here?

could it be how I am trying to log the figure manually?

I may just chunk the exports and see how I get on there 🙂 thanks for your help!

Oh great, thanks! Was trying to figure out how the method knows that the docker image ID belongs to ECR. Do you have any insight into that?

Upload how? Via SSH?