Haha no not that much, I was just trying to play around with removing tasks etc, and didn't want to remove tasks created by co-workers.

Out of interest, is there a reason these are read-only? The code for these tasks is on github right?

I think there is more complexity to what I am trying to achieve, but this will be a good start. Thanks!

I will need to log data set ID, transformer (not the NN architecture, just a data transformer), the model (with all hyperparameters & metadata) etc. and how all things link

I literally cannot reset a single task

Where are you storing your secret JitteryCoyote63 ?

I removed it and I still get the same error 😞

That is a neat way of making it work! Thanks Martin. Once I've added the SSH key to the deployment keys in that repo, then the change in the config should work right? I'm guessing the extra index URL can be a URL to the github repo of interest? (not another privately hosted pypi repo)

We are planning on using airflow as the orchestration, but that may not fit your needs. I would say that the tool of choice is highly context specific.

We will be using airflow to trigger clearml-defined pipelines based on 'trigger' events, such as degradation in model performance, error alerts (e.g. at the data transformation task) etc.

I dont think its that. its a 20kb file upload. This was the last message just printed
ClearML Monitor: Could not detect iteration reporting, falling back to iterations as seconds-from-star

Oh, that may work. Is there any docs/demos on this?

No rush, thanks for looking into it and sorry for tagging everyone here 😂

/home/ubuntu/.clearml/venvs-builds/3.8/lib/python3.8/site-packages/clearml/utilities/plotlympl/mpltools.py:371: MatplotlibDeprecationWarning: The is_frame_like function was deprecated in Matplotlib 3.1 and will be removed in 3.3.This is the last print statement before it hangs

Can I use the task scheduler to schedule an update task every say 10 mins, would that keep it from being deleted?

2021-03-01 20:51:55,655 - clearml.Task - INFO - Completed model upload to s3://15gifts-clearml/artefacts/pre-engine-traits/logistic-regression-paths-and-sales-tfidf-device-brand.8d68e9a649824affb9a9edf7bfbe157d/models/tfidf-logistic-regression-1614631915-8d68e9a649824affb9a9edf7bfbe157d.pkl *****
2021-03-01 20:52:01
2021-03-01 20:51:57,207 - clearml.Task - INFO - Waiting to finish uploads

And how will it know that the container is on ECR instead of some other container repository?

0.17.5rc5

On my local I have clearml 0.17.4

its a seaborn heatmap that needs to be plotted. not sure if that is useful at all

To report the metric to clearML, would that just be a batch update every t interval?

Yeah that could be one approach.

I mean, is it possible to create a trigger task that reads a message from a queue? And that message contains information about whether a pipeline needs to be triggered or not

By script, you mean entering these two lines separately as a list for that extra_docker_shell_scripts arugment?

I can't figure out from the examples how the external trigger works. All of our model performance stats are in the DWH, and we want to build triggers based on that, Is that possible to integrate with Clearml triggers and schedulers?

Say we have a DAG running on airflow every 30 mins. The purpose of this DAG is to aggregate results of model performance. If model performance is poor, then it sends a message to a queue with some config on which model to re-train.

I would like to use a TaskScheduler to poll this queue every X interval, to check whether a training pipeline needs to be kickstarted or not

Out of curiosity, is there a reason why utils is not a package in its own right?

I have found that a private PyPi repo really does help with managing dependencies

Thanks Jake, I will have a look. Is there a reason a lot disk space would be used on the server instance? Is there something in the config I can change to ensure that minimal memory is used on that server, and mostly s3 is used for storage?

I can authorise CodeArtifact if I ssh into the server, and install the private package with no issues. Seems like something is forcing clearml-agent to use github cloning to install, rather than directly pip. Not sure if this is a configuration I have set up myself, or whether the server is configured to do this

Here is the error message from the console
Collecting git+ssh://****@github.com/15gifts/py-db.git Cloning ssh://****@github.com/15gifts/py-db.git to /tmp/pip-req-build-xai2xts_ Running command git clone -q 'ssh://****@github.com/15gifts/py-db.git' /tmp/pip-req-build-xai2xts_ ERROR: Repository not found. fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists.

Ideally, I want to avoid re-inventing the wheel so if this functionality already exists with some examples then it would be great if someone could point me to it

I thought nothing should be stored locally on the agent? Shouldn't all files be logged to the storage rather than the instance itself?

Is there any documentation on how to set up the config for the agent?

Never mind. I think I figured it out. Thanks for your help 🙂