It seems like the configuration is cached in a way even when you change the CLI parameters.

@<1523704461418041344:profile|EnormousCormorant39> nice!
Yes the configuration is cached so that after you set it once you can just call clearml-session again without all the arguments
What was the actual issue ? Should we add something to the printout?

Hi SmoggyGoat53
What do you mean by "feature store" ? (These days the definition is quite broad, hence my question)

Hi DilapidatedDucks58 just making sure, the link is pyrorch nightly artifactory? Or is it a direct link to the package? Reason for asking, I was not aware they have proper artifactory... When the task runs the trains agent will update the installed packages with all the installed packages it used. Could you verify you have the correct version?

Regarding the extra files, you are correct, the docker container is reset every run, so they will get lost. What are those files for? Could you add ...

JitteryCoyote63 This seems like exactly what you are saying, elastic license issue...

GreasyPenguin14 GrittyKangaroo27 the new release contains a fix, could you verify it solves the issue in your scenario as well (there is now a smart timeout to detect the inconsistent state, that means the close/exit procedure might be delayed (10sec) instead of hanging in these specific rare scenarios)

Shouldn't this be a real value and not a template

you mean value being pulled to the pod that failed ?

Like, let's say I want "a 15GB GPU or better" and there's 4 queues, two of which fit the description... is there any way to set it so that ClearML will just queue it up on whichever one's available?

How do you know that? Also if you know that, what do you know about the queues ?
Generally speaking this type of granularity is k8s, but it has lots of caveats, specifically that you need to Know what you need in term of resources, that you can specify resources that do not exist, and that...

WickedGoat98 give me a minute, I'm not sure it is not ClearML related

First let's verify with the manual change, but yes

Let me rerun the code and check

from the notebook run !ls ~/clearml.conf

Nice!!!

some dependencies will sometimes require different pip versions.

none 🙂 maybe setuptools, but not pip version
(pip is just a utility to install packages, it will not be a dependency of one)

Hi SubstantialElk6

Generically, we would 'export' the preprocessing steps, setup an inference server, and then pipe data through the above to get results. How should we achieve this with ClearML?

We are working on integrating the OpenVino serving and Nvidia Triton serving engiones, into ClearML (they will be both available soon)

Automated retraining

In cases of data drift, retraining of models would be necessary. Generically, we pass newly labelled data to fine...

HugeArcticwolf77 I think this issue was resolved with the latest version 1.8.0, can you try to rerun the entire pipeline with the latest version?

You can only edit I on the cloned (copy of) the original experiment. Make sense?

Then it initiate a run on aws, which I want it to use the same task-id.

BoredPigeon26 Clone the Task, it basically creates a new copy (of the setup/configuration etc.)/
Then you can launch it on an aws instance (I'm assuming with clearml-agent)
wdyt?

But it write-over the execution tab in the gui

It does you are correct, it will however Not overwrite the reports (log scalars etc)

👍

Yes, I think we just found out it breaks clearml 🙂
could you test with the latest stable, just in case ?
(I'll make sure we have an RC that supports the hydra dev version)

WittyOwl57 this is what I'm getting on my console (Notice New lines! not a single one overwritten as I would expect)
` Training: 10%|█ | 1/10 [00:00<?, ?it/s]
Training: 20%|██ | 2/10 [00:00<00:00, 9.93it/s]
Training: 30%|███ | 3/10 [00:00<00:00, 9.89it/s]
Training: 40%|████ | 4/10 [00:00<00:00, 9.87it/s]
Training: 50%|█████ | 5/10 [00:00<00:00, 9.87it/s]
Training: 60%|██████ | 6/10 [00:00<00:00, 9.88it/s]
Training: 70%|███████ | 7/10 [00:00<00...

Will do! 🙂

I was able to see a configuration file under artifacts

Okay that is odd, it should have been under the configuration section, but as your screenshot shows this is not the case...
but the code specifically adds it:
None
called here:
None
wdyt?

hmmm How do I reproduce it ?

Will such an docker image need a trains configuration file?

If you need to configure things other than credentials (see above) than yes you might need to map trains.conf into the pod.
Specifically, if you need, map your trains.conf to /root/.trains inside the pod/container

Notice the args will be set on the connect call, so the check on whether they are empty should come after

RipeWhale0 are you taking them from here?
https://artifacthub.io/packages/helm/allegroai/clearml

Hi MiniatureCrocodile39
Which packages to you need to run the viewer? I suppose dicom reader is a must?

ReassuredTiger98

How can I make clearml-agent use pre-installed version from the nvidia/pytorch

If the Same version is required, the agent will not try to reinstall it (the new venv the agent is creating inside the container, inherits from the preinstalled system packages)

Comes with PyTorch Version 1.12 based on a commit

. I tried

torch >= 1.11

,

torch == 1.12

If in your installed packages you have torch==1.12
the agent should not tr...

https://github.com/allegroai/clearml-serving

In Windows setting 

system_site_packages

 to 

true

  allowed all stages in pipeline to start - but doesn't work in Lunux.

Notice that it will inherit from the system packages not the venv the agent is installed in

I've deleted tfrecords from master branch and commit the removal, and set the folder for tfrecords to be ignored in .gitignore. Trying to find, which changes are considered to be uncommited.

you can run git diff it is essentially...