I can see that the data is reloaded each time, even if the machine was not shut down in between.

You can verify by looking into the Task's Log, it will contain all the docker arguments, one of them should be the cache folder mount

With env caching enabled, it won’t reinstall this private dependency, right?

It will, local packages (".") and git packages are alwyas reinstalled even if using venv caching, exactly for that reason 🙂

strange ...

Hi @<1523702932069945344:profile|CheerfulGorilla72>
This is a property on the Model object

model.published

Not sure why we do not have it here...
None
(I'll ask them to fix that)

I see, let me check something 🙂

Any chance you can share the Log?
(feel free to DM it so it will not end up public)

Can you try to manually install it and see what you are getting?
python3.10 -m pip install /home/boris/.clearml/pip-download-cache/cu117/torch-1.12.1+cu116-cp310-cp310-linux_x86_64.whl

If you have the check point (see output_uri for automatically uploading it) then you can always load it. Do you mean if you can change the iteration/ step counter? Or do you mean with trains-agent?

Yeah that sound about right, also you can put the helm chart file as a configuration on the Task when creating it, see https://clear.ml/docs/latest/docs/references/sdk/task#set_configuration_object

no available 😞

Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/tmp/build/80754af9/attrs_1604765588209/work'Seems like pip failed creating a folder
Could it be you are out of space ?

Yes, as long as the client is served from http://app.something.com it will look for the api server at http://api.something.com

Oh that's definitely off 🙂
Can you send a quick toy snippet to reproduce it ?

My bad "ssh://" prefix it not valid, let me try and see why it fails deducing this is a remote repo

What's the OS running the server?

Can't think of a reason it will have such an effect ...

New RC hopefully solves it @<1643060801088524288:profile|HarebrainedOstrich43> could you check if it works for you now?

pip install clearml==1.14.0rc0

LovelyHamster1 verified, this is a UI bug with old limitation enforced.
I will make sure they know about it, it should be fixed for the upcoming release 🙂

NICE!

To clarify, there might be cases where we get helm chart /k8s manifests to deploy a inference services. A black box to us.

I see, in that event, yes you could use clearml queues to do that, as long as you have the credentials the "Task" is basically just a deployment helm task.
You could also have a monitoring code there so that the same Task is pure logic, spinning the helm chart, monitoring the usage, and when it's done taking it down

So essentially, the server helm chart creates randomly generated secret pair and deploys it as a shared k8 secret that pods can access.

This is the tricky part, for the helm chart to be able to create it, it means it can login to the server it means there is a secret embedded in the helm chart that lets you access the default server. you see my point ?

Yey!

your account has 2FA enabled and you must use a personal access token instead of a password.I'm assuming you have created the personal access token and used it, not the pass

So what you are saying is the workers randomly report on one another's experiments ?

Hi ZippyAlligator65
You can configure it in the clearml.conf: see here:
https://github.com/allegroai/clearml-agent/blob/ebb955187dea384f574a52d059c02e16a49aeead/clearml_agent/backend_api/config/default/agent.conf#L202

This makes no sense to me 😞
Both are reading the exact same file, and using the same session / flow ...
Maybe there is an error with the "verify_certificate" on the agent ?

time.sleep(time_sleep)

You should not call time.sleep in async functions, it should be asyncio.sleep,
None

See if that makes a difference

PricklyRaven28 did you set the iam role support in the conf?
https://github.com/allegroai/clearml/blob/0397f2b41e41325db2a191070e01b218251bc8b2/docs/clearml.conf#L86

Just to get the full picture, are we expecting to see the newly created step (aka eager execution) on the original pipeline (i.e. as part od the DAG visualization) ?

Where do you store those ?