Reputation
Badges 1
25 × Eureka!strange ...
Hi @<1523702932069945344:profile|CheerfulGorilla72>
This is a property on the Model object
model.published
Not sure why we do not have it here...
None
(I'll ask them to fix that)
I see, let me check something 🙂
Any chance you can share the Log?
(feel free to DM it so it will not end up public)
Can you try to manually install it and see what you are getting?python3.10 -m pip install /home/boris/.clearml/pip-download-cache/cu117/torch-1.12.1+cu116-cp310-cp310-linux_x86_64.whl
If you have the check point (see output_uri for automatically uploading it) then you can always load it. Do you mean if you can change the iteration/ step counter? Or do you mean with trains-agent?
Yeah that sound about right, also you can put the helm chart file as a configuration on the Task when creating it, see https://clear.ml/docs/latest/docs/references/sdk/task#set_configuration_object
Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/tmp/build/80754af9/attrs_1604765588209/work'Seems like pip failed creating a folder
Could it be you are out of space ?
Yes, as long as the client is served from http://app.something.com it will look for the api server at http://api.something.com
Oh that's definitely off 🙂
Can you send a quick toy snippet to reproduce it ?
My bad "ssh://" prefix it not valid, let me try and see why it fails deducing this is a remote repo
What's the OS running the server?
New RC hopefully solves it @<1643060801088524288:profile|HarebrainedOstrich43> could you check if it works for you now?
pip install clearml==1.14.0rc0
LovelyHamster1 verified, this is a UI bug with old limitation enforced.
I will make sure they know about it, it should be fixed for the upcoming release 🙂
To clarify, there might be cases where we get helm chart /k8s manifests to deploy a inference services. A black box to us.
I see, in that event, yes you could use clearml queues to do that, as long as you have the credentials the "Task" is basically just a deployment helm task.
You could also have a monitoring code there so that the same Task is pure logic, spinning the helm chart, monitoring the usage, and when it's done taking it down
So essentially, the server helm chart creates randomly generated secret pair and deploys it as a shared k8 secret that pods can access.
This is the tricky part, for the helm chart to be able to create it, it means it can login to the server it means there is a secret embedded in the helm chart that lets you access the default server. you see my point ?
your account has 2FA enabled and you must use a personal access token instead of a password.I'm assuming you have created the personal access token and used it, not the pass
So what you are saying is the workers randomly report on one another's experiments ?
Hi ZippyAlligator65
You can configure it in the clearml.conf: see here:
https://github.com/allegroai/clearml-agent/blob/ebb955187dea384f574a52d059c02e16a49aeead/clearml_agent/backend_api/config/default/agent.conf#L202
This makes no sense to me 😞
Both are reading the exact same file, and using the same session / flow ...
Maybe there is an error with the "verify_certificate" on the agent ?
time.sleep(time_sleep)
You should not call time.sleep in async functions, it should be asyncio.sleep,
None
See if that makes a difference
PricklyRaven28 did you set the iam role support in the conf?
https://github.com/allegroai/clearml/blob/0397f2b41e41325db2a191070e01b218251bc8b2/docs/clearml.conf#L86
Just to get the full picture, are we expecting to see the newly created step (aka eager execution) on the original pipeline (i.e. as part od the DAG visualization) ?
Where do you store those ?
BTW: what's the use case? Why do you need to open two Tasks in the same code/script ?
MiniatureCrocodile39 from the screen shot I imagine you are running inside a docker, this means that when you restart the docker, the configuration file is lost.
Could that be the case ?