Which means there will be atleast multiple published models entries of same model over time?

Only the specific one will be published (not all the Models the Task created)

BoredHedgehog47 that actually depends on the container, are you running as root inside the container ?
if not I think the easiest hack is to always map /etc/hosts as a k8s secret file?

AttractiveCockroach17 can I assume you are working with the hydra local launcher ?

Hi @<1689446563463565312:profile|SmallTurkey79>
This call is to set an existing (already created Task's requirements). Since it was just created it waits for the automatic package detection before overriding it.
What you want is " Task.force_requirements_env_freeze " (notice Class level, that need to be called Before Task.init)

Task.force_requirements_env_freeze(requirements_file="requirements.txt")
task = Task.init(...)

Yes my bad 😞
Let's try again:
` docker run -it --gpus "device=1" -e CLEARML_WORKER_ID=Gandalf:gpu1 -e CLEARML_DOCKER_IMAGE=nvidia/cuda:11.4.0-devel-ubuntu18.04 -v /home/dwhitena/.git-credentials:/root/.git-credentials -v /home/dwhitena/.gitconfig:/root/.gitconfig -v /tmp/.clearml_agent.7rjdh80a.cfg:/root/clearml.conf -v /tmp/clearml_agent.ssh.ppsd9sze:/root/.ssh -v /home/dwhitena/.clearml/apt-cache.1:/var/cache/apt/archives -v /home/dwhitena/.clearml/pip-cache:/root/.cache/pip ...

WackyRabbit7 I'll make sure it is fixed

Sorry I need the full log ... feel free to DM it to me

👍

DefiantCrab67
It will always install it on any environment

(RC will be out in a few days)

DistressedGoat23 check this example:
https://github.com/allegroai/clearml/blob/master/examples/optimization/hyper-parameter-optimization/hyper_parameter_optimizer.py
aSearchStrategy = RandomSearchIt will collect everything on the main Task

This is a curial point for using clearml HPO since comparing dozens of experiments in the UI and searching for the best is just not manageable.

You can of course do that (notice you can actually order them by scalars they report, and even do ...

Go to the workers & queues, page right side panel 3rd icon from the top

just to check. Does the k8s glue install torch by default?

SubstantialElk6 what do you mean the glue installs torch ?
The glue will take a Task from the queue create a k8s job (basically use the same docker and inside the docker run get the agent to execute the requested Task). Where would the "torch" come into play?

Ohh sorry. task_log_buffer_capacity is actually internal buffer for the console output, on how many lines it will store before flushing it to the server.
To be honest, I can't think of a reason to expose / modify it...

there is a semaphore warning, not sure if it’s related

Can you resend it?
Is the Task marked as closed when the process ends ?

Doesnt solve the issue if a HPO run is going to take a few days

The HPO Task has a table of the top performing experiments, so when you go to the "Plot" tab you get a summary of all the runs, with the Task ID of the top performing one.
No need to run through the details of the entire experiments, just look at the summary on the HPO Task.

Hi ClumsyElephant70
So do you need both requirements.txt combined ?
How will the agent be able to reproduce both repo on the remote machine ?

I can probably have a python script that checks if there are any tasks running/pending, and if not, run docker-compose down to stop the clearml-server, then use boto3 to trigger the creating of a snapshot of the EBS, then wait until it is finished, then restarts the clearml-server, wdyt?

I'm pretty sure there is a nice way, let me check soemthing

LovelyHamster1 what do you mean by "assume the permissions of a specific IAM Role" ?
In order to spin an ec2 instance (aws autoscaler) you have to have correct credentials, to pass those credentials you must create a key/secret pair to pass to the autoscaler. There is no direct support for IAM Role. Make sense ?

Hi LudicrousParrot69
Not sure I follow, is this pyfunc running remotely ?
Or are you looking for interfacing with previously executed Tasks ?

https://www.youtube.com/watch?v=1C_l5ICJlEo&list=UUH-oDmZ7rvtfV_tZMDrfwFg&index=5

Hi SmugOx94
Hmm are you creating the environment manually, or is it done by Task.init ?
(Basically Task.init will store the entire environment of conda, and if the agent is working with conda package manager it will use it to restore it)
https://github.com/allegroai/clearml-agent/blob/77d6ff6630e97ec9a322e6d265cd874d0ab00c87/docs/clearml.conf#L50

BTW:

If I try to find the right model in the

task.models["output"]

(this time there is just one but in my code there may be several) it appears with the

(see other attached screenshot).

What would make sense here ? (I have to be honest I'm not sure).
To be specific there is "model name" which is not unique , and there is model-key which is unique to the Task (i.e. task.models["output"]["model-key"] )

What do you have under the "installed packages" section? Also you can configure the agent to use poetry to restore the environment (instead of pip)

you can also just create a venv and run the tests there (with the latest python package) ?

(Not sure it actually has that information)

Hi SkinnyPanda43
Do you mean the cleaml-agent or the cleaml python (a.k.a the auto package detection) ?

JitteryCoyote63 any specific reason for the question?

Hi GrotesqueOctopus42

creates a graph of the neural network and would be nice to have it on the experiment logs aswell

I think the main issue is displaying later in the UI, thoughts?
BTW: is this useful for you outside f very local TF debugging ?

SmarmyDolphin68 What's the matplotlib version ? and python version?