I think it's still caching environments... I keep deleting the caches (pip, vcs, venvs-*) and running an experiment. it re-creates all these folders and even prints
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests>=2.20.0->clearml==1.6.4->prediction-service-utilities==0.1.0) (3.4)
Requirement already satisfied: charset-normalizer<4,>=2 in /root/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from requests>=2.20.0->clearml==1.6....
hiya Jake, how do I inject this with the extra_docker_shell_script
setting?
can you elaborate a bit on the token side? i'm not sure exactly what would be a bad practice here
does this make more sense? SuccessfulKoala55
Hi SuccessfulKoala55 I am having some issues with this. I have put a concurrency limit of 2 and I can see 3 workers running
Is this a possible future feature? I have used cometML before and they have this. I'm not sure how they do it though...
using clearML agent
um, this line is not doing anything for me 🤔controller_clearml_task = Task.current_task() controller_clearml_task.set_resource_monitor_iteration_timeout( seconds_from_start=10 )
in particular, I ran the agent with clearml-agent daemon --queue test-concurrency --create-queue --services-mode 2 --docker "ubuntu:20.04" --detached
and enqueued 4 tasks to it that sleep 15 minutes.
I can see all 4 tasks running, see
This is a minimal comet example. I'm afraid I don't know what it does under the hood.. There are no callbacks on the metrics tracked in model.fit
and yet if you check out your project in the website, your training and validation losses are tracked automatically, live.
Ok, going to ask the server admins, will keep you posted, thanks!
well.. it initially worked but now i get the same thing 😕 SuccessfulKoala55
` Using cached repository in "/root/.clearml/vcs-cache/DeployKit_cloud.git.3e6952dd2fa4054e353465fe2d40daa3/DeployKit_cloud.git"
fatal: Could not read from remote repository. `
I can do curl
http://localhost:8080 but it's a remote server so unless I do X forwarding I can't browse it