Reputation
Badges 1
125 × Eureka!@<1523701087100473344:profile|SuccessfulKoala55> hey Jake, how do i check how many envs it caches? doing ls -la .clearml/venvs-cache
gives me two folders
logger.report_media( title=name_title, series="Nan", iteration=0, local_path=fig_nan, delete_after_upload=delete_after_upload, ) clearml_task.upload_artifact( name=name_title, artifact_object=fig_nan, wait_on_upload=True, )
so it tries to find it under /usr/bin/python/
I assume?
when an agent launches a task, it builds a venv, copies the code, runs it, etc. in my case, the code writes files (such as data it downloaded, or model files, etc) and writes them in subfolders. I'm interested in recovering the entire folder structure.
this is because if I run a different task, everything from the previous task is overwritten.
furthermore, I need the folder structure for other things downstream
can you elaborate a bit on the token side? i'm not sure exactly what would be a bad practice here
Not sure why it tries to establish some http connection, or why it's /
...
i'm also not sure what this is-H "Content-Type: application/octet-stream" -H' NV-InferRequest:batch_size: 1 input { name: "dense_input" dims: [-1, 784] } output { name: "activation_2" cls { count: 1 } }'
um, this line is not doing anything for me 🤔controller_clearml_task = Task.current_task() controller_clearml_task.set_resource_monitor_iteration_timeout( seconds_from_start=10 )
hi SuccessfulKoala55 ! has the docker compose been updated with this?>
AgitatedDove14 yeah it should be..
(base) emilio@unicorn:~$ docker version Client: Docker Engine - Community Version: 19.03.13 API version: 1.40 Go version: go1.13.15 Git commit: 4484c46d9d Built: Wed Sep 16 17:02:36 2020 OS/Arch: linux/amd64 Experimental: false (base) emilio@unicorn:~$ docker-compose --version docker-compose version 1.17.1, build unknown
ah.. agent was on a different machine..
it's from the github issue you sent me but i don't know what the "application" part is or the "NV-InferRequest:...."
i'm just interested in actually running a prediction with the serving engine and all
I think it's still caching environments... I keep deleting the caches (pip, vcs, venvs-*) and running an experiment. it re-creates all these folders and even prints
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests>=2.20.0->clearml==1.6.4->prediction-service-utilities==0.1.0) (3.4)
Requirement already satisfied: charset-normalizer<4,>=2 in /root/.clearml/venvs-builds/3.8/lib/python3.8/site-packages (from requests>=2.20.0->clearml==1.6....
hi SuccessfulKoala55 with the clearml server update, does it use a newer ES docker?
Hi SuccessfulKoala55 , do you have an update on this?
i'm guessing the cleanup_period_in_days can only actually run every day or whatever if the script is enqueued to services
SuccessfulKoala55 I can't get it to work... I tried using the pip conf locally and it works, but the agent doesn't seem to be able to install the package
I think the issue is that the host is not trusted... it looks like it looks into the index
i'm not sure how to double check this is the case when it happens... usually we have all requirements specified with git repo
hiya Jake, how do I inject this with the extra_docker_shell_script
setting?