Reputation
Badges 1
19 × Eureka!Hi WackyRabbit7 and ReassuredTiger98
While what Tim said was true, it only applies to the Python binary used to create the venv (i.e. the venv will still be created).
WackyRabbit7 , to specify an pre-existing venv you can use the CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=<path-to-venv-binary>
env var
VexedCat68 this looks like the SDK code simply can't get to the server, which implies, as CostlyOstrich36 suggested, that your clearml.conf
file is not properly configured, or that there's a FW issue when connecting to the server
Hi @<1523701066867150848:profile|JitteryCoyote63> this can be set by the workers.default_timeout
setting in the apiserver.conf file, the default it 600 (seconds)
Take a look here: https://allegro.ai/docs/deploying_trains/trains_agent_install_configure/
Search for system_site_packages: false
It might be the task was stopped/reset from the UI?
I think I understand now that I first need to have clearml server up and running.
Yeah, ClearML CLI tools and SDK always rely on a ClearML Server
I know, I just meant most people don't install the server on the same machine they use for running experiments in docker mode 🙂
In any case, if you make sure the docker containers are using the same docker network, you could refer to the apiserver as http://apiserver:8008
Did you try using the latest agent RC? It's 1.2.0rc3
Is that the entire code? where are the imports etc?
Well, it's certainly in our workplan 🙂
IS it /home
? I think it's only a subdir of home
You can actually use the agent's --create-queue
command line option to make it automatically create a queue for you
And what are the installed packages showing in the task UI panel?
I'll check it out and try to get one 🙂
What you're seeing looks like a payload being stripped from the request body (typically sent using a GET request), which is typical of GCP load-balancers
But since DeleteObject is required for overwrite, and since we use overwrite as well in artifacts and model uploads, I think there's no way around it...
Yeah, that's a possible solution 🙂
what happens if you uninstall it and than try to run the code?
VexedCat68 this likely happens since you already have a token for another ClearML server (perhaps another server you've set up?). The correct fix it to either make sure you clear your browser's cookies, or (if you have more than one server set up), make sure each server uses a different cookie name.
Hi @<1571308079511769088:profile|GentleParrot65> , since this is a server-side setting, no, since that would affect all users
You need to set that in the environment section of the apiserver service in the docker-compose.yaml file. And yes, you'll need to run docker-compose up again
You can probably do that using the docker restart
command on the specific container (just do docker ps
for the list of active containers) - I think docker-compose will allow you to do that
I'm not sure it will work - you can try, though
Hi @<1523701323046850560:profile|OutrageousSheep60> , the only thing you need to do in order to speed things up is start with a docker image that has most of the packages you need preinstalled - once you have that, when using this docker image, the agent can create a venv that inherits the system packages installed, thus avoiding re-download and installation of packages. Creating the venv itself is very fast.