what about having 2 agents, one on each GPU, on the same machine, serving the same queue ? So that when you enqueue, which ever agent (thus GPU) available will take the new task
all good. Just wanted to know in case I missed it
I understand to from the agent, point of view, I just need to update the conf file to use new credential and new server address.
so what was the solution/hack then ?
got it
Thanks @<1523701070390366208:profile|CostlyOstrich36>
I saw that page ... but nothing about number of worker of a queue .... or did I miss it ?
or which worker is in a queue ...
we are usign mmsegmentation by the way
I understand for cleaml-agent
What I mean is that I have 2 self deployed server. I want to switch between the 2 config when running the code locally, not inside the agent
I am not familiar with autoscaler ... are you using the paid version of Clearml ?
@<1523701087100473344:profile|SuccessfulKoala55> Yes, I am aware of that one. It build docker container ... I wanted to build without docker. Like when clearml-agent run in non-docker mode, it is already building the running env inside it caching folder structure. I was wondering if there was a way to stop that process just before it execute the task .py
So I tried:
import livsdk.livbatch
import clearml
clearml.Task.add_requirements("livsdk","
")
task = clearml.Task.init(project_name="hieu-test", task_name='base_config')
print("Done")
Which give me this list of Packages Installed:
# Python 3.10.10 (main, Mar 05 2023, 19:07:49) [GCC]
# Local modules found - skipping:
# livsdk == ../[REDACTED]/livsdk/__init__.py
Augmentor == 0.2.10
Pillow == 9.2.0
PyYAML == 6.0
albumentations == 1.2.1
azure_storage_blob == 12.1...nice !! That is exactly what I am looking for !!
Oh, I was assuming you are passing the entire DB backups to the cloud.
Yes, that is what I want to do.
So I need to migrate both the MongoDB database and elastic search database from my local docker instance to the equivalent in the cloud ?
can you make train1.py use clearml.conf.server1 and train2.py use clearml.conf2 ?? In which case I would be intersted @<1523701087100473344:profile|SuccessfulKoala55>
Task.export_task() will contains what you are looking for.
In this case ['script']['diff']
I don;t think ClearML is designed to handle secrets other than git and storage ...
so it's not suppose to say "illegal output destination ..." ?
is this mongodb type of filtering?
Nice ! That is handy !!
thanks !
nevermind, all the database files are in data folder
from the logs, it feels like after git clone, it spend minutes without outputting anything. @<1523701205467926528:profile|AgitatedDove14> Do you know what is the agent suppose to do after git clone ?
I guess a check that all packages is installed ? But then with CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL=1, what is the agent doing ??
if you want plot, you can simply generate plot with matplotlib and clearml can upload them in the Plot or Debug Sample section
yup, you have the flexibility and option, that what so nice with ClearML
and just came across this: None
That sounds like what you may be looking for
I also use this: None
Which can give more control
