
Reputation
Badges 1
25 × Eureka!Sure, run:clearml-agent init
It is a CLI wizard to configure the initial configuration file.
Maybe we should rename it?! it actually creates a Task but will not auto connect it...
UnsightlyShark53 Awesome, the RC is still not available on pip, but we should have it in a few days.
I'll keep you posted here :)
btw: both should work fine
OddAlligator72 sure thing 🙂
This should sort it out:Task.init('examples', 'train', continue_last_task=True)
If you want to continue a specific Task:continue_last_task='task_id_here'
Getting the previous model:last_checkopoint = task.models['output'][-1]
What do you think?
In order to facilitate the multiple credentials one must use the Clearml SDK obviously.
Yes 🙂
The experiment finished completely this time again
With the RC version or the latest ?
Yea the "-e ." seems to fit this problem the best.
👍
It seems like whatever I add to
docker_bash_setup_script
is having no effect.
If this is running with the k8s glue, there console out of the docker_bash_setup_script ` is currently Not logged into the Task (this bug will be solved in the next version), But the code is being executed. You can see the full logs with kubectl, or test with a simple export test
docker_bash_setup_script
` export MY...
FYI matplotlib imshow will create a debug image, and on complex plots the plot might get converted to image. (But shown under the plots section). All in all you might not be aware of it, but you are uploading image to your files server
Which one of those? the 3d ball dots or the 3d face mesh?
There was an issue in some versions where seeborn plots were blank. Is that the case?
should I update nodejs in centos image ?
I think so, it might have been forgotten
PompousParrot44
Check out the task.execute_remotely()
You can call it right after the task init, and it will enqueue your running Task, and leave the process (if you want).
https://github.com/allegroai/trains/blob/65a4aa7aa90fc867993cf0d5e36c214e6c044270/trains/task.py#L1437
Hmm yes this is exactly what should not happen 🙂
Let me check it
Seems like settings on the clearml-server disappeared (specifically default queue tag?!)
yea the api server configuration also went away
okay that proves it
Hi @<1691620877822595072:profile|FlutteringMouse14>
Do I have to use Hydra
You can, and then the entire configuration is fully captured by ClearML (automatically) while you can still override values with the manual "key.sub=value" both in the UI and in the CLI
Otherwise you can connect nested dict with task.connect (these will be flattened with /
for sub keys).
Or you can connect configuration files ( task.connect_configuration
) and edit them as is in the UI (with override of...
FileNotFoundError: [Errno 2] No such file or directory
Could it be the file you are trying to run is not in the repository ?
Are you running inside a docker ?
Any chance you can send the full log ?
- yes they will! This is exactly the idea :)
- yes it will store it as text file (as is raw text) notice the return value is the file you should open. This is because when running via agent the return file will contain the conf file from the UI. Make sense?
Hi RipeGoose2
What exactly is being uploaded ? Are those the actual model weights or intermediate files ?
Hmm are you getting the warning on the client side , or in the clearml-server ?
I did nothing to generate a command-line. Just cloned the experiment and enqueued it. Used the server GUI.
Who/What created the initial experiment ?
I noticed that if I run the initial experiment by "python -m folder_name.script_name"
"-m module" as script entry is used to launch entry points like python modules (which is translated to "python -m script")
Why isn't the entry point just the python script?
The command line arguments are passed as arguments on the Args section of t...
Yes it does, but these files must be committed to begin with, basically think 'git diff' output is stored and then the agent applies it
Hi JitteryCoyote63 , I have to admit, we have not thought of this scenario... what's the exact use case to clone a Task and change the type?
Obviously you can always change the task type, a bit of a hack but should work:task._edit(type='testing')
in the docker-compose file. Still strange...
hmm yes it is... If you have an idea on what went wrong let me know, we would love to fix it
Yes, I mean use the helm chart to deploy the server, but manually deploy the agent glue.
wdyt?
VirtuousFish83 is the exit(1) called from the main process or a subprocess? Are you running it with an agent?