We use task.export_task() and a hacked version to get console log:
def save_console_log(task: clearml.Task, fs, remote_path, number_of_reports=10000):
from clearml.backend_api.services import events
from clearml.backend_api import Session
# Stollen from Task.get_reported_console_output()
if Session.check_min_api_version('2.9'):
request = events.GetTaskLogRequest(
task=task.id,
order='asc',
navigate_earlier=True,
...my code looks like this :
parser = argparse.ArgumentParser()
parser.add_argument('-c', '--config-file', type=str, default='train_config.yaml',
help='train config file')
parser.add_argument('-t', '--train-times', type=int, default=1,
help='train the same model several times')
parser.add_argument('--dataset_dir', help='path to folder containing the preped dataset.', required=True)
parser.add_argument('--backup', action='s...When i set output uri in the client, artefact are sent to blob storage
When file_server is set to azure:// then model/checkpoint are sent to blob storage
But the are still plot and metrics folder that are stored in the server local disk. Is it correct?
--gpus 0,1 : I believe this basically say that your code launched by the agent has access to both GPUs and that is it. Now it is up to your code to choose which GPU to use and what not and how ...
something like this: None ?
do you have a video showing the use case for clearml-session ? I struggle a bit about how is it used for ?
nevermind, all the database files are in data folder
- Modify the "uncommited changes"
- Enqueue the taskIs it possible to edit "uncommited changes" from a task via Python ??
you are forcing ssh with force_git_ssh_protocol: true
Have you setup ssh keys ?
If you are using ssh keys, why enable_git_ask_pass: true ?
in my case using self-hosted and agent inside a docker container:
47:45 : taks foo pulled
[ git clone, pip install, check that all requirements satisfied, and nothing is downloaded]
48:16 : start training
one specify the venv python, the other tell it to not do anything
(I don't really use HPO feature so I out sorry)
i need to do a git clone
You need to do it to test if it works. Clearml-agent will run it itself when it take in a task
@<1523701087100473344:profile|SuccessfulKoala55> Thanks. Manage to get it working now with
export REQUESTS_CA_BUNDLE=/etc/ssl/certs/zscaler.crt
(Ubuntu system)
like for dataset_dir I would expect a single path, not an array of 2 paths duplicated
may be I will play around a bit and ask more specific questions .... It's just I cannot find much docs around how the pipeline caching work (which is the main point of pipeline ?)
what you mean by different script ?
very hard to diagnose with this tiny bit of log ...
some clearml cache folder
there is a whole discussion about it here: None
I found that if pip is upgraded to latest version 25.0.1 then the package install fine.
The question become: why does the agent downgrade pip ?
Ignoring pip: markers 'python_version < "3.10"' don't match your environment
Collecting pip<22.3
Downloading pip-22.2.2-py3-none-any.whl.metadata (4.2 kB)
Downloading pip-22.2.2-py3-none-any.whl (2.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 3.9 MB/s eta 0:00:00
Installing collected packages: pip
Attempting uninstall: pip
...have you try a different browser ?
I didn;t know that from the client side, you can specify the storage elsewhere than the clearML server. Good to know !
But I still want to know, if possible, to use a blob storage by default, configured on the ClearML server, and each client don't need to do that ...
To me the whole point of having pipeline is to have a system that "know" previous state and make "smart" decision on what should run and what not. If it's just about if then else, then code already handle all that.
And what I struggle a bit is to find doc on how it determine the existing state and how it make decision what to run. thus the initial question
For #2: it's a pull rather than a push system: you need to have a script that do pulling at regular interval and need to keep track what new and what not?
and just came across this: None
That sounds like what you may be looking for