How does a task specify which docker image it needs?

Either in the code itself 'task.set_base_docker' or with the CLI, or set it in the UI when you clone an experiment (everything becomes editable)

Hi CloudySwallow27

how can I just "define" it on my local PC, but not actually run it.

You can use the clearml-task CLI
https://clear.ml/docs/latest/docs/apps/clearml_task#how-does-clearml-task-work
Or you can add the following line in your code, that will cause the execution to stop, and to continue on a remote machine (basically creating the Task and pushing it into an execution queue, or just aborting it)
task = Task.init(...) task.execute_remotely()https://clear.ml/do...

WARNING:root:Could not lock cache folder /home/ronslos/.clearml/venvs-cache: [Errno 11] Resource temporarily unavailable

Hi @<1549927125220331520:profile|ZealousHare78>
could it be you are also working on the same machine ? are you running the agent in docker mode or venv mode ?

This looks exactly like the timeout you are getting.
I'm just not sure what's the diff between the Model autoupload and the manual upload.

Hi WittyOwl57
I think what happens is it auto-logs the joblib load/save calls, these calls track models used/created by the code, and attach them to the model repository representing these model.
I'm assuming there are multiple load/save , and there are multiple model instances pointing to the same local file "file:///tmp/..." . The earning basically says it is re-registering existing models.
Make sense ?

I double checked the code it's always being passed 😞

BroadMole98 as one can expect long answer as well 🙂

I have a workflow with 19000 job nodes in it.

wow, 19k job nodes? as in a single pipeline 19k steps?

The main idea of the trains-agent is to allow multi-node workloads, and creating pipelines on top of a scheduler without worrying about docker packaging (done automatically for you), and to have a proper scheduler with priority (that is missing from k8s)

If the first step is just "logging" all the steps, you can easily add "Task...

shared "warm" folder without having to download the dataset locally.

This is already supported 🙂
Configure the sdk.storage.cache.default_base_dir in your clearml.conf to point to a shared (mounted) folder
https://github.com/allegroai/clearml-agent/blob/21c4857795e6392a848b296ceb5480aca5f98e4b/docs/clearml.conf#L205
That's it 🙂

We should probably add (set_task_type :))

https://github.com/allegroai/trains/blob/master/docs/trains.conf#L47

Sadly, I think we need to add another option like task_init_kwargs to the component decorator.
what do you think would make sense ?

Sounds good?

SoreDragonfly16 could you test with Task.init using reuse_last_task_id=False for example:
task = Task.init('project', 'experiment', reuse_last_task_id=False)The only thing that I can think of is running two experiments with the same project/name on the same machine, this will ensure every time you run the code, you create a new experiment.

MelancholyBeetle72 there is an RC with a fix, check the GitHub issue for details :)

JitteryCoyote63 I think that without specifically adding torch to the requirements, the agent will not be able to automatically resolve the correct cuda/torch version. Basically you should add torch to the requirements.txt file, and provide it to Task create, or use Task.force_requirements_env_freeze

@<1523707653782507520:profile|MelancholyElk85> I just run a single step pipeline and it seemed to use the "base_task_id" without cloning it...
Any insight on how to reproduce ?

Ohh then we can definitely support it, could you maybe post a toy example for testing? Or even better PR it to the examples/tensorboardX folder?

It does not upload, the default behavior is to log the artifact (so you know where you stored, but not enforce unnecessary uploads)
If you were to change:
task = Task.init(project_name='examples', task_name='Keras with TensorBoard example')to:
task = Task.init(project_name='examples', task_name='Keras with TensorBoard example', output_uri=" ")It would also upload the model

UI for some anomalous file,

Notice the metrics are not files/artifacts, just scalars/plots/console

PanickyMoth78 'tensorboard_logger' is an old deprecated package that meant to create TB events without TB, it was created before TB was a separate package. Long story short, it is not supported. That said if you just run the same code and replace tensorboard_logger with tensorboard, you should see all scalars in the UI

background:
ClearML logs TB events as they are created in real-time, TB_logger is not TB, it creates events and dumps them directly into a TB equivalent event file

How does it work with k8s?

You need to install the clearml-glue and them on the Task request the container, notice you need to preconfigure the clue with the correct Job YAML

Hi @<1801424298548662272:profile|ConvolutedOctopus27>

I am getting errors related to invalid git credentials. How do I make sure that it's using credentials from local machine?

configure the git_user/git_pass (app key) inside your clearml.conf on the machine with the agent:
None

Okay, I was able to reproduce it (this is odd) let me check ...

These paths are

pathlib.Path

. Would that be a problem?

No need to worry, it should work (i'm assuming "/src/clearml_evaluation/" actually exists on the remote machine, otherwise useless 🙂

that might be it.
Is the web UI working properly ?
What ports are you using?

Thanks StaleButterfly40 !

after generating a fresh set of keys

when you have a new set, copy paste them idirectly into the 'cleaml.conf' (should be at the top, can't miss it)

FreshParrot56 we could add this capability, but the main caveat is that f your version depends on multiple parent versions you still need to download and extract all the parent versions, which means that when you clear them you might hurt later performance. Does that make sense? What is the use-case / scenario for you?

Month +- 🙂

Hi AdventurousRabbit79
Try:
"extra_clearml_conf" : "aws { s3 {key: A, secret : B, region: C, }} ",Generally speaking no need for the quotes on the secret/key
You also need the comma to separate between keys.
You can test if it is working by adding the same string to your local clearml.conf and importing the cleaml package