FYI matplotlib imshow will create a debug image, and on complex plots the plot might get converted to image. (But shown under the plots section). All in all you might not be aware of it, but you are uploading image to your files server

So that agent on different nodes will probably require different cuda-version images.

That makes sense SarcasticSquirrel56
I would edit the helm chart (or deploy manually) based on a selector that will select the different nodes/gpus and assign the correct containers (i.e. matching CUDA versions to the diff GPUs / drivers)
BTW: you can also playaround with k8s glue, which would dynamically spin pods based on clearml Tasks.
wdyt?

Sorry @<1657918706052763648:profile|SillyRobin38> I missed this reply

Is ClearML-Serving using either System or CUCA shared memory? O

This needs to be set on the docker-compose:
and I think this line actually includes ipc: host which means there is no need to set the shm_size, but you can play around with it and let me know if you see a difference
[None](https://github.com/allegroai/clearml-serving/blob/7ba356efc97a6ae2159283d198d981b3c1ab85e6/docker/docker-compose-triton-gpu.yml#L1...

lastly try to add:

extra_pip_install_flags: ["--use-deprecated=legacy-resolver", ]

None

Nicely done DeterminedToad86 🙂
Wasn't this issue resolved by torch?

Hi SkinnyPanda43
In your local machine do not pass output_uri at all, so nothing will be uploaded.
On the agent's configuration file configure, default_output_uri to the S3 bucket
(Notice you can always override them in the UI, see the bottom of the execution Tab)
https://github.com/allegroai/clearml-agent/blob/e93384b99bdfd72a54cf2b68b3991b145b504b79/docs/clearml.conf#L312

Yeah.. that should have worked ...
What's the exact error you are getting ?

MysteriousBee56 and please this one: "when you run the  trains-agent  with --foreground , before it starts the docker it print the full command line"

ConvolutedSealion94 try scikit not scikitlearn
I think we should add a warning if a key is there and is being ignored... let me make sure of that

I just disabled all of them with

auto_connect_frameworks=False

Yep that also works

yes 😞

'relaunch_on_instance_failure'

This argument is Not part of the Pipeline any longer, are you running the latest clearml python version?

• Agent on laptop, Server on Kube - Fail

So I'm 100% sure there is something wrong with our ClearML Server deployment on Kube

Yeah that feels like a network config issue...

Is there a verbose setting in the agent that could help us diagnose,

yes running with debug turned on on.
since you managed to reproduce on your latop you can try to run the agent with --debug to test, specifically:

clearml-agent --debug daemon ....

if you are running it in venv mode (which I think ...

Hi PompousBeetle71 , what exactly is the scenario / problem we are trying to solve ?

On the machine running the docker-compose (i.e. the clearml-server)

I was just able to reproduce with "localhost"

How are you getting:

beautifulsoup4 @ file:///croot/beautifulsoup4-split_1681493039619/work

is this what you had on the Original manual execution ? (i.e. not the one executed by the agent) - you can also look under "org _pip" dropdown in the "installed packages" of the failed Task

Hi BitingKangaroo95
Are you running the agent on docker-mode or venv mode ?
basically, clearml-session will work on on clearml-agents that are running in docker mode
(I think we already have a fix for the documentation, probably will be deployed soon)

they are just neighboring modules to the function I am importing.

So I think that is you specify the repo,, on the remote machine you will end with the code of the component sitting at the root folder of the repo, from there I assume you can import the rest, the root git path should be part of your PYTHONPATH automatically.
wdyt?

You can get a mutable copy of the entire dataset (original version), with get_mutable_copy() Then change the files on the returned directory, then create a new Dataset with the parent dataset as the original verison, then sync the folder.
You can also just update the specific file (without needing to download the entire original version)

btw: I'm assuming that args is not the ArgParser object, as the ArgParser is automatically "connected" ?

Sure go to the "All Projects" and filter by Task Type, application / service

Hi CluelessElephant89

I'm thinking that different users might want to comment on results of an experiment and stuff. Im sure these things can be done externally on a github thread attached to the experiment

Interesting! Like a "comment section on top of a Task ?
Or should it be a project ?
Basically I have this intuition that Task granularity might be to small (I would want to talk about multiple experiments, not a single one?) and a project might be to generic ?
wdyt?
btw: The addr...

Notice: dataset_rgb.list_files() will list the content of the dataset, Not the local files:
e.g.: /folder/myfile.ext and not /hone/user/cache/folder/myfile.ext
So basically i think you are just not passing actual files, you should probably do:
for local_file in Path(folder_rgb).rglob('*'): ...

Thanks MagnificentSeaurchin79 ! This code snippet is exactly what I needed, let me check if I can reproduce it.

@<1523707653782507520:profile|MelancholyElk85> what are you trying to change ? maybe there is a better way?
BTW: if you do step_base_task.export_task() you can use the parts that you need in the dict and pass them to:
task_overrides argument in add_step (you might need to flatten the nested arguments with '.' , and thinking about it, maybe we should do that automatically?!)

is it planned to add a multicursor in the future?

CheerfulGorilla72 can you expand? what do you mean by multicursor ?

TBH ClearML doesn't seem to be picking the model up so I need to do it manually

This is odd, cleamrl will pick framework level serialization, but not just any pickle call

Why do I need an output_uri for the model saving? The dataset API can figure this out on its own

So that it knows where to upload it, if your are setting True this will be the default files server, you can also set iy for shared files system, S3 GCP storage etc.
If no value is passed, it will just log th...

Thanks!
I think this one will cover both case (the issue is with files on the root of the dataset)
if not (fnmatch(k, path) and fnmatch(k if '/' in k else '/{}'.format(k), '*/' + wildcard))}

Hi @<1643060801088524288:profile|HarebrainedOstrich43>
try this RC let me know if it works 🙂

pip install clearml==1.13.3rc1

Okay Let me check