SmugSnake6 yep, that’s exactly it.

Hope the team is aware and will fix it

Good to hear 🙏

Artifacts, nothing is reaching s3

also, i don’t need to change it during execution, i want it for a specific run

yeah, it gets to that error because the previous issue is saved…i’ll try to work on a new example

I am currently on vacation, I'll ask my team mates. But if not I'll get to it next week

Yes it worked 🙂
I loaded my entire clearml.conf in the “extra conf” part of the auto scaler, that worked

that does happen when you create a normal local task, that’s why i was confused

but it makes sense, because the agent in that case is local

How does this work in the context of a pipeline? One of the steps is a multi gpu training that requires accelerate.

don’t have one ATM

ok i'll check it out 🙂

The pipeline is a bit complex, but it did that with a very dumb example

Glad to hear you were able to reproduce it! Waiting for your reply 🙏

Yes, but it’s more complex because i’m using a pipeline… where i don’t explicitly call Task.init()

what i'm doing is getting

parent = Task.get_task(task.parent)

and then checking
parent.data.user

but the user is some unknown id that doesn't exist in the all_users list

but anyway, this will still not work because fastai’s tensorboard doesn’t work in multi gpu 😞

yes

I Mean that for example a machine with 4 gpus, I would want each task to get 1 gpu

The task is given by the monitor(like in the example) the problem is the user

@<1523701435869433856:profile|SmugDolphin23> @<1523701205467926528:profile|AgitatedDove14>
Any updates? 🙂

you can get updates on the issue i opened
https://github.com/fastai/fastai/issues/3543

but i think the probably better solution would be to create a custom ClearML callback for fastai with the best practices you think are needed…

Or try to fix the TensorBoardCallback, because for now we can’t use multi gpu because of it 😪

AgitatedDove14 So it looks like it started to do something, but now it’s missing parts of the configuration

Missing key and secret for S3 storage access(i’m using boto credential chain, which is off by default…)

why isn’t the config being passed to the inner step properly ?

We tried both subprocess.run and popen

tried your suggestion, still got to file server…

to make it very reproducible, i created a docker file for it, so make sure to run build_docker.sh and then run.sh

looks like it’s working 🙂 tnx

i’ll try to work on something that works on 1.7.2

when i did this with a normal task it worked wonderfully, with pipeline it didn’t

which part?