yes

when i did this with a normal task it worked wonderfully, with pipeline it didn’t

that’s what i started with, doesn’t work in pipelines

I have but i believe i found the issue

AgitatedDove14 So it looks like it started to do something, but now it’s missing parts of the configuration

Missing key and secret for S3 storage access(i’m using boto credential chain, which is off by default…)

why isn’t the config being passed to the inner step properly ?

@<1523701205467926528:profile|AgitatedDove14> i probably missed it, I'll try tomorrow thanks!

The task is given by the monitor(like in the example) the problem is the user

ok i'll check it out 🙂

Hi @<1523701435869433856:profile|SmugDolphin23>
Confirming that rank0 process does not hang with the new version!

The accelerate CLI problem does still reproduce though (it's in my demo)

when u say use Task.current_task() you for logging? which i’m guessing that the fastai binding should do right?

still getting 4 tasks (1 does have data in results)

That’s how fastai distributes between multiple gpus, i’ll try to move the init

i’ll check if it’s already initialized

TimelyPenguin76

@<1523701205467926528:profile|AgitatedDove14>
Only got some time to work on it now, i created a small reproducible example.
I also tried to use your suggestion with import accelerate, it also had issues.

overall, when using debug_pipeline it works ok, but both methods don't work without it, i think it has something to do with wrapping accelerate.

Problem with launching through python module (your suggestion), the argparse breaks.
Problem with launching using a new process - rank0 proce...

reduced to a small snippet
` from fastai.vision.all import *
from fastai.distributed import *
from clearml import Task
from fastai.callback.tensorboard import TensorBoardCallback
from wwf.vision.timm import timm_learner

task = Task.init(project_name='LIOR_TEST', auto_connect_arg_parser={'rank': False})
path = untar_data(URLs.PETS)

size = 460
batch_size = 32

dblock = DataBlock(blocks=(ImageBlock, CategoryBlock),
get_items=get_image_files,
get_y=lambda x: ...

this
from fastai.callbacks.tensorboard import LearnerTensorboardWriter
doesn’t exist anymore in fastai2

It’s a lot of manual work that you need to remember to undo

any idea? 🙏

SmugSnake6 yep, that’s exactly it.

Hope the team is aware and will fix it

@<1523701070390366208:profile|CostlyOstrich36>

Sorry for the (very) late response.

We use the open source version which isn't part of the ClearML setup.

Anyway, we are using a standalone script but we have it source controlled in git... clearml picks this up and tries to clone the entire repo in the agent. i want to prevent this an just use the script.

Yes, but it’s more complex because i’m using a pipeline… where i don’t explicitly call Task.init()

but that means that there is no way to work with clearml+fastai2+multi gpu

I'm getting really weird behavior now, the task seems to report correctly with the patch... but the step doesn't say "uploading" when finished... there is a "return" artifact but it doesn't exist on S3 (our file server configuration)

AgitatedDove14 . so if i understand correctly, what i can possibly do is copy paste the https://github.com/fastai/fastai/blob/master/fastai/launch.py code and add the Task.init there?

and this is for a normal task

sounds good 🙂 I’ll soon check if this fixes our issue and update you

for now we downgraded to 1.7.2, but of course prefer not to stay that way

hi, yes we tried with the same result

to make it very reproducible, i created a docker file for it, so make sure to run build_docker.sh and then run.sh