but that means that there is no way to work with clearml+fastai2+multi gpu

this
from fastai.callbacks.tensorboard import LearnerTensorboardWriter
doesn’t exist anymore in fastai2

maybe the fastai bindings in clearml package are outdated

any idea? 🙏

That’s how fastai distributes between multiple gpus, i’ll try to move the init

unrelated, i’m getting a crash, but seems related to fastai and not clearml

when u say use Task.current_task() you for logging? which i’m guessing that the fastai binding should do right?

reduced to a small snippet
` from fastai.vision.all import *
from fastai.distributed import *
from clearml import Task
from fastai.callback.tensorboard import TensorBoardCallback
from wwf.vision.timm import timm_learner

task = Task.init(project_name='LIOR_TEST', auto_connect_arg_parser={'rank': False})
path = untar_data(URLs.PETS)

size = 460
batch_size = 32

dblock = DataBlock(blocks=(ImageBlock, CategoryBlock),
get_items=get_image_files,
get_y=lambda x: ...

AgitatedDove14 . so if i understand correctly, what i can possibly do is copy paste the https://github.com/fastai/fastai/blob/master/fastai/launch.py code and add the Task.init there?

i get for one of the tasks, but then it fails because it seems that the fastai2 tensorboardcallback isn’t fit for distributed training (which i’m opening an issue for them now)

https://github.com/allegroai/clearml/blob/master/examples/frameworks/fastai/fastai_with_tensorboard.py

TimelyPenguin76

i’m following this guide
https://docs.fast.ai/distributed.html#Learner.distrib_ctx

so you run it like this
python -m fastai.launch <script>

Noting one difference i do is using TensorBoardCallback , because i believe the clearml docs use an outdated fastai 1 version…

still getting 4 tasks (1 does have data in results)

you can get updates on the issue i opened
https://github.com/fastai/fastai/issues/3543

but i think the probably better solution would be to create a custom ClearML callback for fastai with the best practices you think are needed…

Or try to fix the TensorBoardCallback, because for now we can’t use multi gpu because of it 😪

but anyway, this will still not work because fastai’s tensorboard doesn’t work in multi gpu 😞

Hi, yes it's running with autoscaler so it's for sure in docker mode

Are you saying that it should've worked? I got 'docker' attribute doesn't exist error. Maybe it's the version of the clearml server?

and use that instead of the -m fastai.launch part ofcourse

i’ll check if it’s already initialized