Hi, yes it's running with autoscaler so it's for sure in docker mode

Are you saying that it should've worked? I got 'docker' attribute doesn't exist error. Maybe it's the version of the clearml server?

maybe the fastai bindings in clearml package are outdated

We also wanted this, we preferred to create a docker image with all we need, and let the pipeline steps use that docker image

That way you don’t rely on clearml capturing the local env, and you can control what exists in the env

@<1523701118159294464:profile|ExasperatedCrab78>
Hey again 🙂
I believe that the transformers patch wasn’t released yet right? we are getting into a problem where we need new features from transformers but can’t use because of this

this
from fastai.callbacks.tensorboard import LearnerTensorboardWriter
doesn’t exist anymore in fastai2

@<1523701118159294464:profile|ExasperatedCrab78>
Hey 🙂
Any updates on this? We need to use a new version of transformers because of another bug they have in an old version. so we can’t use the old transformers version anymore.

i’ll check if it’s already initialized

@<1523701205467926528:profile|AgitatedDove14>
Only got some time to work on it now, i created a small reproducible example.
I also tried to use your suggestion with import accelerate, it also had issues.

overall, when using debug_pipeline it works ok, but both methods don't work without it, i think it has something to do with wrapping accelerate.

Problem with launching through python module (your suggestion), the argparse breaks.
Problem with launching using a new process - rank0 proce...

Glad to hear you were able to reproduce it! Waiting for your reply 🙏

which part?

Yes tnx for clarifying 😁

@<1523701435869433856:profile|SmugDolphin23> @<1523701205467926528:profile|AgitatedDove14>
Any updates? 🙂

Saw it was merged 🙂 One down, one to go

Hey 🙂 Thanks for the update!

what i’m missing the is the point where you report to clearml between cast and casting back 🤔

@<1523701118159294464:profile|ExasperatedCrab78> Sorry only saw this now,
Thanks for checking it!
Glad to see you found the issue, hope you find a way to fix the second one. for now we will continue using the previous version.
Would be glad if you can post when everything is fixed so we can advance our version.

that makes more sense 🙂
would this work now as a workaround until the version is released?

right, it works on 1.7.1

We tried both subprocess.run and popen

not sure about this, we really like being in control of reproducibility and not depend on the invoking machine… maybe that’s not what you intend

I tried to work on a reproducible script but then i get errors that my clearml task is already initialized (also doesn’t happen on 1.7.2)

SmugDolphin23 BTW, this is using clearml and huggingface’s automatic logging… didn’t log something manual

@<1523701435869433856:profile|SmugDolphin23>
Hey 🙂
Any update?

We are having more issues with transformers and clearml in their new version.
The step that has transformers 4.25.1 isn’t able to upload artifacts.
If we downgrade transformers==4.21.3 it works

will check

sounds good 🙂 I’ll soon check if this fixes our issue and update you

tnx, i just can’t use 1.7.1 because of the pipeline problem from before

This is the next step not being able to find the output of the last step

ValueError: Could not retrieve a local copy of artifact return_object, failed downloading 

Yes it worked 🙂
I loaded my entire clearml.conf in the “extra conf” part of the auto scaler, that worked

that’s what i started with, doesn’t work in pipelines

CostlyOstrich36 This is for a step in the pipeline

tried your suggestion, still got to file server…