
Reputation
Badges 1
107 × Eureka!` from clearml.automation import PipelineDecorator
from clearml import TaskTypes
@PipelineDecorator.component(task_type=TaskTypes.data_processing, cache=True)
def run_demo():
from transformers import AutoTokenizer, DataCollatorForTokenClassification, AutoModelForTokenClassification, TrainingArguments, Trainer
from datasets import load_dataset
dataset = load_dataset("conllpp")
model_checkpoint = 'bert-base-cased'
lr = 2e-5
num_train_epochs = 5
weight_decay =...
Hi, yes it's running with autoscaler so it's for sure in docker mode
Are you saying that it should've worked? I got 'docker' attribute doesn't exist error. Maybe it's the version of the clearml server?
Yes, and the old version only works without the patch.
I see the model on the artifacts tab, but it's not actually uploaded.
BTW, i would expect this to happen automtically when running “local” and “debug”
also, i don’t need to change it during execution, i want it for a specific run
they also appear to be relying on the tensorboard callback which seems not to work on distributed training
when u say use Task.current_task()
you for logging? which i’m guessing that the fastai binding should do right?
i’ll try to work on something that works on 1.7.2
It’s a lot of manual work that you need to remember to undo
It's models not datasets in our case...
But we can also just tar the folder and return that... Was just hoping to avoid doing that
reduced to a small snippet
` from fastai.vision.all import *
from fastai.distributed import *
from clearml import Task
from fastai.callback.tensorboard import TensorBoardCallback
from wwf.vision.timm import timm_learner
task = Task.init(project_name='LIOR_TEST', auto_connect_arg_parser={'rank': False})
path = untar_data(URLs.PETS)
size = 460
batch_size = 32
dblock = DataBlock(blocks=(ImageBlock, CategoryBlock),
get_items=get_image_files,
get_y=lambda x: ...
Yes, but it’s more complex because i’m using a pipeline… where i don’t explicitly call Task.init()
SmugSnake6 yep, that’s exactly it.
Hope the team is aware and will fix it
Noting one difference i do is using TensorBoardCallback
, because i believe the clearml docs use an outdated fastai 1 version…
Glad to hear you were able to reproduce it! Waiting for your reply 🙏
don’t have one ATM
what i'm doing is getting
parent = Task.get_task(task.parent)
and then checkingparent.data.user
but the user is some unknown id that doesn't exist in the all_users
list
@<1523701070390366208:profile|CostlyOstrich36>
Sorry for the (very) late response.
We use the open source version which isn't part of the ClearML setup.
Anyway, we are using a standalone script but we have it source controlled in git... clearml picks this up and tries to clone the entire repo in the agent. i want to prevent this an just use the script.
for now we downgraded to 1.7.2, but of course prefer not to stay that way
hi, yes we tried with the same result
to make it very reproducible, i created a docker file for it, so make sure to run build_docker.sh
and then run.sh