Initially thought use_current_task in Dataset.create does it, but that seems to set the DataprocessingTask itself to the current task

Only one. Will replicate it in detail and see what’s actually up

Something like the Gitlab Kubernetes Executor could work well.

This Docker Mode - https://allegro.ai/clearml/docs/docs/use_cases/clearml_agent_use_case_examples.html#docker-mode

Got it thanks TimelyPenguin76 .

But you have to do config.pbtxt stuff right?

AgitatedDove14 - this was an interesting one. I think I have found the issue, but verifying the fix as of now.

One of the devs was using shutil.copy2 to copy parts of dataset to a temporary directory in a with block - something like:

with TemporaryDirectory(dir=temp_dir) as certificates_directory: for file in test_paths: shutil.copy2(f"{dataset_local}/{file}", f"{certificates_directory}/file")
My suspicion is since copy2 copies with full data and symlin...

` if project_name is None and Task.current_task() is not None:
project_name = Task.current_task().get_project_name()

if project_name is None and not Task.running_locally():
task = Task.init()
project_name = task.get_project_name() `

Would be good to have frequentish releases if possible 🙂

AgitatedDove14 - worked with mutable copy! So was definitely related to the symlinks in some form

Yeah concerns make sense.

The underlying root issue is unnecessary models being added or at least what I think are unnecessary and even happening when you load a model to test.

Do people use ClearML with huggingface transformers? The code is std transformers code.

Will create an issue.

From the code - it’s supposed to not cache if task override is different? I also have task_override that adds a version which changes each run

Yeah sure

It did pick it from the task?

Beyond this have the UI running, have to start playing with it. Any suggestions for agents with k8s?

another place I was using was to see if i am in a pipeline task

How can I ensure that additional tasks aren’t created for a notebook unless I really want to?

workflow

don’t know what’s happening there

Basic question - i am running clearml agent in a ubuntu ec2 machine. Does it use docker by default? I thought it uses docker only if I add the --docker flag?

Is there some doc or relevant code on exactly what's happening? The behaviour has been random

Ref of dvc doing about the same - https://github.com/iterative/dvc/blob/master/dvc/fs/s3.py#L127-L134

Yeah got it

Only allowed to have ssh key, not username-password

AlertBlackbird30 - got it running. Few comments:

Nodeport is set by default despite being parameter in values.yml. For example:` webserver:
extraEnvs: []

service:
type: NodePort
port: 80 `2. Ingress was using 8080 for webserver but service was 80
3. Had to change path in ingress to “/*” instead of “/” to get it working for me

Yeah getting a better sense

As of now my understanding is agent and worker are just interchangeable words

Yeah sounds good to me

Thanks for the fast responses as usual AgitatedDove14 🙂