Ah it is.

If that's the case, wouldn't it apply across the board? This happens in a single task within ray - the other tasks (I have many in a single run) are fine

Or do you mean the contents of the configuration, probably :face_palm: ... one moment

What do you mean 😄 Using logging.config.dictConfig(...)

I believe it is maybe a race condition that's tangent to clearml now...

Removing the PVC is just setting the state to absent AFAIK

Thanks!

Debugging. It's very useful for us to be able to see the contents of the configuration and understand what is going on and what is meant to be going on. Without a preview (which in our case is the entire content of the configuration file), one has to take an annoying route of downloading the files etc. The configurations are uploaded to a single task and then linked across all task to conserve storage space (so the S3 storage point is identical across tasks) Sure, sounds good. I think it's a ...

I can elaborate in more detail if you have the time, but generally the code is just defined in some source files.
I’ve been trying to play around with pipelines for this purpose, but as suspected, it fails finding the definition for the pickled object…

The network is configured correctly 🙂 But the newly spun up instances need to be set to the same VPC/Subnet somehow

@<1537605940121964544:profile|EnthusiasticShrimp49> It’ll take me still some time to find the MVC that generated this, but I do have the ClearML experiment page for it. I was running the thing from ipython , and was trying to create a task from a function:

They are set with a .env file - it's a common practice. The .env file is, at the moment, uploaded to a temporary cache (if you remember the discussion regarding the StorageManager ), so it's also available remotely (related to issue #395)

Thanks LazyFish41 , that explains it 👍

But there's nothing of that sort happening. The process where it's failing is on getting tasks for a project.

minio was a tiny bit of headache to configure, but I'd be happy to help if you want CrookedWalrus33 , I just went through this process yesterday and today (see a few threads up...)

SuccessfulKoala55 The changelog wrongly cites https://github.com/allegroai/clearml/issues/400 btw. It is not implemented and is not related to being able to save CSVs 😅

And this is of course strictly with the update to 1.6.3 (or newer) that should support API 2.20

Hey @<1523701205467926528:profile|AgitatedDove14> , thanks for the reply!

We would like to avoid dockerizing all our repositories. And for the time being we have not used the decorators, but we can do that too.
The pipeline is instead built dynamically at the moment.

The issue is that the components do not have their dependency. For example:

def step_one(...):
    from internal.repo import private
    # do stuff

When step_one is added as a component to the pipeline, it does ...

I'm aware, but it would be much cleaner to define them in the worker's clearml.conf and let ClearML expose them locally to running tasks.

EDIT: Also the above is specifically about serving, which is not the target here 🤔 At least not yet 😄

Then that did not work, but I'll look into it again soon!

It was really easy with the attached code, really 👍

I would only maybe suggest adding in the documentation, that if one uses the default recommended install location, then the script can be run without any command line arguments.
I had to momentarily look at the code to see the default paths match my own (though I could've also looked at --help default values 😛 )

Yes exactly 👍 Good news.

Woot! What about clearml-agent 1.2.0? 🙂

(Following this)

No, fixed 🙂

And actually it fails on quite many tasks for us with this Python 3.6.
I tried to set up a different image ( agent8sglue.defaultContainerImage: "ubuntu:20.04" ) but that did not change much.
I suspect the culprit is agentk8sglue.image , which is set to tag 1.24-21 of clearml-agent-k8s-base . That image is quite very old… Any updates on that? 🤔

That is, we have something like:
` task = Task.init(...)
ds = Dataset.create(dataset_name=task.name, dataset_project=task.get_project_name(), use_current_task=True)

upload files

dataset.upload(show_progress=True)
dataset.finalize()

do stuff with task and dataset

task.close() `But because the dataset is linked to the task, the task is then moved and effectively becomes invisible 😕

Any thoughts AgitatedDove14 SuccessfulKoala55 ?

SmugDolphin23 we've been working with this for 2 weeks now, and it creates a lot of junk in our UI. Is there anyway to have better control over this?

Ah right, I missed that in the codebase. It just adds the .dataset convention to the dataset task.

JitteryCoyote63 yes exactly, sorry, I forgot to add the Task.get_task in my response. That's exactly what we do 😅