I'll have some reports tomorrow I hope TimelyPenguin76 SuccessfulKoala55 !

None, they're unusable for us.

These are per-user. Essentially we log user DB access as well (for various backtracking afterwards), so it's beneficial for us to pass the user DB secrets to the task and not have it configured once on the agent.

It's of course not an MLOps issue so I understand it's not high on the priority list, but would be kinda cool to just have a simple view presenting the content of users.get_all 😄

You could probably either:
Start the task first (using Task.init ), and then set the parameters if needed Attach the dataset to the task itself

The api.files_server is set to the MinIO endpoint s3://ip:9000/clearml (both locally and remotely) The sdk.development.default_output_uri is set to the MinIO endpoint (both locally and remotely) When we call Task.init I do not set the output_uri at all I get the logger directly with task.get_logger()

No I don't this at all

with s3

I thought so too - so I added flush calls just in case, but nothing's changed.
This is somewhat weird since it always happens in the above scenario (Ray + ClearML), and always in the last task/job from Ray

https://clearml.slack.com/archives/CTK20V944/p1652350709961419

We’d be happy if ClearML captures that (since it uses e.g. pip, then we have the git + commit hash for reproducibility), as it claims it would 😅

Any thoughts CostlyOstrich36 ?

Basically you have the details from the Dataset page, why should it be mixed with the others ?

Because maybe it contains code and logs on how to prepare the dataset. Or maybe the user just wants increased visibility for the dataset itself in the tasks view.

why would you need the Dataset Task itself is the main question?

For the same reason as above. Visibility and ease of access. Coupling relevant tasks and dataset in the same project makes it easier to understand that they're...

AgitatedDove14 Basically the fact that this happens without user control is very frustrating - https://github.com/allegroai/clearml/blob/447714eaa4ac09b4d44a41bfa31da3b1a23c52fe/clearml/datasets/dataset.py#L191

Maybe they shouldn't be placed under /tmp if they're mission critical, but rather the clearml cache folder? 🤔

@<1523701205467926528:profile|AgitatedDove14> this

Looks great, looking forward to the all the new treats 😉
Happy new year! 🎉

I think -

• Creating a pipeline from tasks is useful when you already ran some of these tasks in a given format, and you want to replicate the exact behaviour (ignoring any new code changes for example), while potentially changing some parameters.
• From decorators - when the pipeline logic is very straightforward and you'd like to mostly leverage pipelines for parallel execution of computation graphs
• From functions - as I described earlier :)

I see that the GUI AutoScaler is only in the paid version, wonder why the GCP driver is not open source?

SweetBadger76 TimelyPenguin76
We're finally tackling this (since it has kept us back at 1.3.2 even though 1.6.2 is out...), and noticed that now the bucket name is also part of the folder?

So following up from David's latest example:
StorageManager.download_folder(remote_url='s3://****-bucket/david/', local_folder='./')Actually creates a new folder ./****-bucket/david/ and puts it contents there.

EDIT: This is with us using internal MinIO, so I believe ClearML parses that end...

Yes, thanks AgitatedDove14 ! It's just that the configuration object passed onwards was a bit confusing.
Is there a planned documentation overhaul? 🤔

Any updates on this? We can't do anything with our K8s since this 404...

I see, okay that already clarifies some stuff, I'll dig a bit more into this then! Thanks!

@<1523704157695905792:profile|VivaciousBadger56> It seems like whatever you pickled in the zip file relies on some additional files that are not pickled.

I can also do this via Mongo directly, but I was hoping to skip the K8S interaction there.

I'm also getting the following warning, I guess it's some ClearML dependency?
IPython could not be loaded!

Where do I import this APIClient from AgitatedDove14 ? I meanwhile edited it directly in mongo, but editing a db directly on a Friday is a big nono

An internal project I've accidentally made with a hidden tag while playing around with the ClearML internal code.

I'd like to remove the hidden system tag from a project

Hm. Is there a simple way to test tasks, one at a time?

We can change the project name’s of course, if there’s a suggestion/guide that will make them see past the namespace…