Generally, really. I've struggled recently (and in the past), because the documentation seems:
Very complete wrt available SDK (though the formatting is sometimes off) Very lacking wrt to how things interact with one anotherA lot of what I need I actually find from pluging into the source code.
I think ClearML would benefit itself a lot if it adopted a documentation structure similar to numpy ecosystem (numpy, pandas, scipy, scikit-image, scikit-bio, scikit-learn, etc)

SweetBadger76 TimelyPenguin76
We're finally tackling this (since it has kept us back at 1.3.2 even though 1.6.2 is out...), and noticed that now the bucket name is also part of the folder?

So following up from David's latest example:
StorageManager.download_folder(remote_url='s3://****-bucket/david/', local_folder='./')Actually creates a new folder ./****-bucket/david/ and puts it contents there.

EDIT: This is with us using internal MinIO, so I believe ClearML parses that end...

On my remote server..?

The last kind when launching the agents.
The former on task execution.

Say I upload each of these yamls as a configuration object (as with the above). Once I try to load bar.yaml remotely it will crash, since foo.yaml is missing (and is instead a clearml configuration object).
Does that make sense?

UPDATE: Apparently the quotation type matters for furl ? I switched the ' to \" and it seems to work now

This is with:
Task.set_offline_mode(True) task = Task.init(..., auto_connect_streams=False)

Maybe it's better to approach this the other way, if one uses Task.force_requirements_env_freeze() , then the locally updated packages aren't reflected in poetry 🤔

Interesting, why won’t it be possible? Quite easy to get the source code using e.g. dill .

I'll try it out, but I would not like to rewrite that code myself maintain it, that's my point 😅

Or are you suggesting I Task.import_offline_session ?

QuaintPelican38 did you have a workaround for this then? Some cleanup service or similar?

I guess I'll have to rerun the experiment without tags for this?

Hm, I'm not sure I follow 🤔 How does the API server config relate to the file server?

Yes it would be 🙂
Visualization is always a difficult topic... I'm not sure about that, but a callback would be nice.

One idea that comes to mind (this is of course limited to DataFrames), but think the git diff , where I imagine 3 independent section:
Removed columns (+ truncated preview of removed values) (see below) Added columns (+ truncated preview of removed values)
The middle column is then a bit complicated, but I would see some kind of "shared columns" dataframe, where each ...

The bucket is not a folder, it's just a container. Whether it's implemented as a folder in MinIO should be transparent, shouldn't it?

Since the "fix" in 1.4.0 onwards, we now have to download the folder, and then move all the downloaded files/folders to the correct level.
This now entails we also have to check which storage is used, so we can check if the downloaded folder will contain the bucket name or not, which seems very inconsistent?

Yes, you're correct, I misread the exception.
Maybe it hasn't completed uploading? At least for Datasets one needs to explicitly wait IIRC

Does it make sense to you to run several such glue instances, to manage multiple resource requirements?

Oh, well, no, but for us that would be one way solution (we didn't need to close the task before that update)

The overall flow I currently have is e.g.
Start an internal task (not ClearML Task; MLOps not initialized yet) Call some pre_init function with args so I can upload the environment file via StorageManager to S3 Call some start_run function with the configuration dictionary loaded, so I can upload the relevant CSV files and configuration file Finally initialize the MLOps (ClearML), start a task, execute remotely
I can play around with 3/4 (so e.g. upload CSVs and configuratio...

Yeah that works too. So one can override the queue ID but not the worker 🤔

Added the following line under volumes for apiserver , fileserver , agent-services :
- /data/clearml:/data/clearml

TimelyPenguin76 CostlyOstrich36 It seems a lot of manual configurations is required to get the EC2 instances up and running.
Would it not make sense to update the autoscaler (and example script) so that the config.yaml that's used for the autoscaler service is implicitly copied to the EC2 services, and then any extra_clearml_conf are used/overwritten?

No need for S3 or other resources?

Doh, thanks!

Right and then for text (file path) use some regex or similar for extraction, and for dictionary simply parse the values?

Running a self-hosted server indeed. It's part of a code that simply adds or uploads an artifact 🤔

Yes

Am I making sense ?

No, not really. I don't see how task.connect_configuration interacts with our existing CLI? Additionally, the documentation for task.connect_configuration say the second argument is the name of a file, not the path to it? So something is off

But it does work on linux 🤔 I'm using it right now and the environment variables are not defined in the terminal, only in the .env 🤔

Okay so the only missing thing of the puzzle I think is that it would be nice if this propagates to the autoscaler as well; that then also allows hiding some of the credentials etc 😮