Each user creates a .env file for their needs or exports them in the shell running the python code. Currently I copy the environment variables to an S3 bucket and download it from there.

One way to circumvent this btw would be to also add/use the --python flag for virtualenv

This seems to be fine for now, if any future lookups finds this thread, btw
with mock.patch('clearml.datasets.dataset.Dataset.create'): ...

You mean the host is considered the bucket, as I wrote in my earlier message as the root cause?

Still the same issue it seems (or maybe the behaviour is the new behaviour?)

Yes

Adding bucket = clearml in aws.s3.credentials did not help either

Okay so the new functionality is maintained; thanks!

Thanks David! I appreciate that, it would be very nice to have a consistent pattern in this!

Created - https://github.com/allegroai/clearml/issues/709

I have no tried to download the entire file, but yes

One more thing that may be helpful SweetBadger76 , I've gone ahead and looked into clearml.storage.helper , and found that at least if I specify the bucket name directly in the aws.s3.credentials configuration for MinIO, then:
In [4]: StorageHelper._s3_configurations.get_config_by_uri(' ` ')
Out[4]: S3BucketConfig(bucket='clearml', host='some_ip:9000', key='xxx', secret='xxx', token='', multipart=False, acl='', secure=False, region='', verify=True, use_credentials_chain=False)...

Could also be that the use of ./ is the issue? I'm not sure what else I can provide you with, SweetBadger76

The bucket is not a folder, it's just a container. Whether it's implemented as a folder in MinIO should be transparent, shouldn't it?

Since the "fix" in 1.4.0 onwards, we now have to download the folder, and then move all the downloaded files/folders to the correct level.
This now entails we also have to check which storage is used, so we can check if the downloaded folder will contain the bucket name or not, which seems very inconsistent?

It could be related to ClearML agent or server then. We temporarily upload a given .env file to internal S3 bucket (cache), then switch to remote execution. When the remote execution starts, it first looks for this .env file, downloads it using StorageManager, uses dotenv, and then continues the execution normally

SweetBadger76 TimelyPenguin76
We're finally tackling this (since it has kept us back at 1.3.2 even though 1.6.2 is out...), and noticed that now the bucket name is also part of the folder?

So following up from David's latest example:
StorageManager.download_folder(remote_url='s3://****-bucket/david/', local_folder='./')Actually creates a new folder ./****-bucket/david/ and puts it contents there.

EDIT: This is with us using internal MinIO, so I believe ClearML parses that end...

StorageManager.download_folder(remote_url=' s3://some_ip:9000/clearml/my_folder_of_interest ', local_folder='./') yields a new folder structure, ./clearml/my_folder_of_interest , rather than just ./my_folder_of_interest

https://pypi.org/project/python-dotenv/

Well you can install the binary in the additional start up commands.
Matter of fact, you can just include the ECR login in the "startup steps" offered by the scaler, so no need for this repository. I was thinking these are local instances.

Not sure if ClearML has any built in support, but we used the above for a similar issue but with Prefect2 :)

(the extra_vm_bash_script is what you're after)

That's up and running and is perfectly fine.

Thought it might be via docker, thanks!

I would expect the service to actually implicitly inject it to new instances prior to applying the user's extra configuration 🤔

Either, honestly, would be great. I meant even just a link to a blank comparison and one can then add the experiments from that view

Hey AgitatedDove14 🙂
Finally managed; you keep saying "all projects" but you meant the "All Experiments" project instead. That's a good start 👍 Thanks!

Couple of thoughts from this experience:
Could we add a comparison feature directly from the search results (Dashboard view -> search -> highlight some experiments for comparison)? Could we add a filter on the project name in the "All Experiments" project? Could we add the project for each of the search results? (see above pictur...

2. I guess? 🤔 I mean the same filter option one has for e.g. tags in the table view. In the "all experiments" project I think it would make sense for one to be able to select the projects of interest, or even filter for textual matches.

3. Sorry I meant the cards indeed :)

Managed now 🙂 Thank you for your patience!
I edited the previous post with some suggestions/thoughts

Unfortunately I can't take a photo of not being able to compare tasks by navigating around the WebUI...

Does that clarify the issue CostlyOstrich36 ?

Ah okay 😄 Was confused by what you quoted haha 👍