Created this for follow up, SuccessfulKoala55 ; I'm really stumped. Spent the entire day on this 🥹
https://github.com/allegroai/clearml-agent/issues/134

Ah. In the extra_vm_bash_script of the AWS autoscaler.

I'll try that in a bit (that requires some access control changes). Any idea how can I modify the dynamically created virtualenv?

` Poetry Enabled: Ignoring requested python packages, using repository poetry lock file!
The currently activated Python version 3.10.6 is not supported by the project (~3.8.0).
Trying to find and use a compatible version.
Using python3.8 (3.8.16)
Creating virtualenv ... in /root/.clearml/venvs-builds/3.10/task_repository/...git/.venv
Installing dependencies from ...

Let me have a quick look.

I'm not entirely sure I understand the flow but I'll give it a go. I have two final questions:
This seems to only work for a single file (weights_path implies a single file, not multiple ones). Is that the case? Why do you see this as preferred to the dataset method we have now? 🤔

Ah, uhhhh whatever is in the helm/glue charts. I think it’s the allegroai/clearml-agent-k8s-base , but since I hadn’t gotten a chance to try it out, it’s hard to say with certainty which would be the best for us 😛

Since the additional credentials are available to the autoscaler when it boots up (via the config file), I thought it could use those natively?

https://pypi.org/project/python-dotenv/

I... did not, ashamed to admit. The documentation says only boolean values.

There used to be a good example but it's now missing. I'm not sure what does Use only for automation (externally), otherwise use Task.connect_configuration mean when e.g. looking at Task.set_configuration_object , etc.

Could you clarify a bit, CostlyOstrich36 or AgitatedDove14 ?

It does, but I don't want to guess the json structure (what if ClearML changes it or the folder structure it uses for offline execution?). If I do this, I'm planning a test that's reliant on ClearML implementation of offline mode, which is tangent to the unit test

This is with:
Task.set_offline_mode(True) task = Task.init(..., auto_connect_streams=False)

I'm running tests with pytest , it consumes/owns the stream

It is

The bucket is not a folder, it's just a container. Whether it's implemented as a folder in MinIO should be transparent, shouldn't it?

Since the "fix" in 1.4.0 onwards, we now have to download the folder, and then move all the downloaded files/folders to the correct level.
This now entails we also have to check which storage is used, so we can check if the downloaded folder will contain the bucket name or not, which seems very inconsistent?

Sounds like incorrect parsing on ClearML side then, doesn't it? At least, it does not fully support MinIO then

I don't imagine AWS users get a new folder named aws-key-region-xyz-bucket-hostname when they download_folder(...) from an AWS S3 bucket, or do they? 🤔

You mean the host is considered the bucket, as I wrote in my earlier message as the root cause?

That's fine for the current use-case I believe.
Once the team is happy with the logging functionality, we'll move on to remote execution and things will update.

No, I have no running agents listening to that queue. It's as if it's retained in some memory somewhere and the server keeps creating it.

The network is configured correctly 🙂 But the newly spun up instances need to be set to the same VPC/Subnet somehow

AgitatedDove14 yeah I see this now; this was an issue because I later had to "disconnect" the remote task, so it can, itself, create new tasks (using clearml.config.remote.override_current_task_id(None) ). I guess you might remember that discussion? 😁

EDIT: It's the discussion we had here, for reference. https://clearml.slack.com/archives/CTK20V944/p1640955599257500?thread_ts=1640867211.238900&cid=CTK20V944
So probably not needed in JitteryCoyote63 's case, we still have some...

It's self-hosted TimelyPenguin76
SuccessfulKoala55 WebApp: 1.4.0-175 • Server: 1.4.0-175 • API: 2.18

That could work, given that:
Could we add a preview section? One reason I don't like using the configuration section is that it makes debugging much much harder. Will the clearml-agent download and unzip the files, placing them into the same local folder as needed for execution? What if we want to include non-configuration objects? (i.e. the model case I listed)

AFAIK that's the only way right now (see my comment here - https://clearml.slack.com/archives/CTK20V944/p1657720159903739?thread_ts=1657699287.630779&cid=CTK20V944 )
Or then if you have the ClearML paid service, I believe there is a "vaults" service, right AgitatedDove14 ?

https://clear.ml/docs/latest/docs/references/sdk/services_monitor

Then you can run this as a task, see also this example https://clear.ml/docs/latest/docs/guides/services/slack_alerts

I'm not too worried about the dataset appearing (or not) in the Datasets tab. I would like it (the original task ) to to not disappear from the original project I assigned it to

I'll try with 1.1.5 first, then 1.1.6rc0

Thanks LazyFish41 , that explains it 👍

That's weird -- the concept of "root directory" is defined to a bucket. There is no "root dir" in S3, is there? It's only within a bucket itself.
And since the documentation states:

If we have a remote file

then StorageManager.download_folder(‘

’, ‘~/folder/’) will create ~/folder/sub/file.ext

Then I would have expected the same outcome from MinIO as I do with S3, or Azure, or any other blob container

Running a self-hosted server indeed. It's part of a code that simply adds or uploads an artifact 🤔