Hi WittyOwl57 , currently, ClearML only supports an account key when connecting to azure. We'd love a contribution to support other methods, if you have the time 🙂
Hi Jake, thanks for the reply. I've tried the account key method, works fine - but unfortunately clearml expects an old version of azure-storage-blob (<2.1), which is incompatible with the recent versions (^12.). Any clues of how we could work around this one? Thanks again.
Well, same answer 🙂 - we haven't gotten around to supporting the new azure versions
but again, we'd be thrilled to get a PR on the subject 🙂
If we decide go forward with clearml we'll probably do just that 🙂
Cool 🙂 - let me know if there's anything else I can help with 👍
Hi again. After looking into the matter a little bit, I realise I'd have liked having the option of using a StoreManager ABC which I would implement myself using whatever storage provider I happen to use and whatever package versions happened to support it. To put it differently, instead of you implementing managers for gcs, azure, aws, etc, it would be a much nicer alternative (for me, and I suspect eventually for you too) for clearml's store manager to wrap whatever object the user provides, as long as it implements the necessary methods ( list , upload_file , download_file , download_folder , etc). I'd be happy to help with this.
Hi WittyOwl57
That's actually how it works (original idea/design was borrowed from libclound), basically you need to create a Drive, then the storage manger will use it.
Abstract class here:
https://github.com/allegroai/clearml/blob/6c96e6017403d4b3f991f7401e68c9aa71d55aa5/clearml/storage/helper.py#L51
Is this what you had in mind ?
BTW: if you could implement _AzureBlobServiceStorageDriver with the new Azure package, it will be great:
Basically update this class:
https://github.com/allegroai/clearml/blob/6c96e6017403d4b3f991f7401e68c9aa71d55aa5/clearml/storage/helper.py#L1620
Hi Martin, to expand on my previous comments: the template for _Driver already exists; I'm suggesting to make it public. Consequently, StorageHelper should accept a driver parameter to __init__ , defaulting to None . Only when its value is not provided by the user should the library go out of its way to do the right thing and check all the known storage providers, fetch credentials, what not - stuff that will not work for most users, most of the time (even if you put in an inordinate amount of effort, you'll still not be able to match whatever version of, say, azure-blob-storage the user happens to have installed, or to fetch the credentials correctly in every setting - e.g. when the user uses managed identity).
I'm suggesting to make it public.
Actually I'm thinking of enabling users to register Drivers in runtime, expanding the capability to support any type of URL link, meaning you can register "azure://" with AzureDriver, and the StorageHelper will automatically use the driver you provide.
This will make sure Any part of the system will be able to transparently use any custom driver.
wdyt?
WittyOwl57 I think this is a great idea, can you open a feature issue on GitHub so this is not forgotten ?
BTW: regardless, if you have time to upgrade to the new the azure package upgrade, it will be great 🙂 this is on our to do list for a while, but since not a lot of users complained it got pushed ...