StorageManager.download_folder(remote_url=' s3://some_ip:9000/clearml/my_folder_of_interest ', local_folder='./') yields a new folder structure, ./clearml/my_folder_of_interest , rather than just ./my_folder_of_interest

No I don't this at all

We're still working these quirks out. But one issue after we changed the AMI is that the VPC (SubnetId?) was missing from the instance so it could not reach the ClearML API server.

I think maybe the autoscaler service is missing some additional settings...

Thought it might be via docker, thanks!

It was really easy with the attached code, really 👍

I would only maybe suggest adding in the documentation, that if one uses the default recommended install location, then the script can be run without any command line arguments.
I had to momentarily look at the code to see the default paths match my own (though I could've also looked at --help default values 😛 )

SuccessfulKoala55 TimelyPenguin76
After looking into it, I think it's because our AMI does not have docker, and that the default instance suggested by ClearML auto scaler example is outdated

Nope, no other config files

Is there some default Docker image you ship with ClearML that you'd recommend, or can/should we use our own? 🙂

I’d like to refrain from manually specifying the dependencies, since it adds a lot of overhead to extend

Sorry to keep this up - what about support for minio using the environment variable? Do I set the CLEARML_FILES_HOST to the end point instead of an s3 bucket?

Ah! Makes sense. Thanks!

With clearml-agent 1.5.1

FYI @<1523701087100473344:profile|SuccessfulKoala55> (or I might be doing something wrong), but it seems the python migration code comes with carriage returns, so it fails on linux by default (one has to tr -d '\r' to use it)

EDIT: And also it defaults to /opt/allegro/data rather than the recommended /opt/clearml/data which is suggested when installing the server 🤔

Anything specific we should look into TimelyPenguin76 ?

Is it CLEARML_CONFIG_FILE ? (I had to dig this from the GH code 😅 )

It's a bit hard to read when they'll all clustered together:

i.e.
ERROR Fetching experiments failed. Reason: Backend timeout (600s)
ERROR Fetching experiments failed. Reason: Invalid project ID

Generally the StorageManager seems a bit slow, even a simple StorageManager.list(...) on a local path seems to take a long time

No worries @<1537605940121964544:profile|EnthusiasticShrimp49> ! I made some headway by using Task.create , writing a temporary Python script, and using task.update in a similar way to how pipeline steps are created.

I'll try and create an MVC to reproduce the issue, though I may have strayed from your original suggestion because I need to be able to use classes and not just functions.

From the log you shared, the task is picked up by the 

worker_d1bd92a3b039400cbafc60a7a5b1e52b_4e831c4cbaf64e02925b918e9a3a1cf6_<hostname>:gpu0,1

 worker

I can try and target the default one if it helps..?

I can elaborate in more detail if you have the time, but generally the code is just defined in some source files.
I’ve been trying to play around with pipelines for this purpose, but as suspected, it fails finding the definition for the pickled object…

Great to hear @<1523701087100473344:profile|SuccessfulKoala55> ! Is there an estimated timeline for these releases?

@<1537605940121964544:profile|EnthusiasticShrimp49> It’ll take me still some time to find the MVC that generated this, but I do have the ClearML experiment page for it. I was running the thing from ipython , and was trying to create a task from a function:

This is related to my other thread, so I’ll provide an example there -->

Thanks SuccessfulKoala55 and AgitatedDove14 ! We'll go through the hoops of setting up mongo on AWS then.
We're working to decouple the data from the helm chart, seems like a dangerous idea to store long term data on k8s in case of failure 😅

Heh, well, John wrote that in the first reply in this thread 🙂
And in Task.init main documentation page (nowhere near the code), it says the following -

Could also be that the use of ./ is the issue? I'm not sure what else I can provide you with, SweetBadger76

Now, the original pyhocon does support include statements as you mentioned - https://github.com/chimpler/pyhocon

Seemed to work fine again in detached mode, what went wrong there :shocked_face_with_exploding_head:

SuccessfulKoala55 help me out here 🙂
It seems all the changes I make in the AWS autoscaler apply directly to the virtual environment set for the autoscaler, but nothing from that propagates down to the launched instances.
So e.g. the autoscaler environment has poetry installed, but then the instance fails because it does not have it available?