I mean usually it would read if cached_file: return cached_file

😄

In the larger context I'd look on how other object stores treat similar problems, I'm not that advanced in these topics.

But adding a simple force_download flag to the get_local_copy method could solve many cases I can think of, for example I'd set it to true in my case as I don't mind the times it will re-download when not necessary as it is quite small (currently I always delete the local file, but it looks pretty ugly)

I might, I'll look at the internals later cause at a glance I didn't really get the logic inside get_local_copy ... the if there is ending with if ... not cached_file: return cached_file which from reading doesn't make much sense

👍

found it here:
https://github.com/allegroai/trains/blob/98ea965e6d0a39019b31e5fec146ab55da0d52e8/trains/storage/manager.py#L41

I assume that at some points in the execution, the client (where the task is running) is sending JSONs to the mongo service, and that is what we see in the web UI.
Since we are talking about a case where there is no internet available, maybe these could be dumped into files/stdout and let the user manually insert them.
The manual insertion UX could be something like a CLI copy-paste or and endpoint for files - but since your UX is so good ( 🙂 ) I'm sure you'll figure this part out better

glad I managed to help back in some way

The scenario I'm going for is never to run on the dev machine, so all I'll need to do once the server + agents are up is to add task.execute_remotely... after the Task.init line and after the execution of the script is called on the dev machine, it won't actually run but rather enqueue itself for the agent to run it?

I was sure you are on Israel times as well, sorry for the night time thing 😄

I guess so

💪

The latest, I curl ed the docker-compose like 10 minutes ago

cool the /login worked

Okay, I'll try that

can you tell me which API call exactly are you using for spinning up? I would like to debug and try to use boto3 myself in order to spin up an instance, so I can understand where the problem is coming from

no need to do it again, I ahve all the settings in place, I'm sure it's not a settings thing

So just to correct myself and sum up, the credentials for AWS are only in the cloud_credentials_*

I doubled checked the credentials in the configurations, and they have full EC2 access

So what's our next move FriendlySquid61 ? O_O

the trains ones

Now I remind you that using the same credentials exactly, the auto scaler task could launch instances before

and when looking at the running task, I still see the credentials

the ones in the export are like you said

Actually I removed the key pair, as you said it wasn't a must in the newer versions

I'll try to debug this and updat

I'll try to put it again

now I get this error in my Auto Scaler task
Warning! exception occurred: An error occurred (AuthFailure) when calling the RunInstances operation: AWS was not able to validate the provided access credentials Retry in 15 seconds

I have them in two different places, once under Hyperparameters -> General

I ran the wizard liek a milion times

and I'm using the same credentials