Can you compare the instance you're starting with the one the autoscaler is starting? You can see the User Data (System log) section in the AWS dashboard which will show exactly what the bash script running there does, and you know the exact command the agent there uses to run the image - you can manually do both and see where it fails, I assume this is either the AMI you're using, or some other system configuration

Hi @<1597762318140182528:profile|EnchantingPenguin77> , isn't the problem simply an issue with not enough disk space on your AWS EC2 instance?

Well, it's a little verbose, I agree 🙂
I'm sure you know, but for the community's sake I'll just say it's possible to use a more pythonic syntax instead of the for loop:
model = next(m for m in Task.get_task(training_task_id).get_models()["output"] if m.name == model_name)This will raise StopIteration error if no such model is found, or alternatively:
model = next((m for m in Task.get_task(training_task_id).get_models()["output"] if m.name == model_name), None)Which will retu...

If you're running the agent in docker mode, and assuming you're running in GCP or some other cloud solution, you can theoretically use the custom bash script configuration option for the agent to pull a file from some sort of secrets vault solution provided by the cloud provider and place on the machine for the duration of the task execution (or set the secret in an env var)

It's not in your code, right?

Well, if it won't be logged, the agent won't be able to use it when running remotely...

Well, I think the paid version should have some way of handling this in a centralized way

Hi TartSeal39 ,
Using this value as an argv (i.e. using argparse) will store it in the task's hyperparameters section and will make it accessible when executing remotely

Hi @<1687643893996195840:profile|RoundCat60> ,

What are the credentials that are referred to in the logs? Where do I get these?

These are part of the server's docker-compose configuration. See here: None

How do I ensure this container is run automatically if it keeps restarting using the docker-compose file

I'm not sure I understand - docker-compose will constantly try to run the container. Right no...

we turn off the server every evening...

In that case the issue is definitely not related to the mount points

(without the actual credentials, of course)

Not necessarily, is there any data in those directories?

Hi @<1687643893996195840:profile|RoundCat60> ,

Passwords should be encrypted before being stored

That's definitely an option (we had the same idea), however it would mean a more complex configuration procedure and might male it hard for less-savvy people to setup the server. Considering the fact that anyone with access to the server can basically access all data anyway, as @<1523701205467926528:profile|AgitatedDove14> said, having the passwords for the account encrypted won't make o...

That's right 🙂

Can you perhaps attach your docker-compose.yml file's contents?

Hi @<1687653458951278592:profile|StrangeStork48> , the easiest way is to read Amazon's AWS docs regarding the Secrets Manager. Once you manage to use that, you can simply "inject" any values you require into config files or environment variables.

Hi @<1697419077955358720:profile|HappyAnt94> , I think your best bet is either an automated script on startup (simple), or an agent running in a docker container (and let the docker service start it automatically on server restart)

And when you create them yourself you can see them?

did you sign up with your own email?

Hi @<1681836303299121152:profile|RoundElk14> , are you using None ?

but did you go through the sign up process before trying to sign in?

We're certainly working hard on improving the documentation (and I do apologize for the frustrating experience)

Hi ShortCoral15 , this does sound like an issue related to a very large download - how big are your plots?

Hi @<1545216070686609408:profile|EnthusiasticCow4> , it sure looks like a rate limit issue - are you using a free GitHub account or a paid one?

Hi @<1717350332247314432:profile|WittySeal70> , can you please share the full log?

Hi UnevenHorse85 , this is perfectly acceptable 🙂 - you're basically using the internal reverse proxy that redirects internally to the appropriate services

Yes, sounds right

Yeah, I'm afraid neither one can be used to repopulate the other...

WickedElephant66 can you share an example?

Hi all, sorry for not being so responsive today 🙏