I think ClearML boots up only afterwards, so those environment variables may not be available yet.
You should set them manually in the bootstrap code unfortuantely.

the environment setting you added to your vault is only applied inside the instance when the agent starts running there, not as part of the command that starts the instance.
The most common DevOps practice for having these kind of variables in the init script but not completely exposed to the naked eye is by adding something like

export MY_ENV_VAR=$(echo '<base64-encoded secret>' | base64 --decode)

to the init script （編集済み)

You are right Idan,
I consulted our Private ClearML channel.
you cannot insert these environment variables any other place,
only in init script.
Here is the full quote:

I tried.
it looks like this,

sudo apt update
sudo apt install amazon-ecr-credential-helper
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin ****

But my problem is that I can't even see whether it passes my init script properly (tried to add printing comment but I cannot see the output) anywhere (nor scaler, nor task)

Important to notice I am running my instances on GCP, but the container is on ECR (AWS)

Does it help to also run docker login in the init bash script?
You should be able to access your AWS credentials from the environment (the agent will inject them based on your config)

I see. Are you able to manually boot a VM on GCP and then manually SSHing into it and running the docker login command from there? Just to be able to cross out networking or permissions as possible issues.