I have started the autoscaler with python? @<1523701070390366208:profile|CostlyOstrich36>

@<1523701070390366208:profile|CostlyOstrich36>

@<1523701070390366208:profile|CostlyOstrich36> I don't get that.

@<1523701070390366208:profile|CostlyOstrich36> I have been exploring. The problem seems to be when the docker container is using the cached dir.

Using cached repository in "/root/.clearml/vcs-cache/****.git.0081a6bc4d7afe6adde369e6aeab9406/****.git"

When inside that directory and tries to fetch, it asks for credentials. when it clones, it doesn't.

cloning: git@github.com:****/****.git
Using user/pass credentials - replacing ssh url 'git@github.com:****/****.git' with https ...

The devops changed the url and I had to go through some steps to find out what the problem was.

I don't say I am expert in this thing, but does the autoscaler have different version than the clearml?

I will do that

deployed it myself. It worked fine before I changed to ubuntu24.04 yesterday. We have been using clearml for years this way

That is the configuration yaml.

Thank you so much.

The server is running on cloud.

Yeah actually you are right. I can report the stats as a table, not the whole data.

It doesn't work when I insert the credentilas individually either. I am using EC2 as clearml server.

Thanks CostlyOstrich36
But, I am not using report_media() function. The debug samples (confusion matrices) are saved from tensorboard.

clearml==1.14.1
That is the version.

import yaml
from clearml.automation.auto_scaler import AutoScaler, ScalerConfig
from gcp_driver import GCPDriver

with open('gcp_autoscaler.yaml') as f:
    conf = yaml.load(f, Loader=yaml.SafeLoader)

    driver = GCPDriver.from_config(conf)

    conf = ScalerConfig.from_config(conf)
    autoscaler = AutoScaler(conf, driver)
    autoscaler.start()

That is the python code.

or can I put something like the following in clearml.conf?

cat << EOF >> ~/clearml.conf
agent.enable_git_ask_pass=true
agent.git_user="{GIT_USER}"
agent.git_pass="{GIT_PASSWORD}"

The

agent.git_user="{GIT_USER}"
agent.git_pass="{GIT_PASSWORD}"

already existed. I only added

`agent.enable_git_ask_pass=true

I see. So, is it same thing when network is slow and there is a mistake in url?

because, the param name that takes in the df is table_plot

I don't think I am on the pro version, is that a paid one?

@<1576381444509405184:profile|ManiacalLizard2>

Oh, I think that is for a very small data. I don't think it works for me.

I didn't write this conf, but it works.

they are different tasks. I start a new task but it can be same commit sometimes.

It works on other machines. Can I clearml-init on a virtual environment? I installed clearml in a virtual environment.

It is an autoscaler for gcp. I think, there are unnecessary configs that were used in aws.

The worker machines are on gcp

created virtual environment CPython3.10.13.final.0-64 in 511ms
  creator CPython3Posix(dest=/root/.clearml/venvs-builds/3.10, clear=False, no_vcs_ignore=False, global=True)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
    added seed packages: pip==23.3.1, setuptools==69.0.2, wheel==0.42.0
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
...

CloudySwallow27

configurations:
  extra_clearml_conf: 'sdk.aws.s3.region="us-west-2"
                        agent.extra_docker_arguments=["--shm-size=90g"]
                        agent.extra_docker_shell_script=["git config --global credential.helper cache --timeout=604800",]'
  extra_trains_conf: ''
  extra_vm_bash_script: ''
  queues:
    gcp-v100:
    - - gcp-v100
      - 4
    gcp-l4:
      - - gcp-l4
        - 4
    gcp-cpu:
    - - gcp-cpu
      - 4
  resource_configurations:
    gcp-v100:
    ...