Hei @<1523701087100473344:profile|SuccessfulKoala55> it just worked. Maybe there was some github refresh delay … not sure but thanks anyways for the debug suggestion. 👍

Hi @<1523701087100473344:profile|SuccessfulKoala55> thanks for your reply. Not sure where I can find more about the extra docker bash script that you mention… I would appreciate if you can point me in the right direction. Thanks.

Update

I ran:

clearml-agent build --id <task-id> --docker <custom-docker> --log-level DEBUG --entry-point reuse_task

and got a similar problem:

Host key verification failed.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
error: Could not fetch origin

Also @<1619867994005966848:profile|HungryTurtle13> 👆

@<1523701205467926528:profile|AgitatedDove14> None

I can’t see anything under /mnt so no fileserver there (?)

None

A related question… how does the server know how to delete artifacts when the project is deleted if it doesn’t have a clearml.conf with the S3 credentials to do so?

What I am referring to is this information about the Storage Configuration:
None

I just ran a dummy experiment logging images, plots, etc and I can see them in my server’s Web UI.

ok so the documentation is confusing here:

there under fileserver it should read /opt/clearml/data/fileserver

Responding to my own question, in case someone else has the same issue. You have to edit the security group and enable TCP 8080.

I haven’t figure out the missing fileserver? :man-shrugging:

3fdcf5db64d allegroai/clearml:1.12.1-397 “/opt/clearml/wrappe…” 10 days ago Up 9 minutes 8008/tcp, 8080/tcp, 0.0.0.0:8081->8081/tcp, :::8081->8081/tcp clearml-fileserver

Hi @<1523701087100473344:profile|SuccessfulKoala55> it’s failing again.. I haven’t rebooted the agent or changed anything and I am able to connect with ssh with ssh -vT git@github.com on a different tmux sess.

This is the error I am seeing running the agent with the -debug flag:

Using cached repository in "/home/ubuntu/.clearml/vcs-cache/clearml-tutorial.git.e1c2351b09f3d661b6f0dbf85e92be2e/clearml-tutorial.git"
git@github.com: Permission denied (pub...

Hey @<1523701087100473344:profile|SuccessfulKoala55> I am not sure this is the case as the instance can checkout code in poetry/pip mode. This issue only happens if I try to run the agent in docker mode. I read in the docs that when you run the agent in docker mode the . ssh directory of the host is copied to the container under /root/.ssh so I have the theory that when I am building the custom docker image I don’t end up with a /root folder (?) I haven’t had the time to debug th...

thanks!

@<1523701087100473344:profile|SuccessfulKoala55> thanks so much for your reply. I can see now the source of my confusion:

After I finished deploying the server in AWS , the next step in that page is “ configuring ClearML for [ ClearM...

Hi @<1523701435869433856:profile|SmugDolphin23> thanks for your answer. I am not sure that I understand. I ran a test by cloning and experiment and editing the OmegaConf object under Configuration > Hyperparameters > OmegaConf.

Unless I also change the allow_omegaconf_edit flag to True , I won’t see my changes reflected. That is my question. As a new user, it seems counterintuitive that I have to also change the flag. Does this make sense to you? Thanks.

yeah that’s why it’s weird …

so it looks like the server is there (docker ps), I can see the artifacts (web ui), but not sure where things are as per documentation there is no /mnt/fileserver (?)

Do you have any insights on the missing fileserver @<1523701205467926528:profile|AgitatedDove14> ?

I see this in the docker-compose.yml file:

 fileserver:
    networks:
      - backend
      - frontend
    command:
    - fileserver
    container_name: clearml-fileserver
    image: allegroai/clearml:1.12.1-397
    environment:
      CLEARML__fileserver__delete__allow_batch: "true"
    restart: unless-stopped
    volumes:
    - /opt/clearml/logs:/var/log/clearml
    - /opt/clearml/data/fileserver:/mnt/fileserver
    - /opt/clearml/config:/opt/clearml/config
    ports:
    - "8081:...