Hi @<1556812486840160256:profile|SuccessfulRaven86> , just to make things easier, can you comment out these 3 lines in the config file? This will cause the sdk to have default behavior. Afterwards try with store_code_diff_from_remote: false

What do you see in the uncommitted changes section of the experiment?

Sure, here is the updated clearml.conf file of the AWS autoscaler instance:

agent {
    vcs_cache.enabled: false

    package_manager: {
          type: poetry,
          poetry_version: "1.4.2",
     }  
}

sdk {
    development {
         store_code_diff_from_remote: false,
    }

}

I see uncommited changes, where as I would like to have nothing.

These changes reflect the modifications I have in my staging area (not commited, not put in staging area with git add ) But I would like to remove this uncommited section from clearml and not be blocked by it

If I may ask as well for another issue in that thread that is taking me a big amount of time:

Poetry Enabled: Ignoring requested python packages, using repository poetry lock file!
Creating virtualenv alfred-Rp77Shgw-py3.9 in /root/.cache/pypoetry/virtualenvs
Installing dependencies from lock file
2023-04-17 10:17:57
Package operations: 351 installs, 1 update, 1 removal
failed installing poetry requirements: Command '['poetry', 'install', '-n']' returned non-zero exit status 1.
Ignoring pip: markers 'python_version >= "3.10"' don't match your environment
Collecting pip<20.2
  Using cached pip-20.1.1-py2.py3-none-any.whl (1.5 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 23.0.1
    Uninstalling pip-23.0.1:
      Successfully uninstalled pip-23.0.1
Successfully installed pip-20.1.1

I am supposed to use poetry package manager, and the snippet above proves it. However, it directly fails and uses pip manager. Do you have any ideas why? I tried adding -vvv flag but I have no more logs about the issue. Very surprisely, when I run the same code on a google colab clearm agent with same configuration, I do not have this poetry fail

I am spending ages on this issue, but until now, no fixes have been found 😞
I was telling myself that I could switch to Pip (which is the default behavior when poetry fails), but not all packages are correclty retrieved from clearml, so it cannot work.

Ok. I spinned up three AWS autoscalers, each with different conf. I also fixed a submodule issue in my repo (which I was believing was the problem of the git diff) and every run now passes and fails after (not this problem). So I think store_code_diff_from_remote is of no help from me but my problem is gone...