How to make sure that the python version is correct?

@<1523701070390366208:profile|CostlyOstrich36> poetry is installed as part of the bash script of the task.

The init script of the AWS autoscaler only contains three export variables I set.

I tried too. I do not have more logs inside the ClearML agent 😞

I literrally connected to it at runtime, and ran poetry install -n and it worked

I also did that in the following way:

• I put a sleep inside the bash script
• I ssh-ed to the fresh container and did all commands myself (cloning, installation) and again it worked...

where did you specify it?

Yes should be correct. Inside the bash script of the task.

And I just tried with Python 3.8 (default version of the image) and it still fails.

Poetry Enabled: Ignoring requested python packages, using repository poetry lock file!
Creating virtualenv debug in /root/.clearml/venvs-builds/3.8/task_repository/clearmldebug.git/.venv
Using virtualenv: /root/.clearml/venvs-builds/3.8/task_repository/clearmldebug.git/.venv
2023-04-18 15:03:52
Installing dependencies from lock file
Finding the necessary packages for the current system
Package operations: 6 installs, 0 updates, 0 removals
failed installing poetry requirements: Command '['poetry', 'install', '-n', '-v']' returned non-zero exit status 1.

and are you sure these are the same env vars available when the agent does the same?

the autoscaler always uses docker mode

You can theoretically do that in the docker init bash script that will be executed before the task is cloned and run

I think you should try to manually start such a docker container and try to see what fails in the process. Attaching to an existing one has too many differences already

I am currently trying with a new dummy repo and I iterate over the dependencies of the pyproject.toml.

I am literrally trying with 1 package and python and it fails. I tried with python 3.8 3.9 and 3.9.16. and it always fail --> not linked to python version. What is the problem then? I am wondering if there is not an intrinsic bug

How is it still up is the task failed?

into the same docker container running the task?

The same container

@<1556812486840160256:profile|SuccessfulRaven86> , did you install poetry inside the EC2 instance or inside the docker? Basically, where do you put the poetry installation bash script - in the 'init script' section of the autoscaler or on the task's 'setup shell script' in execution tab (This is basically the script that runs inside the docker)

It sounds like you're installing poetry on the ec2 instance itself but the experiment runs inside a docker container

My issue has been resolved going with pip.

@<1523701087100473344:profile|SuccessfulKoala55> Do you think it is possible to ask to run docker mode in the aws autoscaler, and add the cloning and installation inside the init bash script of the task?

It just allows me to have access to poetry and python installed on hte container

@<1556812486840160256:profile|SuccessfulRaven86> , to make things easier to debug, can you try running the agent locally?

Yes, the problem is it's still really hidden (the error, I mean)

How do you explain that it works when I ssh-ed into the same AWS container instance from the autoscaler?

or just to the AWS instance?

When the task finally failed, I was kicked of from the container

This is really extremely hard to debug. I am thinking to create another repo and iterate on the packages to hopefully find the problem, but it will take ages.

Yes I take the export statements from my bash script of the task

Yes same error

So the same?