The same container

@<1523701070390366208:profile|CostlyOstrich36> @<1523701087100473344:profile|SuccessfulKoala55> I tried with dummy repo. Using Python and stripe packages ONLY in the pyproject.toml

Here is my result (still failing) :

Poetry Enabled: Ignoring requested python packages, using repository poetry lock file!
Creating virtualenv debug in /root/.clearml/venvs-builds/3.9/task_repository/clearmldebug.git/.venv
Using virtualenv: /root/.clearml/venvs-builds/3.9/task_repository/clearmldebug.git/.venv
Installing dependencies from lock file
Finding the necessary packages for the current system
Package operations: 6 installs, 0 updates, 0 removals
failed installing poetry requirements: Command '['poetry', 'install', '-n', '-v']' returned non-zero exit status 1.
Ignoring pip: markers 'python_version >= "3.10"' don't match your environment

I tried too. I do not have more logs inside the ClearML agent 😞

@<1556812486840160256:profile|SuccessfulRaven86> , did you install poetry inside the EC2 instance or inside the docker? Basically, where do you put the poetry installation bash script - in the 'init script' section of the autoscaler or on the task's 'setup shell script' in execution tab (This is basically the script that runs inside the docker)

It sounds like you're installing poetry on the ec2 instance itself but the experiment runs inside a docker container

I guess it makes no sense because of the steps a clearml-agent works...
I also thought about going to pip mode but not all packages are detected from our poetry.lock file unfortunately so cannot do that.

You can theoretically do that in the docker init bash script that will be executed before the task is cloned and run

Yes should be correct. Inside the bash script of the task.

Because I was ssh-ing to it before the fail. When poetry fails, it installs everything using PIP

I see it's running inside 3.9, so I assume it's correct

I literrally connected to it at runtime, and ran poetry install -n and it worked

So the same?

into the same docker container running the task?

My issue has been resolved going with pip.

Yes indeed, but what about the possibility to do the clone/poetry installation ourself in the init bash script of the task?

I think you should try to manually start such a docker container and try to see what fails in the process. Attaching to an existing one has too many differences already

where did you specify it?

the autoscaler always uses docker mode

Yes I take the export statements from my bash script of the task

@<1556812486840160256:profile|SuccessfulRaven86> can you try with -vvv instead of -v ?

It just allows me to have access to poetry and python installed on hte container

When the task finally failed, I was kicked of from the container

I am currently trying with a new dummy repo and I iterate over the dependencies of the pyproject.toml.

I also did that in the following way:

• I put a sleep inside the bash script
• I ssh-ed to the fresh container and did all commands myself (cloning, installation) and again it worked...

or just to the AWS instance?

How is it still up is the task failed?

but I still had time to go inside the container, export the PATH variables for my poetry and python versions, and run the poetry install command there

How to make sure that the python version is correct?

Yes same error

@<1523701070390366208:profile|CostlyOstrich36> poetry is installed as part of the bash script of the task.

The init script of the AWS autoscaler only contains three export variables I set.

I am literrally trying with 1 package and python and it fails. I tried with python 3.8 3.9 and 3.9.16. and it always fail --> not linked to python version. What is the problem then? I am wondering if there is not an intrinsic bug