Hey @<1523701205467926528:profile|AgitatedDove14> , thank you for your input
Could you clarify what you mean by clearml-serving session?

Are you refering to the servingTaskId ?

On the helm charts clearml repos, can we use the clearml-serving chart alone ?

@<1523701087100473344:profile|SuccessfulKoala55> Do you think it is possible to ask to run docker mode in the aws autoscaler, and add the cloning and installation inside the init bash script of the task?

@<1523701070390366208:profile|CostlyOstrich36> @<1523701205467926528:profile|AgitatedDove14> Any ideas on this one?

Yes I take the export statements from my bash script of the task

Because I was ssh-ing to it before the fail. When poetry fails, it installs everything using PIP

When the task finally failed, I was kicked of from the container

Yes should be correct. Inside the bash script of the task.

@<1523701070390366208:profile|CostlyOstrich36> @<1523701087100473344:profile|SuccessfulKoala55> I tried with dummy repo. Using Python and stripe packages ONLY in the pyproject.toml

Here is my result (still failing) :

Poetry Enabled: Ignoring requested python packages, using repository poetry lock file!
Creating virtualenv debug in /root/.clearml/venvs-builds/3.9/task_repository/clearmldebug.git/.venv
Using virtualenv: /root/.clearml/venvs-builds/3.9/task_repository/clearmldebug.git/...

I tried too. I do not have more logs inside the ClearML agent 😞

Yes same error

I also did that in the following way:

• I put a sleep inside the bash script
• I ssh-ed to the fresh container and did all commands myself (cloning, installation) and again it worked...

but I still had time to go inside the container, export the PATH variables for my poetry and python versions, and run the poetry install command there

It just allows me to have access to poetry and python installed on hte container

Related to that,
Is it possible to do Dataset.add_external_files() with source_url and destination_url being two separate azure storage containers?

Great, and can we specify an environment variable of ClearML that directly updates the clearml.conf file regarding the azure config or do something similar. I do not want to ask every engineer of my team to modify its clearml.conf file? @<1523701070390366208:profile|CostlyOstrich36> Thanks

I guess it makes no sense because of the steps a clearml-agent works...
I also thought about going to pip mode but not all packages are detected from our poetry.lock file unfortunately so cannot do that.

My issue has been resolved going with pip.

How do you explain that it works when I ssh-ed into the same AWS container instance from the autoscaler?

Task.set_base_docker 🙂

In production, we should use the clearml-helm-charts right? Docker-compose in the clearml-serving is more for local testing

This is really extremely hard to debug. I am thinking to create another repo and iterate on the packages to hopefully find the problem, but it will take ages.

@<1523701070390366208:profile|CostlyOstrich36> poetry is installed as part of the bash script of the task.

The init script of the AWS autoscaler only contains three export variables I set.

Hi @<1523701087100473344:profile|SuccessfulKoala55> , the EC2 instance is spinned-up from the AWS autoscaler provided by ClearML. I use this following docker image: nvidia/cuda:11.8.0-devel-ubuntu20.0

So the EC2 instance runs a docker container

How to set that up inside clearml.conf or something else to know which credentials to load?

The same container

I literrally connected to it at runtime, and ran poetry install -n and it worked

I tried playing with those, but I do not succeed to have a role on the source code detection. I can modify the env variables, nothing happen on CLearML server unfortunately.

Yes indeed, but what about the possibility to do the clone/poetry installation ourself in the init bash script of the task?

If I may ask as well for another issue in that thread that is taking me a big amount of time:

Poetry Enabled: Ignoring requested python packages, using repository poetry lock file!
Creating virtualenv alfred-Rp77Shgw-py3.9 in /root/.cache/pypoetry/virtualenvs
Installing dependencies from lock file
2023-04-17 10:17:57
Package operations: 351 installs, 1 update, 1 removal
failed installing poetry requirements: Command '['poetry', 'install', '-n']' returned non-zero exit status 1.
Ignorin...