Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hello Channel, I Am Struggling A Lot On An Issue Linked To

Hello channel,

I am struggling a lot on an issue linked to ClearMl agent and AWS Autoscaler .
This issue is very problematic and urgent, please help me out! We need the autoscaler to work to be able to pull and execute our tasks correctly.

Context:
I want to install a poetry environment from a poetry.lock file located at the root of my repository.
I have a bash script running allowing to install poetry and install python 3.9.16 using Pyenv (needed by my poetry env).
The installation goes on, my repository is cloned, poetry is detected and then poetry installation fails directly without any logs (even when using the verbose mode).

Here is the clearml.conf of my AWS autoscaler config and the logs of one of the latest runs (you can find the bash script inside it) + snippet of the error :

Poetry Enabled: Ignoring requested python packages, using repository poetry lock file!
Creating virtualenv alfred in /root/.clearml/venvs-builds/3.9/task_repository/alfred.git/.venv
Using virtualenv: /root/.clearml/venvs-builds/3.9/task_repository/alfred.git/.venv
Installing dependencies from lock file
Finding the necessary packages for the current system
Package operations: 351 installs, 1 update, 1 removal, 22 skipped
failed installing poetry requirements: Command '['poetry', 'install', '-n', '-v']' returned non-zero exit status 1.
Ignoring pip: markers 'python_version >= "3.10"' don't match your environment 

agent {
    vcs_cache.enabled: false

    package_manager: {
          type: poetry,
          poetry_version: "1.4.2",
          poetry_install_extra_args: ["-v"],
     }  
}

sdk {
    development {
         store_code_diff_from_remote: false,
    }

}

Issue and what has been done:

  • I tested all clearml.co nf parameters without success
  • I created an AWS instance, ssh-ed into it and try the installation of poetry and it worked...
  • Last but not least which leads to my full confusion : I setted up SSH access to the AWS autoscaler instances, SSHed to it during runtime, went into the created docker container, waited until the git clone command made effect and poetry installation failed to be able to try myself the poetry install -n -v command. Again, It worked... I am extremely confused because it works on the instance itself created by the autoscaler, but it does not work when running on ClearML. What am I missing??
    Thank you a lot for your time,
    CC @<1523701070390366208:profile|CostlyOstrich36> @<1523701205467926528:profile|AgitatedDove14> @<1523701087100473344:profile|SuccessfulKoala55>
  
  
Posted 2 years ago
pic
More
SuccessfulRaven86
0 × 1
Report
Votes Newest

Answers 40

@<1556812486840160256:profile|SuccessfulRaven86> , to make things easier to debug, can you try running the agent locally?

  
  
Posted 2 years ago
pic
More
CostlyOstrich36
0
Report

I literrally connected to it at runtime, and ran poetry install -n and it worked

  
  
Posted 2 years ago
pic
More
SuccessfulRaven86
0 × 1
Report

@<1556812486840160256:profile|SuccessfulRaven86> can you try with -vvv instead of -v ?

  
  
Posted 2 years ago
pic
More
SuccessfulKoala55
0 × 1
Report

You can theoretically do that in the docker init bash script that will be executed before the task is cloned and run

  
  
Posted 2 years ago
pic
More
SuccessfulKoala55
0 × 1
Report

Yes same error

  
  
Posted 2 years ago
pic
More
SuccessfulRaven86
0 × 1
Report

Yes should be correct. Inside the bash script of the task.

  
  
Posted 2 years ago
pic
More
SuccessfulRaven86
0 × 1
Report

I think you should try to manually start such a docker container and try to see what fails in the process. Attaching to an existing one has too many differences already

  
  
Posted 2 years ago
pic
More
SuccessfulKoala55
0 × 1
Report

or just to the AWS instance?

  
  
Posted 2 years ago
pic
More
SuccessfulKoala55
0 × 1
Report

and are you sure these are the same env vars available when the agent does the same?

  
  
Posted 2 years ago
pic
More
SuccessfulKoala55
0 × 1
Report

where did you specify it?

  
  
Posted 2 years ago
pic
More
SuccessfulKoala55
0 × 1
Report

I also did that in the following way:

  • I put a sleep inside the bash script
  • I ssh-ed to the fresh container and did all commands myself (cloning, installation) and again it worked...
  
  
Posted 2 years ago
pic
More
SuccessfulRaven86
0 × 1
Report

I guess it makes no sense because of the steps a clearml-agent works...
I also thought about going to pip mode but not all packages are detected from our poetry.lock file unfortunately so cannot do that.

  
  
Posted 2 years ago
pic
More
SuccessfulRaven86
0 × 1
Report

Yes I take the export statements from my bash script of the task

  
  
Posted 2 years ago
pic
More
SuccessfulRaven86
0 × 1
Report

into the same docker container running the task?

  
  
Posted 2 years ago
pic
More
SuccessfulKoala55
0 × 1
Report

I am literrally trying with 1 package and python and it fails. I tried with python 3.8 3.9 and 3.9.16. and it always fail --> not linked to python version. What is the problem then? I am wondering if there is not an intrinsic bug

  
  
Posted 2 years ago
pic
More
SuccessfulRaven86
0 × 1
Report

Is it a bug inside the AWS autoscaler??

  
  
Posted 2 years ago
pic
More
SuccessfulRaven86
0 × 1
Report

@<1523701087100473344:profile|SuccessfulKoala55> Do you think it is possible to ask to run docker mode in the aws autoscaler, and add the cloning and installation inside the init bash script of the task?

  
  
Posted 2 years ago
pic
More
SuccessfulRaven86
0 × 1
Report

the autoscaler always uses docker mode

  
  
Posted 2 years ago
pic
More
SuccessfulKoala55
0 × 1
Report

The autoscaler just runs it on an AWS instance, inside a docker container - there's no difference from running it yourself inside a docker container - did you try running it inside a docker container as well?

  
  
Posted 2 years ago
pic
More
SuccessfulKoala55
0 × 1
Report

Using a pyenv virtual env then exporting LOCALPYTHON env var

  
  
Posted 2 years ago
pic
More
SuccessfulRaven86
0 × 1
Report

I tried too. I do not have more logs inside the ClearML agent 😞

  
  
Posted 2 years ago
pic
More
SuccessfulRaven86
0 × 1
Report

How is it still up is the task failed?

  
  
Posted 2 years ago
pic
More
SuccessfulKoala55
0 × 1
Report

I am currently trying with a new dummy repo and I iterate over the dependencies of the pyproject.toml.

  
  
Posted 2 years ago
pic
More
SuccessfulRaven86
0 × 1
Report

My issue has been resolved going with pip.

  
  
Posted 2 years ago
pic
More
SuccessfulRaven86
0 × 1
Report

So the same?

  
  
Posted 2 years ago
pic
More
SuccessfulKoala55
0 × 1
Report

Yes indeed, but what about the possibility to do the clone/poetry installation ourself in the init bash script of the task?

  
  
Posted 2 years ago
pic
More
SuccessfulRaven86
0 × 1
Report

It just allows me to have access to poetry and python installed on hte container

  
  
Posted 2 years ago
pic
More
SuccessfulRaven86
0 × 1
Report

Because I was ssh-ing to it before the fail. When poetry fails, it installs everything using PIP

  
  
Posted 2 years ago
pic
More
SuccessfulRaven86
0 × 1
Report

The same container

  
  
Posted 2 years ago
pic
More
SuccessfulRaven86
0 × 1
Report

Yes, the problem is it's still really hidden (the error, I mean)

  
  
Posted 2 years ago
pic
More
SuccessfulKoala55
0 × 1
Report
Show more results replies
Write your answer
173K Views
40 Answers
2 years ago
2 years ago
Tags
mlops
Similar posts