Hi @<1601023807399661568:profile|PompousSpider11>
Yes "activating" a conda/python environment in a docker is more complicated then it should be ...
To debug, what are you getting when you do:

docker run -it <docker name here> bash -c "set"

Hi Martin, I'll check today what the output of the command that you sent is. However, when I started the container yesterday with:

docker run -it <docker name> bash

, and then run the following:

which python
which pip

, they both point to the environment set up when the image is built. Also, when starting the Python interpreter inside the container, and running the following

import torch
import torch_scatter
import <other modules>

works with all of the modules that were installed when the image was built. That's the reason we concluded that the correct environment is set in the entrypoint of the container...perhaps we were mistaken, though...

...sorry about the wall of text...

When I start the agent, asking it to use a specific docker image, I get the following messages (from the agent):

Executing Conda: /miniconda/condabin/conda install -p /root/.clearml/venvs-builds/3.10 -c pytorch -c conda-forge -c defaults -c pyg 'pip<20.2 ; python_version < '"'"'3.10'"'"'' 'pip<22.3 ; python_version >= '"'"'3.10'"'"'' --quiet --json
Conda error: DirectoryNotACondaEnvironmentError: The target directory exists, but it is not a conda environment.
Use 'conda create' to convert the directory to a conda environment.
  target directory: /root/.clearml/venvs-builds/3.10

Local file not found [asttokens @ file:///home/conda/feedstock_root/build_artifacts/asttokens_1694046349000/work], references removed
Local file not found [backcall @ file:///home/conda/feedstock_root/build_artifacts/backcall_1592338393461/work], references removed
Local file not found [backports.functools-lru-cache @ file:///home/conda/feedstock_root/build_artifacts/backports.functools_lru_cache_1687772187254/work], references removed

From the above I can see/deduce the following:

• conda is trying to use the environments available in the agent machine that are mounted when the docker is started. Rightfully so, the system is complaining that /root/.clearml/venvs-builds/3.10 is not a conda environment...these are venv:s
• conda is looking for packages in /home/conda/feedstock: file:///home/conda/f which is not available in the docker image

Running

docker run -it <docker name here> bash -c "set"

produces:

unknown shorthand flag: 'i' in -it
See 'docker --help'.

Usage:  docker [OPTIONS] COMMAND

A self-sufficient runtime for containers

Common Commands:
  run         Create and run a new container from an image
  exec        Execute a command in a running container
  ps          List containers
  build       Build an image from a Dockerfile
  pull        Download an image from a registry
  push        Upload an image to a registry
  images      List images
  login       Log in to a registry
  logout      Log out from a registry
  search      Search Docker Hub for images
  version     Show the Docker version information
  info        Display system-wide information

I probably should have mentioned that we create the Docker images in another machine than where it is finally run, but since the Docker image contains everything is needs, this should not matter, right...otherwise you could not use autoscalers...?

If I build a docker image using clearml-agent build as follows:
clearml-agent build --id 75b9e1c8424f458088bcfa93ed37a343 --docker clearml-test --target clearml-test-env
I can see it building the agent, but docker image ls doesn't list the target container clearml-test-env , so I'm a bit lost here...
Output from the command is:

Docker build done
Committing docker container to: /home/jarno/test/docker/clearml-test-env
None

there is no directory/file called: /home/jarno/test/docker/clearml-test-env

In the Docker script we do the following:

# Add Miniconda to PATH
ENV PATH="/miniconda/bin:${PATH}"
.
.
.
# This is to ensure that the Conda environment is activated when you run the Docker container
ENTRYPOINT ["conda", "run", "--no-capture-output", "-n", "deep-learning"]

Our idea was to test a docker image running in an agent and then use the same docker image in an autoscaler setting