Hi @<1523701087100473344:profile|SuccessfulKoala55> , I thought of passing environment parameters directly to the docker, but that approach would be a bit unsafe. Your idea makes definitely more sense...

...sorry about the wall of text...

I probably should have mentioned that we create the Docker images in another machine than where it is finally run, but since the Docker image contains everything is needs, this should not matter, right...otherwise you could not use autoscalers...?

If I build a docker image using clearml-agent build as follows:
clearml-agent build --id 75b9e1c8424f458088bcfa93ed37a343 --docker clearml-test --target clearml-test-env
I can see it building the agent, but docker image ls doesn't list the target container clearml-test-env , so I'm a bit lost here...
Output from the command is:

Docker build done
Committing docker container to: /home/jarno/test/docker/clearml-test-env
None

there is no directory/file called: /home/jarno/te...

In the Docker script we do the following:

# Add Miniconda to PATH
ENV PATH="/miniconda/bin:${PATH}"
.
.
.
# This is to ensure that the Conda environment is activated when you run the Docker container
ENTRYPOINT ["conda", "run", "--no-capture-output", "-n", "deep-learning"]

@<1523701070390366208:profile|CostlyOstrich36> thanks for the hint...I'll give it a go....

Our idea was to test a docker image running in an agent and then use the same docker image in an autoscaler setting

Running

docker run -it <docker name here> bash -c "set"

produces:

unknown shorthand flag: 'i' in -it
See 'docker --help'.

Usage:  docker [OPTIONS] COMMAND

A self-sufficient runtime for containers

Common Commands:
  run         Create and run a new container from an image
  exec        Execute a command in a running container
  ps          List containers
  build       Build an image from a Dockerfile
  pull        Download an image from a registry
  push        Upload an i...

Hi Martin, I'll check today what the output of the command that you sent is. However, when I started the container yesterday with:

docker run -it <docker name> bash

, and then run the following:

which python
which pip

, they both point to the environment set up when the image is built. Also, when starting the Python interpreter inside the container, and running the following

import torch
import torch_scatter
import <other modules>

works with all of the modules...

When I start the agent, asking it to use a specific docker image, I get the following messages (from the agent):

Executing Conda: /miniconda/condabin/conda install -p /root/.clearml/venvs-builds/3.10 -c pytorch -c conda-forge -c defaults -c pyg 'pip<20.2 ; python_version < '"'"'3.10'"'"'' 'pip<22.3 ; python_version >= '"'"'3.10'"'"'' --quiet --json
Conda error: DirectoryNotACondaEnvironmentError: The target directory exists, but it is not a conda environment.
Use 'conda create' to conve...