It seems like the clearml docker image (and the docker-compose.yaml file provided in the allegroai/clearml-server repo) make some assumptions about container names and ports.
I wanted to look through the Dockerfile to see how the clearml-server image is constructed, in order to see how much flexibility there is to set different values through environment variables or config files.
SuccessfulKoala55 Ah, that makes sense. It is possible we won't have any of those sorts of tasks, so it sounds like it may be possible we won't need the agent-services container running.
I'll have to keep this in mind though in case we do end up needing something like this.
Well, you're welcome to ask questions here. I can tell right now that most of what the ClearML server uses (ports, folders etc.) is fully configurable using both config files and environment variables.
A good "service" example is the AWS Auto-scaler found here: https://github.com/allegroai/clearml/tree/master/examples/services/aws-autoscaler
Without the Dockerfile available, I'll have to poke around the the actual clearml Docker image.
SuccessfulKoala55 Thanks for the offer! You're correct, I am interested in more of the actual server configuration, rather than just the Dockerfile .
So I have poked around a little in the Docker image, so I have a very basic idea of how things are working together.
I have two main questions:
In the allegroai/clearml image, it appears that the /etc/nginx/nginx.conf file hard-codes the name of the apiserver container to, well, apiserver . But in my docker-compose.yaml , I'd like to call the service clearml-apiserver . But if I do that, then the webserver container won't start, because it tries to start nginx , and nginx can tell that there is no host called apiserver . I can create a docker-compose alias from my clearml-apiserver container to the apiserver name, but I'd rather just be able to tell the webserver container that it needs to look for clearml-apiserver instead of just apiserver . It appears that the apiserver , webserver , and fileserver containers all run fine without the agent-services container running. However, I guess this agent-services container will be necessary if I want to run my own clearml-agents?
Hi JoyousElephant80 , we haven't published these docker files yet. As mentioned before, our reasons range from "we didn't have time to clean them up and make them presentable" to the fact we're still using some internal forks so we'll be able to produce stable docker images and still be able to push development and PRs on the master branches 🙂
Out of curiosity, are these files important to you? Is there something you're thinking of that we might be able to add or that you might be able to contribute to these files?
SuccessfulKoala55 Yes, I believe adding an option to configure the apiserver service name using an environment variable would be helpful. Thanks.
Currently I had added a network alias, but I'll have to see if extra_hosts is easier to use.
JoyousElephant80 regarding #1, it seems the current scripts used to build this container require some work in orde to be shared. We might add an option to configure apiserver service name using an environment variable in the next release, would that be something of value?
Meanwhile, you can take a look at Docker-compose' extra_hosts feature ( https://docs.docker.com/compose/compose-file/compose-file-v3/#extra_hosts ) which basically allows you to edit the hosts file inside the container itself, so in the webserver service you can simply add a routing from apiserver to clearml-apiserver and use clearml-apiserver for the apiserver service.
BTW - I'm not sure that the kind of information you're after is contained in the Dockerfile anyway, sounds to me more like a complete server configuration specs (which I'll be happy to walk you through 🙂 )
Ah, that's unfortunate.
We have a product that has a small ML component that provides a way for the end-user to train a model on their own data.
We would like to use the ClearML Server to track experiments run by the end user. We'll have to make sure that our product sends results correctly to the ClearML Server we are running locally.
We do development with docker-compose , so we'd like to run the various ClearML Server containers in our own docker-compose setup (instead of using the docker-compose provided in https://github.com/allegroai/clearml-server/blob/master/docker/docker-compose.yml ).
For #2: You're partially right. They all run fine without the agent-services container. This container is also not required for running your own agents. It only functions as a "services" oriented cpu-only agent designed to run maintenance tasks - since these kind of tasks do not require q GPU and take up relatively low CPU, this agent basically takes advantage of the already-running server machine 🙂