Hi! Quick Question. I'M New To Clearml, And Seem To Have Problems Catching Experiments When Done Through A Container (Docker). I Have The Clearml-Server Running And When I Try Kick Off A Simple Script On My Local Machine It Gets Recognized By Clearml And

Answered

Hi! Quick question. I'm new to clearml, and seem to have problems catching experiments when done through a container (Docker). I have the clearml-server running and when I try kick off a simple script on my local machine it gets recognized by clearml and is shown in the dashboard correctly. However, when I kick off the experiment with a docker container it seems to still want to point to the demoapp server that you guys have running.

I kicked off the docker like this:
docker run --runtime=nvidia -v "MY_LOCAL_FOLDER:/app/" simple_cnn:1.0

Also tried this, but to no avail:
clearml-agent daemon --detached --gpus 0 --queue test_queue --docker simple_cnn:1.0
where test_queue is a queue I had previously created of course.

Any idea what I have done wrong?

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					SarcasticSnake58
				
					0
					 × 1

Votes Newest

Answers 3

I think this still has to do with the ports

Can you check that?
Web application on port 8080 API service on port 8008 File storage service on port 8081

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					TimelyPenguin76
				
					0
					 Administrator

Thanks a lot Alon! That was indeed something I forgot. Now at least something else happens 🙂 I do however get this error message:
Retrying (Retry(total=239, connect=239, read=240, redirect=240, status=240)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1b548e6490>: Failed to establish a new connection: [Errno 111] Connection refused')': /auth.login

I think this still has to do with the ports? It seems not be able to authenticate but I just the correct credentials from the server in my conf-file.

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					SarcasticSnake58
				
					0
					 × 1

Hi SarcasticSnake58 ,

The connection to the server is in the api part in your ~/clearml.conf file. I think you don’t have such file in your running container, so you are directed to the demoapp server.

If you are running the container from your local machine, you can start the container with the local ~/clearml.conf , add -v "~/clearml.conf":"/root/clearml.conf" to the docker command. You can also connect the docker and create a new configuration file with clearml-init wizard

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					TimelyPenguin76
				
					0
					 Administrator

Write your answer

1K Views

3 Answers

3 years ago

one year ago