Hi Everyone! After Clearml Worked Well For Use For About 2 Years, We Now Have The First Problem When Updating The Open Source Server. I Have Performed The Update As Advised Here With

Answered

Hi Everyone! After ClearML worked well for use for about 2 years, we now have the first problem when updating the open source server.

I have performed the update as advised here with

cd /opt/clearml/

# Stop old server and move yml
docker-compose -f docker-compose.yml down
mv docker-compose.yml docker-compose-old.yml

# Pull new yml
curl

 -o /opt/clearml/docker-compose.yml

docker-compose -f docker-compose.yml pull
docker-compose -f docker-compose.yml up -d

However, the service does not start again, not even after several minutes.
The logs indicate something about the api server not starting right or not being found, I have attached a full log.
A main indicator seems to be a problem with the API server:

[36;1mclearml-agent-services |[0m curl: (7) Failed to connect to apiserver port 8008 after 10 ms: Couldn't connect to server

Could you help us resolve this? Already tried a language model, no luck I am afraid.
What is more, when just going back to the old .yml, the service also does not start in the old variant, preventing us from checking or starting model trainings.

  				
Posted 
	one month ago

					More
				  		
  Report
		
					ScantChimpanzee51
				
					0
					 × 1

Votes Newest

Answers 7

New state: After starting with the old YML again, the web app looks new (presumably because the image allegroai/clearml:latest is used), but the server version still lists WebApp: 2.1.0-664 • Server: 1.9.2-317 • API: 2.23 .

Creating tasks and reporting things works again, but I regularly see the UI error shown attached. Any way to resolve things?

  				
Posted 
	one month ago

					More
				  		
  Report
		
					ScantChimpanzee51
				
					0
					 × 1

Also as an update, I tried to start the containers one by one and resolve the errors that came up. The only real one I found was that Redis crashed when loading the previous data base. Since I figured I wouldn't necessarily need the cache, I cleared the dump file, then all the services started - this all refers to the old server version.

When trying the same with the most recent docker-compose.yml , the services all started by themselves, but when I start the full docker compose, the dashboard shows up but none of the projects load and eventually our VM instance crashes all together.

Any idea on how to proceed?

  				
Posted 
	one month ago

					More
				  		
  Report
		
					ScantChimpanzee51
				
					0
					 × 1

Is there for example a good way to start the new version without any legacy data base and then migrate the data more or less manually? Is it enough to migrate the Mongo DB to see our old tasks again and are the schematics still compatible?

  				
Posted 
	one month ago

					More
				  		
  Report
		
					ScantChimpanzee51
				
					0
					 × 1

Hi @<1523702496097210368:profile|ScantChimpanzee51> , according to the clearml logs the initial problem was due to the MongoDB not able to start with your data. The reason is that MongoDB cannot be jumped more than one version at a time. Your previous version used MongoDB 4.4 and the latest version uses MongoDB 6.0. In order to upgrade from you initial version to the latest one you should first upgrade to clearml version 1.17 that upgrades MongoDB from v4.4 to 5.0
Then you can upgrade to clearml version 2.1 which is the latest version right now and it will upgrade MongoDB from v5 to v6
None

  				
Posted 
	one month ago

					More
				  		
  Report
		
					ResponsiveKoala38
				
					0

Hi @<1523701070390366208:profile|CostlyOstrich36> , thank you for answering!
We are upgrading from v. 1.9 or so (I think) to the most recent one.

Attached below are 3 logs from api server, elastic search and file server - does this help to debug?

  				
Posted 
	one month ago

					More
				  		
  Report
		
					ScantChimpanzee51
				
					0
					 × 1

Hi @<1523702496097210368:profile|ScantChimpanzee51> , from what version to what version did you try upgrading? Did you perform a backup?

  				
Posted 
	one month ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

Also can you please provide a log of only the apiserver and also one from the elastic container

  				
Posted 
	one month ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

Write your answer

236 Views

7 Answers

one month ago