Hi! I'M With That F***Ed Up Migration To A Newer Clearml Server Version Again. We Did Everything Possible To Carefully Move All The Data, But I Still Cannot Access Some Of The Old Experiments On The New Server, Neither From Ui Nor From The Python Client

Answered

Hi! I'm with that f***ed up migration to a newer ClearML server version again. We did everything possible to carefully move all the data, but I still cannot access some of the old experiments on the new server, neither from UI nor from the python client

When I call task.get_task(project_name=..., task_name=...) it returns
ValueError: Task ID "c7fe399d687a45f1a27d3b30790384dd" could not be found

This is correct ID of the task I'm trying to fetch though, and when I call the old server it responds properly and gets me the task config

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					MelancholyElk85
				
					0
					 × 1

Votes Newest

Answers 5

Hi MelancholyElk85 ,
If the server can't find the task, it simply means either the task is not there or was moved there incorrectly - did you merge the data, or simply copied it?

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

You can actually check using mongodb CLI if the task is there

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

The SDK error you shared seems to indicate the task cannot be found in the database...

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					SuccessfulKoala55
				
					0
					 × 1

I suspect this has something to do with task.output_model

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					MelancholyElk85
				
					0
					 × 1

I see the task on the web UI, but get Fetch experiment failed when I click on it, as I described. It even fetches the correct ID by it's name. I'm almost sure it will be present in mongodb

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					MelancholyElk85
				
					0
					 × 1

Write your answer

586 Views

5 Answers

2 years ago

one year ago