ClearML FAQ | Hi, We Are Having Some Issues With Model Snapshots Uploading To The Fileserver. We Configured Sdk.Development.Default_Output_Uri To Point To Our File Server, And When We Run Some Experiment We Can See Under The Models Tab Some Url Pointing To

Answered

Hi, We Are Having Some Issues With Model Snapshots Uploading To The Fileserver. We Configured Sdk.Development.Default_Output_Uri To Point To Our File Server, And When We Run Some Experiment We Can See Under The Models Tab Some Url Pointing To

Hi, we are having some issues with model snapshots uploading to the fileserver. We configured sdk.development.default_output_uri to point to our file server, and when we run some experiment we can see under the models tab some url pointing to http://192.168.1.240:8081/uploading_file but the file is not accessible and its not clear if it was successfully stored. any clues?

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					RipeGoose2
				
					0
					 × 1

Votes Newest

Answers 6

AgitatedDove14 it is happening on an offline network, would be tricky to set it up we will try. so far the errors we observed were either:
Calling upload callback when starting upload: maximum recursion depth exceeded
Or
something like pending for upload (might be because we archived a run while it was uploading)

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					RipeGoose2
				
					0
					 × 1

RipeGoose2 That sounds familiar. Could you test with the latest RC?
pip install trains==0.16.4rc0

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

thanks!

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					RipeGoose2
				
					0
					 × 1

it worked!

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					RipeGoose2
				
					0
					 × 1

Hi RipeGoose2
Any logs on the console ?
Could you test with a dummy example on the demoserver ?

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

YEY 🙂

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

Write your answer

1K Views

6 Answers

3 years ago

one year ago

Tags

Similar posts

Hi All, We Have Clearml-Server Running On A Kube Pod, And Then A Gpu Server Running The Clearml-Agent Which We Use To Queue Jobs. For Some Reason, Our Kube Pod Restarted (We'Re Looking Into Why), But In The Process Of This Happening All Jobs On The Worke

Hi There, Our. Self-Hosted Server Is Periodically Very Slow To React In The Web Ui. We'Ve Been Debugging For Quite Some Time, And It Would Seem That Elastisearch Might Be The Culprit. Looking At The Elastisearch Index, We Have An Index Of Around 80G Of Tr

Hi, I Am Fairly New To Clearml And Would Appreciate Some Help With This Problem. My Team Currently Uses The Virtual Environment Mode To Run The Experiments. But We Also Have A Docker File That We Use For Local Debug So I Want To Consolidate Everything And

Could I Get Some Feedback From People With Experience Using Clearml Pipelines On The Best Way To Handle Caching? My Team Is Working On Configuring Clearml Pipelines For Our Data Processing Workflow. We Currently Have An Experimental Pipeline Configured F

Hey Is There A Way For One To Extend Clearml Somehow? We Have Some Custom Evaluations We Want To Do And Our Ideal Scenario Would Be To Do Them Within Clearml Itself.

Hello Everyone , I Am New Bee To Clearml And Finding Option To Accommodate Opensearch Since We Have Already Opensearch Running In Our Env, Is Opensearch Supported In Clearml Instead Of Elasticsearch ? Please Shed Some Light On That

Hey! We'Ve Set An Hyperparameter Optimization Task On One Of Our Experiments, And It Seems As Though The Optimization Itself Is Running - Yet We Don'T See The Iterations Table In The 'Plots' Section In The Ui. Providing Some Screenshots To How We'Ve Set I

We Have An Nginx Container In Front Of Clearml, Functions Use Sort Of " Ssl Reverse Proxy" We Are Seeing This Errors With Jobs That Are Trying To Upload Some Artifacts To Fileserver

Hi Guys, We Are Running Clearml-Serving On A Kube Cluster On Aws And We Have Noticed That We Are Getting Some 502 Errors Once In A While That We Can'T Seem To Trace Back.

Hi We Want To Reuse The Configuration When Having Cloned A Task. For Some Reason It Does Not Return It But I Get {} For This Former_Config: Self.Current_Task = Task.Init(); Former_Config = Self.Current_Task.Connect({}) Can You Help Us?