ClearML FAQ | Hi All - I Have A Large Dataset And Have Preprocessed It And Saved Each Item In .Pt Files, Which Are Loaded Using

Answered

Hi All - I Have A Large Dataset And Have Preprocessed It And Saved Each Item In .Pt Files, Which Are Loaded Using

Hi all - I have a large dataset and have preprocessed it and saved each item in .pt files, which are loaded using torch.load in my Dataset . The issue is that ClearML thinks these are input models and tries to log them. I think this is making my training slow due to a whole bunch on unnecessary logging.

It seems like it's possible to turn off this behavior in Task.init() : None

However, I'm adding my tasks to a queue using the CLI, and I'm not seeing an option to disable this: None

Is there a way to disable this in the CLI?

  				
Posted 
	11 months ago

					More
				  		
  Report
		
					DepravedBee82
				
					0
					 × 1

Votes Newest

Answers 3

Also is there a way to disable this by default?

The reason I ask is that I want to send many jobs to a queue via the CLI. so I don't really want to be messing around with Task.init() .

I've even tried renaming my files to *pth and *.data to stop this behaviour

  				
Posted 
	11 months ago

					More
				  		
  Report
		
					DepravedBee82
				
					0
					 × 1

Hi @<1724960464275771392:profile|DepravedBee82> , you have the auto_connect_frameworks parameter in Task.init() , this way you can disable the auto connection to Pytorch
None

  				
Posted 
	11 months ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

Thanks John, but is there a way to do this via the CLI?

Or is Task.init() the only way?

  				
Posted 
	11 months ago

					More
				  		
  Report
		
					DepravedBee82
				
					0
					 × 1

Write your answer

843 Views

3 Answers

11 months ago