Hello, I Am Running Into An Issue With Clearml Pipelines. I Have A Training Script That I Broke Up Into Prepare_Data, Train And Evaluate. I Am Using

Answered

Hello,

I am running into an issue with ClearML pipelines.
I have a training script that i broke up into prepare_data, train and evaluate. I am using PipelineDecorator.component() for each of the steps.
I am running into an issue at the train_model step. I am trying to train a deep learning model using pytorch lightning. My decorator function is PipelineDecorator.component(task.type=TaskTypes.training) but when I look at the artifacts, ClearML is storing the checkpoints for each epoch. I do not want this. I simply want clearml to log the best model at the end of training. How do I do this in a pipeline. Thank you very much

  				
Posted 
	14 days ago

					More
				  		
  Report
		
					MassiveGoldfish6
				
					0
					 × 1

Votes Newest

Answers 2

Thank you! It solved my problem but I'm now seeing something else.
I have a data_prepping step which contains a LightningDataModule. In it, I load the data and prep it. My function then returns an initialized datamodule which i give to the training function. I have PipelineDecorator.component(task_type = TaskTypes.data_processing,cache= False) . When I am done training, the pipeline saves my entire dataset(64GB) as an artifact and I am not sure why. Would you happen to know what I am doing wrong? Would you have a example of how the pipeline decorator is used with a Pytorch Lightning ML pipeline?

  				
Posted 
	13 days ago

					More
				  		
  Report
		
					MassiveGoldfish6
				
					0
					 × 1

Hi @<1547028031053238272:profile|MassiveGoldfish6> , you have the auto_connect_frameworks parameter of the component to disable auto logging of checkpoints - None

  				
Posted 
	14 days ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

Write your answer

133 Views

2 Answers

14 days ago

13 days ago