Hi All, I Created A Pipeline That Creates A Task With A Script That Runs Some Hugingface Code And Then Runs A Slurm Batch That Executes "Clearml-Agent Exe" To Run The Task. But When The Script Runs, It Gives The Following Error: Valueerror: Task Object

Answered

Hi All,

I created a pipeline that creates a task with a script that runs some HugingFace code and then runs a slurm batch that executes "clearml-agent exe" to run the task. But when the script runs, it gives the following Error:

ValueError: Task object can only be updated if created or in_progress [status=completed fields=['hyperparams']]

The error raises when I execute trainer.train()

Please see the pipline and task code, and the entire output log attached

Would appreciate your help,

Oren

  				
Posted 
	one year ago

					More
				  		
  Report
		
					DelightfulArcticwolf22
				
					0
					 × 1

Votes Newest

Answers 5

Hi @<1577468638728818688:profile|DelightfulArcticwolf22> , it looks like you're trying to update some parameter of a task after it finished running...

  				
Posted 
	one year ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

Hi John, but the task is running the script code, so how can that be?

  				
Posted 
	one year ago

					More
				  		
  Report
		
					DelightfulArcticwolf22
				
					0
					 × 1

I would suggest adding print outs during the code to better understand when this happens

  				
Posted 
	one year ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

OK I will try that.

  				
Posted 
	one year ago

					More
				  		
  Report
		
					DelightfulArcticwolf22
				
					0
					 × 1

Hi John. It turns out that the task that ended was the pipeline step. Apparently since I create the task that I then run with clearml-agent execute, in the step, it has to remain running. I added task.wait_for_status() at the end of the step and now my pipeline works.

I have another question: Do you know if it is possible to use PipelineDecorator.component as a class decorator?

  				
Posted 
	one year ago

					More
				  		
  Report
		
					DelightfulArcticwolf22
				
					0
					 × 1

Write your answer

967 Views

5 Answers

one year ago