Hi, I Followed The Following Instruction

Answered

Hi, I followed the following instruction https://clear.ml/docs/latest/docs/guides/services/aws_autoscaler#remote-execution
to launch the AWS Auto-scaler. After that I enqueued a task from the UI to my queue gpu_queue which was created by the scaler.
I could verify that there's an instance running from AWS console.
But the weird thing is that there's no task running.
And by checking the logs from the Auto-scaler, I found that the task_id is None which is weird 🤔 .

2022-01-06 18:02:20,262 - clearml.auto_scaler - INFO - Found 1 tasks in queue 'gpu_queue' 2022-01-06 18:02:20,262 - clearml.auto_scaler - INFO - Spinning new instance resource='aws4gpu', prefix='dynamic_worker', queue='gpu_queue', task_id=None 2022-01-06 18:02:57 2022-01-06 18:02:54,825 - clearml.auto_scaler - INFO - New instance ID: i-02fc8db63a2069958

I took a look at clearml/automation/auto_scaler.py . It seems that the if clause at line 286 ( https://github.com/allegroai/clearml/blob/master/clearml/automation/auto_scaler.py#L286 ) is always False thus we always have task_id = None.

Did you encounter the same problem? Do you have any solution for this problem?
Thanks.

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					PunyBee36
				
					0
					 × 1

Votes Newest

Answers 5

I meant that it worked with my PR

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					PunyBee36
				
					0
					 × 1

Yes it worked

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					PunyBee36
				
					0
					 × 1

Hi PunyBee36 , what about the pulling of the task? works?

About the running task, I can read in the logs that a new instance was created (i-02fc8...), can you check if you have a running clearml agent on it? if so, the agent will pull the task from the queue, if not, can you check in this instance logs for errors and share?

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					TimelyPenguin76
				
					0
					 Administrator

Hi TimelyPenguin76 , I've made a PR to solve this problem https://github.com/allegroai/clearml/pull/534

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					PunyBee36
				
					0
					 × 1

Hi PunyBee36 ,

Thanks for reporting this, the log message will be fixed in the next clearml version, will update here about it 🙂

About the running task, I can read in the logs that a new instance was created (i-02fc8...), can you check if you have a running clearml agent on it? if so, the agent will pull the task from the queue, if not, can you check in this instance logs for errors and share?

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					TimelyPenguin76
				
					0
					 Administrator

Write your answer

905 Views

5 Answers

2 years ago

one year ago