Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi, I Followed The Following Instruction 

Hi, I followed the following instruction  https://clear.ml/docs/latest/docs/guides/services/aws_autoscaler#remote-execution
to launch the AWS Auto-scaler. After that I enqueued a task from the UI to my queue  gpu_queue  which was created by the scaler.
I could verify that there's an instance running from AWS console.
But the weird thing is that there's no task running.
And by checking the logs from the Auto-scaler, I found that the task_id is None which is weird  🤔  .

2022-01-06 18:02:20,262 - clearml.auto_scaler - INFO - Found 1 tasks in queue 'gpu_queue' 2022-01-06 18:02:20,262 - clearml.auto_scaler - INFO - Spinning new instance resource='aws4gpu', prefix='dynamic_worker', queue='gpu_queue', task_id=None 2022-01-06 18:02:57 2022-01-06 18:02:54,825 - clearml.auto_scaler - INFO - New instance ID: i-02fc8db63a2069958

I took a look at  clearml/automation/auto_scaler.py  . It seems that the if clause at line 286 ( https://github.com/allegroai/clearml/blob/master/clearml/automation/auto_scaler.py#L286 ) is always False thus we always have task_id = None.

Did you encounter the same problem? Do you have any solution for this problem?
Thanks.

  
  
Posted 2 years ago
Votes Newest

Answers 5


Hi PunyBee36 ,

Thanks for reporting this, the log message will be fixed in the next clearml version, will update here about it 🙂

About the running task, I can read in the logs that a new instance was created (i-02fc8...), can you check if you have a running clearml agent on it? if so, the agent will pull the task from the queue, if not, can you check in this instance logs for errors and share?

  
  
Posted 2 years ago

Hi PunyBee36 , what about the pulling of the task? works?

About the running task, I can read in the logs that a new instance was created (i-02fc8...), can you check if you have a running clearml agent on it? if so, the agent will pull the task from the queue, if not, can you check in this instance logs for errors and share?

  
  
Posted 2 years ago

Yes it worked

  
  
Posted 2 years ago

Hi TimelyPenguin76 , I've made a PR to solve this problem https://github.com/allegroai/clearml/pull/534

  
  
Posted 2 years ago

I meant that it worked with my PR

  
  
Posted 2 years ago
507 Views
5 Answers
2 years ago
one year ago
Tags