Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi, I Followed The Following Instruction 

Hi, I followed the following instruction  https://clear.ml/docs/latest/docs/guides/services/aws_autoscaler#remote-execution
to launch the AWS Auto-scaler. After that I enqueued a task from the UI to my queue  gpu_queue  which was created by the scaler.
I could verify that there's an instance running from AWS console.
But the weird thing is that there's no task running.
And by checking the logs from the Auto-scaler, I found that the task_id is None which is weird  🤔  .

2022-01-06 18:02:20,262 - clearml.auto_scaler - INFO - Found 1 tasks in queue 'gpu_queue' 2022-01-06 18:02:20,262 - clearml.auto_scaler - INFO - Spinning new instance resource='aws4gpu', prefix='dynamic_worker', queue='gpu_queue', task_id=None 2022-01-06 18:02:57 2022-01-06 18:02:54,825 - clearml.auto_scaler - INFO - New instance ID: i-02fc8db63a2069958

I took a look at  clearml/automation/auto_scaler.py  . It seems that the if clause at line 286 ( https://github.com/allegroai/clearml/blob/master/clearml/automation/auto_scaler.py#L286 ) is always False thus we always have task_id = None.

Did you encounter the same problem? Do you have any solution for this problem?
Thanks.

  
  
Posted one year ago
Votes Newest

Answers 5


Hi PunyBee36 ,

Thanks for reporting this, the log message will be fixed in the next clearml version, will update here about it 🙂

About the running task, I can read in the logs that a new instance was created (i-02fc8...), can you check if you have a running clearml agent on it? if so, the agent will pull the task from the queue, if not, can you check in this instance logs for errors and share?

  
  
Posted one year ago

Hi TimelyPenguin76 , I've made a PR to solve this problem https://github.com/allegroai/clearml/pull/534

  
  
Posted one year ago

I meant that it worked with my PR

  
  
Posted one year ago

Hi PunyBee36 , what about the pulling of the task? works?

About the running task, I can read in the logs that a new instance was created (i-02fc8...), can you check if you have a running clearml agent on it? if so, the agent will pull the task from the queue, if not, can you check in this instance logs for errors and share?

  
  
Posted one year ago

Yes it worked

  
  
Posted one year ago
85 Views
5 Answers
one year ago
4 months ago
Tags