Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi All, I Am Trying To Debug A Strange Issue With The Most Minimal Version Of A

Hi all, I am trying to debug a strange issue with the most minimal version of a http://clear.ml session (the simple "hello world" Task instance), within a rather involved training script. For some reason early on in the run the script crashes with the following two lines:
2021-07-11 14:34:45,055 - clearml.Task - WARNING - ### TASK STOPPED - USER ABORTED - STATUS CHANGED ### KilledNo other information is given beyond that, so it's hard to figure out what's triggering this failure. The training script doesn't crash without the ClearML code -- only when I create the Task object. ClearML is running locally using the Docker image. Does anyone have any idea how to figure out where this crash is coming from?

  
  
Posted 2 years ago
Votes Newest

Answers 9


SuccessfulKoala55 Locally, I think. Also, I forgot to add that the training script is run on the same machine that's running the ClearML server.

  
  
Posted 2 years ago

Hi, SmallDeer34 Unfortunately no. The code stack is pretty substantial, and I'm not sure what portion of it would be useful.

  
  
Posted 2 years ago

ZealousSeal58 are you running locally or using a ClearML agent?

  
  
Posted 2 years ago

Yes, my apologies. I should provide you a more comprehensive log. I am working on pinpoint the line that triggers this message. It behaves almost like a segfault, but I'll do my best to locate the error prone line. Sorry for the incomplete picture -- I'll post an update today.

  
  
Posted 2 years ago

Hi ZealousSeal58
What's the clearml version you are using ?

If there was a "debug mode" for viewing the stack trace before the crash that would've been most helpful...

import traceback traceback.print_stack()

  
  
Posted 2 years ago

And when exactly are you getting the "user aborted" message)?
How do you start the process (are you manually running it, or is it an agent, or maybe pycharm?)
Can you provide the full log ?

  
  
Posted 2 years ago

If there was a "debug mode" for viewing the stack trace before the crash that would've been most helpful...

  
  
Posted 2 years ago

AgitatedDove14 clearml==1.0.4

  
  
Posted 2 years ago

Can you share the code?

  
  
Posted 2 years ago