Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hello! I'M Running Clearml-Server On Kubernetes, And It Seems My Models Are Not Really Saved. I See That Doing Task.Init(Output_Uri=True) Should Send Models To Fileserver. The Models Are Visible In The Ui But The Download Button Is Greyed Out And When I D

Hello! i'm running clearml-server on kubernetes, and it seems my models are not really saved. i see that doing task.init(output_uri=True) should send models to fileserver. The models are visible in the UI but the download button is greyed out and when i do .get_local_copy() i get None

  
  
Posted 2 years ago
Votes Newest

Answers 31


image

  
  
Posted 2 years ago

this is my cmdline: clearml-task --name hla --requirements requirements.txt --project examples --output-uri http://clearml-fileserver:8081 --queue aws-instances --script keras_tensorboard.py

  
  
Posted 2 years ago

i added --skip-task-init too

  
  
Posted 2 years ago

task = Task.init(project_name='examples', task_name='moemwap', output_uri=True, reuse_last_task_id=False)

  
  
Posted 2 years ago

for comparison: this is when i use --output-uri

  
  
Posted 2 years ago

its as if the line is not there

  
  
Posted 2 years ago

no i don't think so, i think rather Task.init is only used for running outside of agent

  
  
Posted 2 years ago

and if you add --skip-task-init ?
I think what happens is that the clearml-Task, adds a Task.init call without the output_uri that is called before "your" Task.init, and this is what causes it to be ignored. Could that be the case?

  
  
Posted 2 years ago

but i still think the same should be possible using the Task.init

  
  
Posted 2 years ago

it didn''t make a difference

  
  
Posted 2 years ago

don't know.. but i see for instance when using clearml-task i can put any (even nonsensical) values in Task.init

  
  
Posted 2 years ago

and ... clearml-agent takes a --project and a --name argument that are mandatory , so these are never taken from Task.init

  
  
Posted 2 years ago

so it seems that it takes output_uri from the clearml commandline but not from the Task.init inside the scripot

  
  
Posted 2 years ago

(same for environment variable)

  
  
Posted 2 years ago

well it made a difference (the code for the init() is not added anymore) but it still didn't take my output uri

  
  
Posted 2 years ago

AgitatedDove14 your trick seems to work (i had to change the url to reflect the fact i run on k8s)

  
  
Posted 2 years ago

it was to test if reuse_last_task_id made any effect (i have the impression it doesn't)

  
  
Posted 2 years ago

well it doesn't fail. but whatever i set gets ignored

  
  
Posted 2 years ago

this is now in my python script:

  
  
Posted 2 years ago

and also, on the tutorials that do something with task.init, the example always talks about running locally and not in the agent

  
  
Posted 2 years ago

i set reuse_last_task_id to false to force creation of a new task in all cases

  
  
Posted 2 years ago

this is the output of the training. it doens't try to upload (note that this is my second try so it already found a model with that name, but on my first try it didn't work either)

  
  
Posted 2 years ago

AntsyElk37

and when i try to use --output-uri i can't pass true because obviously i can't pass a boolean only strings

hmm, that sounds right, I think we should fix that so when using --output-uri true the value that is passed is actually True, not the string "true".
Regrading the issue itself:
are you saying --skip-task-init is being ignored ? and it always adds the Task.init call? you can also pass --output-uri https://files.clear.ml (which is the same as True) , does that work?

  
  
Posted 2 years ago

...I'm not sure I follow, the clearml-task is designed to always be used so that at the end the agent will be running the Task. What am I missing?

  
  
Posted 2 years ago

but i still think the same should be possible using the Task.init

This is the part the I find confusing:
Task.init(..., output_uri=True) is working for me, what is that setup that caused this line to "fail"?

  
  
Posted 2 years ago

this is the script shown by clearML ui. so the task.init call looks right

  
  
Posted 2 years ago

and when i try to use --output-uri i can't pass true because obviously i can't pass a boolean only strings

  
  
Posted 2 years ago

hello, i'm still not able to save clearml models. They are generated and registered okay, but they are not on the fileserver. i now have Task.init(output_uri=True) and i also have --skip-task-init in clearml commandline so that it doesn't overwrite the task.init call

  
  
Posted 2 years ago

it seems that whatever i pass to Task.init is ignored

  
  
Posted 2 years ago

the model has this information ... the /tmp seem local URIs suggesting that it doesn't even try to upload them

  
  
Posted 2 years ago
20K Views
31 Answers
2 years ago
7 months ago
Tags