Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi, When Using

Hi, when using clearml-task --script my_script.py --args key1=value1 key2=value2 how can we access the args? This snippet in my_script.py
if __name__ == '__main__': import sys print("these are input args") for arg in sys.argv: print(arg) [...other code including argparser...]Gives me this output
these are input args my_script.py

  
  
Posted one year ago
Votes Newest

Answers 20


Hi ResponsiveHedgehong88
With clearml-task the assumption is that you are using argparse. Does that make sense? You can also manually access it with task.get_parameters
https://clear.ml/docs/latest/docs/references/sdk/task#get_parameters

  
  
Posted one year ago

ResponsiveHedgehong88 so I would suggest using execute_remotely in your code, basically you start locally you make sure everything is passed as intended, then from within the code you call task.execute_remotely(...) which will stop the current process and enqueue the Task on the selected queue for the agent to execute.
https://github.com/allegroai/clearml/blob/0397f2b41e41325db2a191070e01b218251bc8b2/examples/advanced/execute_remotely_example.py#L127
This way you can both easily test/debug, and still run it on hardware that actually supports the parameters.
wdyt?
(it will also expose exactly how the artgparse is being stored on the Task, so later you understand how to pass arguments from clearml-task command line)

  
  
Posted one year ago

Not in this setup, no. I don't have the memory or processing resources for these training tasks. If i run it locally for simple tests, i just run my script directly and pass the arguments from the command line, without the clearml agent

  
  
Posted one year ago

Ok thanks. I'm also using --skip-task-init because i'm manually initializing a task and connecting configurations in my_script. I guess then the get_parameters won't give what i'm looking for?
I have a specific situation, maybe you can help me with this
my_script looks something like this
thirdparty_argparser1() [...my_code...] thirdparty_argparser2()I cannot interfere with implementations of these thirdparty argparsers. The first argparser expects arg1=val1 --arg2=val2 , and the second expects arg1=val1 . The problem is that thirdparty_argparser2 also has an optional arg2, but of a different data type. I actually do not need the --arg2 in the thirdparty_argparser2 , but it still fails because it is automagically forwarded from the --args... Do you know how i could intercept this? (without changing the thirdparty_argparser2 implementation)
Thanks

  
  
Posted one year ago

ps. i've tried calling the thirdparty_argparser2 with subprocess
subprocess.call(["python3", "thirdparty_argparser2.py", arg1_val])
but somehow (?) it still got the second arg2 and failed due to type error 😅

  
  
Posted one year ago

I see, when you run it manually (i.e. not via an agent) what do you have under the configuration tab in the UI (meaning do you see both argparser arguments there)?

  
  
Posted one year ago

my worker is a remote instance with admin access only, so i cannot really run experiments manually there. I can try to setup a minimalistic environment locally and check what happens

  
  
Posted one year ago

Wait ResponsiveHedgehong88 I'm confused, if you integrated your code with clearml, didn't you to run it manually even once (on any machine, local/remote)?

  
  
Posted one year ago

(it will also expose exactly how the artgparse is being stored on the Task, so later you understand how to pass arguments from clearml-task command line) (edited)

does this mean that Task stores --args (and propagates these further through the code as CLI arguments) somewhere where i can get and manipulate them from my code? Are they propagated from Configuration ARGS? In this case, can i just update them from my code between different argparse calls (using connect_configuration or sth similara)?

  
  
Posted one year ago

does this mean that Task stores --args (and propagates these further through the code as CLI arguments) somewhere where i can get and manipulate them from my code?

Yes it changes the actual argparse object and pushes the new values in runtime, basically you args.parse() will return the values from the UI (backend)

  
  
Posted one year ago

somehow set docker_args and docker_bash_setup_script equivalent??task.set_base_docker(...)
# somehow setup repo and branch to download to remote instance before runningThis is automatically detected based on your local commit/branch as well ass uncommitted changes

  
  
Posted one year ago

Uh, why not Current task? Couldn't this do? (seems that wasn't the final question 😞 )
task = Task.init() thirdparty_argparser1() #this one takes both arg1 and arg2 [...my_code...] task.set_parameters({'Args/arg1':<leave_as_is>, 'Args/arg2': None}) thirdparty_argparser2() #this one takes arg1, but not arg2

  
  
Posted one year ago

Ok, getting is easy then with an additional argparser, and what about manipulating? I really hope this is my final question. Can i modify an argument among CONFIGURATION > HYPER PARAMETERS - Args from code?

  
  
Posted one year ago

Correct 🙂
I'm assuming the Task object is not your Current task, but a different one?

  
  
Posted one year ago

Why would you need to manually change the current run? you just provided the values with either default/command-line ?
what am I missing here?
ResponsiveHedgehong88 I'm not sure I state dit, but the argparser arguments and values are collected automatically from your current run and put on the Task, there is no need to manually set them if you have the argparser running on your machine. Basically it collects the current (i.e. the process running on your machine) settings, and "copies" them to the remote one
You can think of it as storing the "args" object (i.e. the return value of thirdparty_argparser1 ) on the Task itself.
Does that make sense ?

  
  
Posted one year ago

Would this do?
task.set_parameters({'Args/arg1':..., 'Aegs/arg2': None})

  
  
Posted one year ago

I'm sorry for bad explanations... My problem is that I need to pass an argument to one function, but then I call a second function that breaks because of this argument. So i'd like to use the command line argument it in the first argparse, and then hide/delete/override before running the second argparse.

  
  
Posted one year ago

AgitatedDove14 thanks so much for your help. I managed finally by updating parameter -the argument that was in the way 🙂

  
  
Posted one year ago

. So i'd like to use the command line argument it in the first argparse, and then hide/delete/override before running the second argparse.

Nice, hack!

  
  
Posted one year ago

Hi AgitatedDove14 thanks for the suggestion. I might have a go with this, i just need a bit more help to clear things out. When running with clearml-task , i also use --repo and --branch options to setup the code version, and other options (- -docker_args --docker_bash_setup_script --packages --output-uri ) to export some env variables, install some dependencies etc. How can i do this in case of running a script locally and switching to remote? Is this somewhere close to what's the idea behind with execute_remotely?

` task.init_task(project_name,task_name,output_uri=output_uri)

custom_argparse()

setup arg1 and arg2

thirdparty_argparser1()

setup arg1 only for thirdparty_argparser2 later on

task.add_requirements(packages_to_install)

somehow set docker_args and docker_bash_setup_script equivalent??

somehow setup repo and branch to download to remote instance before running

task.execute_remotely(queue_name=queue_name)

custom code...

thirdparty_argparser2() `

  
  
Posted one year ago
508 Views
20 Answers
one year ago
one year ago
Tags