Hi ResponsiveHedgehong88
With clearml-task the assumption is that you are using argparse. Does that make sense? You can also manually access it with task.get_parameters
https://clear.ml/docs/latest/docs/references/sdk/task#get_parameters

ResponsiveHedgehong88 so I would suggest using execute_remotely in your code, basically you start locally you make sure everything is passed as intended, then from within the code you call task.execute_remotely(...) which will stop the current process and enqueue the Task on the selected queue for the agent to execute.
https://github.com/allegroai/clearml/blob/0397f2b41e41325db2a191070e01b218251bc8b2/examples/advanced/execute_remotely_example.py#L127
This way you can both easily test/debug, and still run it on hardware that actually supports the parameters.
wdyt?
(it will also expose exactly how the artgparse is being stored on the Task, so later you understand how to pass arguments from clearml-task command line)

Not in this setup, no. I don't have the memory or processing resources for these training tasks. If i run it locally for simple tests, i just run my script directly and pass the arguments from the command line, without the clearml agent

ps. i've tried calling the thirdparty_argparser2 with subprocess
subprocess.call(["python3", "thirdparty_argparser2.py", arg1_val])
but somehow (?) it still got the second arg2 and failed due to type error 😅

I see, when you run it manually (i.e. not via an agent) what do you have under the configuration tab in the UI (meaning do you see both argparser arguments there)?

Hi AgitatedDove14 thanks for the suggestion. I might have a go with this, i just need a bit more help to clear things out. When running with clearml-task , i also use --repo and --branch options to setup the code version, and other options (- -docker_args --docker_bash_setup_script --packages --output-uri ) to export some env variables, install some dependencies etc. How can i do this in case of running a script locally and switching to remote? Is this somewhere close to what's the idea behind with execute_remotely?

` task.init_task(project_name,task_name,output_uri=output_uri)

custom_argparse()

setup arg1 and arg2

thirdparty_argparser1()

setup arg1 only for thirdparty_argparser2 later on

task.add_requirements(packages_to_install)

somehow set docker_args and docker_bash_setup_script equivalent??

somehow setup repo and branch to download to remote instance before running

task.execute_remotely(queue_name=queue_name)

custom code...

thirdparty_argparser2() `

my worker is a remote instance with admin access only, so i cannot really run experiments manually there. I can try to setup a minimalistic environment locally and check what happens

(it will also expose exactly how the artgparse is being stored on the Task, so later you understand how to pass arguments from clearml-task command line) (edited)

does this mean that Task stores --args (and propagates these further through the code as CLI arguments) somewhere where i can get and manipulate them from my code? Are they propagated from Configuration ARGS? In this case, can i just update them from my code between different argparse calls (using connect_configuration or sth similara)?

does this mean that Task stores --args (and propagates these further through the code as CLI arguments) somewhere where i can get and manipulate them from my code?

Yes it changes the actual argparse object and pushes the new values in runtime, basically you args.parse() will return the values from the UI (backend)

Ok, getting is easy then with an additional argparser, and what about manipulating? I really hope this is my final question. Can i modify an argument among CONFIGURATION > HYPER PARAMETERS - Args from code?

Why would you need to manually change the current run? you just provided the values with either default/command-line ?
what am I missing here?
ResponsiveHedgehong88 I'm not sure I state dit, but the argparser arguments and values are collected automatically from your current run and put on the Task, there is no need to manually set them if you have the argparser running on your machine. Basically it collects the current (i.e. the process running on your machine) settings, and "copies" them to the remote one
You can think of it as storing the "args" object (i.e. the return value of thirdparty_argparser1 ) on the Task itself.
Does that make sense ?

I'm sorry for bad explanations... My problem is that I need to pass an argument to one function, but then I call a second function that breaks because of this argument. So i'd like to use the command line argument it in the first argparse, and then hide/delete/override before running the second argparse.

. So i'd like to use the command line argument it in the first argparse, and then hide/delete/override before running the second argparse.

Nice, hack!

AgitatedDove14 thanks so much for your help. I managed finally by updating parameter -the argument that was in the way 🙂

Ok thanks. I'm also using --skip-task-init because i'm manually initializing a task and connecting configurations in my_script. I guess then the get_parameters won't give what i'm looking for?
I have a specific situation, maybe you can help me with this
my_script looks something like this
thirdparty_argparser1() [...my_code...] thirdparty_argparser2()I cannot interfere with implementations of these thirdparty argparsers. The first argparser expects arg1=val1 --arg2=val2 , and the second expects arg1=val1 . The problem is that thirdparty_argparser2 also has an optional arg2, but of a different data type. I actually do not need the --arg2 in the thirdparty_argparser2 , but it still fails because it is automagically forwarded from the --args... Do you know how i could intercept this? (without changing the thirdparty_argparser2 implementation)
Thanks

Wait ResponsiveHedgehong88 I'm confused, if you integrated your code with clearml, didn't you to run it manually even once (on any machine, local/remote)?

Uh, why not Current task? Couldn't this do? (seems that wasn't the final question 😞 )
task = Task.init() thirdparty_argparser1() #this one takes both arg1 and arg2 [...my_code...] task.set_parameters({'Args/arg1':<leave_as_is>, 'Args/arg2': None}) thirdparty_argparser2() #this one takes arg1, but not arg2

somehow set docker_args and docker_bash_setup_script equivalent??task.set_base_docker(...)
# somehow setup repo and branch to download to remote instance before runningThis is automatically detected based on your local commit/branch as well ass uncommitted changes

Correct 🙂
I'm assuming the Task object is not your Current task, but a different one?

Would this do?
task.set_parameters({'Args/arg1':..., 'Aegs/arg2': None})