Hi ShallowGoldfish8 , can you elaborate please? You mean train with different data?
yes, variations of the data, using only a subset of the features
yes, but is there a way to generate multiple tasks like I mentioned using task.init in different points of a .py and and run each of them as a separate remote exercution? Didn you just say that once I trigger the task.execute_remotely it will ignore the task.init?
regarding (2), if use run_remote, does it also ignore the init?
Considering something along the lines of
https://github.com/allegroai/clearml/blob/master/examples/advanced/execute_remotely_example.py
ShallowGoldfish8 this call does that:
https://github.com/allegroai/clearml/blob/0397f2b41e41325db2a191070e01b218251bc8b2/examples/advanced/execute_remotely_example.py#L127
Could I just build it and log these parameters using
task.set_parameters()
so that I call
task.get_parameters()
later?
instead of manually calling set/get, you call task.connect(some_dict_or_object) , it does both:
When running manually (i.e. without an agent) it logs the keys/values on the Task,
when running with an agents, it takes the values from the backend (Task) and sets them on the dict/object
Make sense ?
Looks quite good indeed! Thanks! Is there in the repository the experiment template used in this example? Just not fully sure how the parameters are used/connected in it. Could I just build it and log these parameters using task.set_parameters() so that I call task.get_parameters() later?
Oh think I understand you point now.
basically you can:
Create the initial Task, once it is in the system clone it and adjust parameters externally. A simple example here:
https://github.com/allegroai/clearml/blob/0397f2b41e41325db2a191070e01b218251bc8b2/examples/automation/manual_random_param_search_example.py#L41
wdyt?
I was checking here, and apparently if I use a parameter as suggested, together with a Task.init(task_name=f'{task name in this loop}') for each of the loops it should work, right? Creating different tasks in the server
Is there a way to do that to trigger separate remote executions?
using only a subset of the features
ShallowGoldfish8 if you have some parameter that controls it (i.e. select different features) then you can launch it with two sets f parameters.
Am I missing something?
for example:
` my_features_select = {"type": "set_a"}
Task.current_task().connect(my_features_select)
if my_features_select["type"] == "set_a":
do something
else
do something else `wdyt?
So yes, it creates the Task on your machine (with the name and project etc.)
Then it stops the local process, and pushes it into the exxecution queue, when the agent pulls it, and re-executes the code, it will ignore the Task.init
Correct, but do notice that (1) task names are not unique and you can change them after the Task was executed (2) when you clone the Task, you can actually rename it, when an agent is running the Task, basically the init function is ignored, because the Task already exists. Make sense ?