CostlyOstrich36
hello, I tried some stuff, namely creating the pipeline with hydra with functions, based on https://github.com/allegroai/clearml/blob/master/examples/pipeline/pipeline_from_functions.py .
Before the task starts remotely, I can access the hydra config no problem, but once in remote and the task starts, I got :Starting Task Execution: Primary config directory not found. Check that the config directory '/root/.clearml/venvs-builds/3.8/task_repository/clearml-example.git/src/config' exists and readable Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.This is my function for the pipeline:
` @hydra.main(config_path="config", config_name="config")
def executing_pipeline(cfg):
model_settings = cfg.model_settings
dataset_settings = cfg.dataset_settings
print(model_settings, dataset_settings)
# create the pipeline controller
pipe = PipelineController(
project='clearml-example2',
name='pipeline demo',
version='1.1',
add_pipeline_tags=False,
)
# set the default execution queue to be used (per step we can override the execution)
pipe.set_default_execution_queue('default')
# Use the pipeline argument to start the pipeline and pass it ot the first step
print('launch step 1')
pipe.add_parameter(
name='model_settings',
description='model_settings',
default=model_settings
)
pipe.add_parameter(
name='dataset_settings',
description='dataset_settings',
default=dataset_settings
)
pipe.add_function_step(
name='basic_training',
function=basic_training,
function_kwargs=dict(dataset_settings='${pipeline.dataset_settings}', model_settings='${pipeline.model_settings}'),
function_return=['task_id'],
cache_executed_step=True,
execution_queue="default",
)
pipe.add_function_step(
name='HPO',
# parents=['step_one'], # the pipeline will automatically detect the dependencies based on the kwargs inputs
function=run_hpo,
function_kwargs=dict(base_task_id='${basic_training.task_id}'),
function_return=['best_model_settings'],
cache_executed_step=True,
execution_queue="default",
)
pipe.add_function_step(
name='train_best_model',
function=train_best_model,
function_kwargs=dict(data='${HPO.best_model_settings}'),
function_return=['model', 'config_pbtxt'],
cache_executed_step=True,
execution_queue="default",
)
# For debugging purposes run on the pipeline on current machine
# Use run_pipeline_steps_locally=True to further execute the pipeline component Tasks as subprocesses.
# pipe.start_locally(run_pipeline_steps_locally=False)
# Start the pipeline on the services queue (remote machine, default on the clearml-server)
pipe.start()
print('pipeline completed') `
YummyLion54 , Hi
What versions of clearml/clearml-agent are you using?
When running it without the agent, do the hydra configurations show up properly in the UI?
Hello, I am using clearml 1.1.4 and clearml-agent 1.1.0. I used hydra before without pipelines locally and it worked well and showed up in the UI properly yes
Could you guys have a sample for pipelines using hydra? I think quite a few people would interested in that, and it isn't as straightforward as I thought it would be
