I think you should use get_tasks for that ( https://clear.ml/docs/latest/docs/clearml_sdk/task_sdk#querying--searching-tasks )
why aren't you coping the Pipeline ID from the UI? --> I have tried this and it works but let's say if I updated the pipeline code, then will have to rerun the pipeline code then manually get the id and update the task. But if there is way in which I can directly fetch the id using pipeline name, it'll allow me to get the latest pipeline to be scheduled without manual intervention. Please correct if my understanding is wrong.
task_to_schedule = Task.get_task(project_name='MyProject/.pipelines/PipelineName', task_name='PipelineName')
I have tried the given code Snippet and it throws below error:
raise ValueError('No {entity}s found when searching for `{query}`'.format(**locals()))
ValueError: No projects found when searching for `MyProject/.pipelines/PipelineName`
Failed auto-generating package requirements: _PyErr_SetObject: exception SystemExit() is not a BaseException subclass
But this is a different script for TaskScheduler , something like below:
scheduler = TaskScheduler(...)
task_to_schedule = Task.get_task(project_name='MyProject',
task_name='MyTask')
scheduler.add_task(schedule_task_id=task_to_schedule.id, ...)
# Do we have something similar to this? Like below
# pipeline_to_schedule = Task.get_task(project_name='MyProject',
# task_name='MyPipeline')
# scheduler.add_task(schedule_task_id=pipeline_to_schedule.id, ...)
Oh I see
but now I'm confused if this is from code, why aren't you coping the Pipeline ID from the UI?
regrading the query, it should be something like
task_to_schedule = Task.get_task(project_name='MyProject/.pipelines/PipelineName', task_name='PipelineName')
Hi @<1587615463670550528:profile|DepravedDolphin12>
Is there anyway to get the id of the pipeline using pipeline name?
In the UI top right "details" panel should have the Pipeline ID
Is this what you are looking for ?
Hey @<1523701205467926528:profile|AgitatedDove14> I need to access it programmatically. Just like we get the task id by Task.get_task().id
Hi @<1523701205467926528:profile|AgitatedDove14> , It works if I dont specify the project name and just give the task name. So I modified the above code a bit and now I'm able to get the id of the latest pipeline run.
task_to_schedule: Task = Task.get_task(task_id='b0732c334115432f8dc0cec0dbfdbdcb')
print(task_to_schedule.get_project_name())
# Output: Project Team NASA/.pipelines/NASA Pipeline
print(task_to_schedule.name)
# Output: NASA Pipeline #11
# task: Task = Task.get_task(project_name=task_to_schedule.get_project_name(),
# task_name=task_to_schedule.name)
# Error: ValueError: No projects found when searching for `Project Team NASA/.pipelines/NASA Pipeline`
tasks = Task.get_tasks(task_name= task_to_schedule.name,
task_filter={
"status": ["completed"],
"order_by": ["-last_update"]
})
# Returns multiple tasks by the same name
# (The pipeline was run by scheduler multiple times, so same name)
required_task: Task = tasks[0]
# Here I'm picking the latest completed pipeline
print(required_task.id)
# Output: 80836a14e8b0482cab14fa58484b99d0 (id of latest pipeline run)
Hi @<1523701205467926528:profile|AgitatedDove14> , The code returns the project, but if I use the same project in get_task it's throwing ValueError
task_to_schedule: Task = Task.get_task(task_id='b0732c334115432f8dc0cec0dbfdbdcb')
print(task_to_schedule.get_project_name())
# Output: Project Team NASA/.pipelines/NASA Pipeline
print(task_to_schedule.name)
# Output: NASA Pipeline #11
task: Task = Task.get_task(project_name=task_to_schedule.get_project_name(),
task_name=task_to_schedule.name)
# Error: ValueError: No projects found when searching for `Project Team NASA/.pipelines/NASA Pipeline`
Same error
task_to_schedule: Task = Task.get_task(task_id='b0732c334115432f8dc0cec0dbfdbdcb')
print(task_to_schedule.get_project_name())
# Output: Project Team NASA/.pipelines/NASA Pipeline
print(task_to_schedule.name)
# Output: NASA Pipeline #11
task: Task = Task.get_task(project_name=task_to_schedule.get_project_name(),
task_name=task_to_schedule.name,
task_filter={"search_hidden": True})
print(task.id)
# Error: ValueError: No projects found when searching for `Project Team NASA/.pipelines/NASA Pipeline`
@<1523701205467926528:profile|AgitatedDove14> I think this.kigjt be since the project itself is hidden?
Try passing search_hidden: True in the tasks filer
@<1523701087100473344:profile|SuccessfulKoala55> @<1523701205467926528:profile|AgitatedDove14> Same results, it's not able to find the Project
tasks = Task.get_tasks(project_name=task_to_schedule.get_project_name(),
task_name=task_to_schedule.name,
task_filter={"search_hidden": True})
print([task.name for task in tasks])
# Output:
# 2023-07-09 11:08:11,693 - clearml - WARNING - No projects were found with name(s): Project Team NASA/.pipelines/NASA Pipeline
# []
Hi @<1587615463670550528:profile|DepravedDolphin12> , it seems this is an internal issue in the SDK (resolving the project is done without using the task filter) - we will fix this for the next version
then will have to rerun the pipeline code then manually get the id and update the task.
Makes total sense to me!
Failed auto-generating package requirements: _PyErr_SetObject: exception SystemExit() is not a BaseException subclass
Not sure why you are getting this one?!
ValueError: No projects found when searching for
MyProject/.pipelines/PipelineName
hmm, what are you getting with:
task = Task.get_task(pipeline_uid_here)
print(task.get_project_name())
Hi
, It works if I dont specify the project name and just give the task name
But now it searches for it globally , which is not very stable:
Let me check why it fails to find the project...
So inside the pipeline logic you can do Task.current_task().id
Or inside a component Task.current_task().parent