Ah, ok thanks so much, I'll work on this now.

@<1523701070390366208:profile|CostlyOstrich36> here is the INFO section:

Yes, I started the agent in docker mode with clearml-agent daemon --queue direct-relief-forecasting --docker directrelief_ml_clearml --cpu-only -d

I am using the pipeline id of when I last ran the pipeline and got this through the UI in ClearML.

@<1523701070390366208:profile|CostlyOstrich36> , a quick follow up, I've been looking at the ClearML API documentation to see how to trigger a pipeline via the API. Do you use queues and add_task , as specified here: None ?

Here is an example of the pipeline code, simplified:

"""Forecasting Pipeline"""

from clearml.automation.controller import PipelineDecorator
from clearml import TaskTypes

@Pipe...

I'm just installing the required packages (one of which is an internal package on CodeArtifact).

It says N/A for Status?

Ok, so the image needs ssh in order to clone the repo, not just the server.

This is my dockerfile:

# Use a slim Python image as the base image
FROM python:3.9.9-slim as builder

ARG CODEARTIFACT_AUTH_TOKEN
ARG POETRY_HTTP_BASIC_ARTIFACT_USERNAME
ARG POETRY_HTTP_BASIC_ARTIFACT_PASSWORD

# Set environment variables for Poetry
ENV POETRY_HOME=/opt/poetry \
    POETRY_NO_INTERACTION=1 \
    POETRY_VIRTUALENVS_CREATE=false \
    PATH="/opt/poetry/bin:$PATH"\
    PYTHONPATH="/ml_pipeline:${...

Following up, I found this in the code on Github: None

A taskid is required though - to get this id would we run the pipeline manually as it then shows up in the Web UI and then just use the id of the task that we can get from the UI by clicking on the pipeline run info?

Hi @<1523701070390366208:profile|CostlyOstrich36> , I've gotten past the environment set up phase - I have the agent running in --services-mode for the pipeline, but the pipeline step is failing and I can't see why from the error message. When I run this locally it works (ie. if I add PipelineDecorator.run_locally() ).

Environment setup completed successfully

Starting Task Execution:


1710160150496 direct-relief:cpu:0:service:5062dce7ff1d49cfbaf36c96abe3282c DEBUG ClearML res...

@<1523701205467926528:profile|AgitatedDove14> Hi! Could you give any feedback on the above? We are trying to figure out if/how we can run pipelines on a schedule and also trigger them with an external event.

Hi @<1523701205467926528:profile|AgitatedDove14> , thanks for your reply. We want to use the Scheduler but to run a pipeline. Looking at this example here, it looks like it only works with tasks: None

So, in my code example above, where I have executing_pipeline as the pipeline function created with the decorator, can this be scheduled to run with the TaskScheduler , ie. used as the function...

Hi @<1523701070390366208:profile|CostlyOstrich36> , another quick question, can you run a pipeline on a schedule or are schedules only for Tasks? We are battling to figure out how to automate the pipelines.

I have an agent running and assigned to the queue megan-testing :

Thanks @<1523701070390366208:profile|CostlyOstrich36> , I've figured this out and specified how to build the ClearML agent now with the correct python version.

Yes, I have a log in for the clearml server and I've set up an agent using a docker image. And I've added the ssh keys here. I then start the agent with clearml-agent daemon --queue direct-relief-forecasting --cpu-only -d But, then when I try run the pipeline remotely, I'm getting the error above.

Ah, great, thanks.

Yes, it does. And I can clone this repo and branch to the server outside of clearml just using git clone .... as I've added ssh keys and authenticated.

Run is still marked as "Completed"

I've figured out that this is because I have a config.yaml file with secrets in it in the repository. This is not committed to git. So, when running remotely, the file is no present. Is the recommendation to put this in the docker image and then I have to specify an entry point in the dockerfile? Previously, I was hoping to just get away with creating a docker image with the installed packages for the agent, not with the repository code as well. Is this not the recommended approach?

Thanks, @<1523701070390366208:profile|CostlyOstrich36> .

If I use Task.current_task() , I then get a task object back that looks like <clearml.task.Task object at 0x7f7723b90a00> . How does one get the url from this?

Here is a sample from the end of the logs where it failed:

/home/megan/code/direct-relief-forecasting/forecast.py:277: FutureWarning:
The behavior of DatetimeProperties.to_pydatetime is deprecated, in a future version this will return a Series containing python datetime objects instead of an ndarray. To retain the old behavior, call `np.array` on the result
2024-07-04 12:14:04.589 | INFO     | forecast:infer:281 - Inferring for dates: ['2024-06-13']
2024-07-04 12:14:04.734 | INFO     |...

Hi @<1523701205467926528:profile|AgitatedDove14> , I'm still having issues with this set up. See my latest comment here: None

I created a new queue megan-testing and have an agent running on my machine that I assigned to it. It works when I just use a simple task and schedule it, but when I try run the pipeline, it says it can't find the queue.

Thanks @<1523701070390366208:profile|CostlyOstrich36> , that does sound like an option. Can you point me to the documentation for this API call as I haven't been able to find anything?

Ah ok, so if I have two steps, this means 3 queues with 3 agents, one for the controller and one each dedicated to the steps, just to confirm?

Oh my word 🙈 Yes, I just saw this just and came back to reply on here. Thank you!

I think I found it: None

Will this work for pipelines too, which are also Tasks, right?

For reference for anyone else, this is what I did:

current_task = Task.current_task()
model_run_url = Task.get_task_output_log_web_page(
        task_id=current_task.id,
        project_id="abc")

Ah, I see, this makes sense now. Thank you!

I've tried changing the clearml.conf file to specify the python_binary with:

 python_binary: "usr/bin/python3"

But, still getting the same error.