I found this issue but not sure if it's the same thing as it's from awhile back: None and also not sure what the final solution here was.

Ah, ok thanks so much, I'll work on this now.

I think I found it: None

Will this work for pipelines too, which are also Tasks, right?

Run is still marked as "Completed"

Ok, thanks! Going to try this now. I included an entry point from reading some other messages on Slack here when trying to figure out how to use Docker for running remotely.

Ah, great, thanks.

I'm just installing the required packages (one of which is an internal package on CodeArtifact).

Sorry, was trying to figure out how to do this. So, looks like I can't as get an error NameError: name 'ssh' is not defined

Also, we use Dagster for orchestration.

@<1523701070390366208:profile|CostlyOstrich36> here is the INFO section:

It says N/A for Status?

Yes, I started the agent in docker mode with clearml-agent daemon --queue direct-relief-forecasting --docker directrelief_ml_clearml --cpu-only -d

Hi @<1523701070390366208:profile|CostlyOstrich36> , another quick question, can you run a pipeline on a schedule or are schedules only for Tasks? We are battling to figure out how to automate the pipelines.

Thanks @<1523701070390366208:profile|CostlyOstrich36> , that does sound like an option. Can you point me to the documentation for this API call as I haven't been able to find anything?

Here is a sample from the end of the logs where it failed:

/home/megan/code/direct-relief-forecasting/forecast.py:277: FutureWarning:
The behavior of DatetimeProperties.to_pydatetime is deprecated, in a future version this will return a Series containing python datetime objects instead of an ndarray. To retain the old behavior, call `np.array` on the result
2024-07-04 12:14:04.589 | INFO     | forecast:infer:281 - Inferring for dates: ['2024-06-13']
2024-07-04 12:14:04.734 | INFO     |...

Ok, so the image needs ssh in order to clone the repo, not just the server.

This is my dockerfile:

# Use a slim Python image as the base image
FROM python:3.9.9-slim as builder

ARG CODEARTIFACT_AUTH_TOKEN
ARG POETRY_HTTP_BASIC_ARTIFACT_USERNAME
ARG POETRY_HTTP_BASIC_ARTIFACT_PASSWORD

# Set environment variables for Poetry
ENV POETRY_HOME=/opt/poetry \
    POETRY_NO_INTERACTION=1 \
    POETRY_VIRTUALENVS_CREATE=false \
    PATH="/opt/poetry/bin:$PATH"\
    PYTHONPATH="/ml_pipeline:${...

I found this issue but not sure if it's the same thing as it's from awhile back: None And the link I'm trying to clone doesn't start with https but with git@bitbucket

Yes, it does. And I can clone this repo and branch to the server outside of clearml just using git clone .... as I've added ssh keys and authenticated.

For reference for anyone else, this is what I did:

current_task = Task.current_task()
model_run_url = Task.get_task_output_log_web_page(
        task_id=current_task.id,
        project_id="abc")

Hi @<1523701205467926528:profile|AgitatedDove14> , thanks for your reply. We want to use the Scheduler but to run a pipeline. Looking at this example here, it looks like it only works with tasks: None

So, in my code example above, where I have executing_pipeline as the pipeline function created with the decorator, can this be scheduled to run with the TaskScheduler , ie. used as the function...

I am using the pipeline id of when I last ran the pipeline and got this through the UI in ClearML.

Ah ok, so if I have two steps, this means 3 queues with 3 agents, one for the controller and one each dedicated to the steps, just to confirm?

Thanks @<1523701070390366208:profile|CostlyOstrich36> , I've figured this out and specified how to build the ClearML agent now with the correct python version.

Oh my word 🙈 Yes, I just saw this just and came back to reply on here. Thank you!

Hi @<1523701205467926528:profile|AgitatedDove14> , I'm still having issues with this set up. See my latest comment here: None

I created a new queue megan-testing and have an agent running on my machine that I assigned to it. It works when I just use a simple task and schedule it, but when I try run the pipeline, it says it can't find the queue.

I have an agent running and assigned to the queue megan-testing :

Thanks, I'll have a look at the youtube videos. I've been going over the documentation a lot and haven't found much about actually running and deploying pipelines, and a lot of details seem to be missing.

Schedule testing code:

# Schedule for running the pipeline daily

from clearml import Task
from clearml.automation import TaskScheduler


def simple_function():
    print('This code is executed in a background thread, '
          'on the same machine as the TaskScheduler process')
    # add some logic here
    print('done')

if __name__ == "__main__":

    scheduler = TaskScheduler(force_create_task_name='megan-test-remote-pipeline')

    scheduler.add_task(
        name='Simple Pipeli...

@<1523701205467926528:profile|AgitatedDove14> Hi! Could you give any feedback on the above? We are trying to figure out if/how we can run pipelines on a schedule and also trigger them with an external event.

This is schedule.py where the schedule_task_id is the id for a pipeline that has previously been run and now we want to schedule it:

from clearml import Task
from clearml.automation import TaskScheduler

if __name__ == "__main__":

    scheduler = TaskScheduler(force_create_task_name='megan-test-remote-pipeline')

    scheduler.add_task(
        name='Simple Pipeline Schedule Run Test',
        schedule_task_id='94a7e898bb3f4494a31828187710f5bd',
        queue='megan_testing',
...