Gotcha. Are the parameters in @PipelineDecorator.pipeline() ignored in the remote context? Settings like the docker image and gitlab repo would already be used before the pipeline is kicked off on the agent.

Okay well I have to supply them again for the function to work, but the values are ignored so i can just have a hard-coded version for remote.

I am still struggling to figure out how to update the parameter defaults, though. I would like to be able to do the equivalent of the PipelineController.add_parameter() so that I can supply a local config with new defaults that are used on the remote execution. Otherwise, I’m stuck with whatever defaults are in the function signature.

Are those fixed from the local environment or do i need to also supply those again in the remote context?

Hi @<1523701205467926528:profile|AgitatedDove14> , sorry for the delayed reply. So what you’re saying is to first kick off a new run and then rename the underlying Pipeline Task, which will cause that particular run to become a new pipeline name? But you have to do this only after you’ve started the run.

What would be most ideal would be to be able to right-click on a pipeline run and have a “clone” option, like you can with a task, where you can start a new run with a new name in a single ...

Sure, I can do that

@<1523701205467926528:profile|AgitatedDove14> : FYI here it is None

import json
import os
import sys
from argparse import ArgumentParser
from logging import getLogger
from pathlib import Path
from typing import Callable

from clearml import PipelineDecorator, Task

from clearml_pipelines_examples.base.pipeline_settings import ExecutionMode
from clearml_pipelines_examples.pipelines.examples.train_model_on_random_data import (
    TrainModelPipelineKwargs,
    TrainModelPipelineSettings,
)
from clearml_pipelines_examples.tasks.examples import generate_dat...

The requirements option using git+https does work, at least for the main install_requires dependences in my setup.cfg. It didn’t install extra dependencies of i tried to do something like pip install my-package[with-optional] @ git+ None

The bash setup script option doesn’t work because that runs before the repo is cloned. I could add the git clone step there, but not sure how to access the git credentials stored in the agent.

Hi @<1523701205467926528:profile|AgitatedDove14> , I've actually hit on something accidentally that might be a clue. I have noticed that when running inside an agent, there is a bug wherein both Task.current_task() and Logger.current_logger() return None . If these are being used by the clearml package under the hood, this could be the reason we aren't seeing the metrics.

As a workaround, I created this utility function, which works for explicit logging (though it doesn't c...

Unfortunately, it's turning out to be quite time consuming to manually remove all of the private info in here. Is there a particular section of the log that would be useful to see? I can try to focus on just sharing that part.

Sure. I can send it on Monday. Thank you.

Hi @<1523701205467926528:profile|AgitatedDove14> , on the resource logging: I tried with a sleep test and it works when I'm running it from my local machine, but when I run remotely in an agent, i do not see resource logging.

And, similarly, with tensorboard logging, it works fine when running from my machine, but not when running remotely in an agent. For this, I've decided to just re-write the logging code to use ClearML's built-in logging methods, which work fine in the agent. Would stil...

Hi @<1523701205467926528:profile|AgitatedDove14> , CLEARML_TASK_ID is set inside the agent's process, which is how I was able to get the task by running Task.get_task(environ["CLEARML_TASK_ID") . However I believe I've sorted out how to make both the resource logging and the tensorboard logging work in the agent. It seems that using Task.current_task() to get the task object does not work when running remotely, but calling Task.init() again does work. And after having called ...

I think this is what you're looking for but let me know if you meant something different:

{
    "meta": {
        "id": "76fffdf3b04247fa8f0c3fc0743b3ccb",
        "trx": "76fffdf3b04247fa8f0c3fc0743b3ccb",
        "endpoint": {
            "name": "tasks.get_by_id_ex",
            "requested_version": "2.30",
            "actual_version": "1.0"
        },
        "result_code": 200,
        "result_subcode": 0,
        "result_msg": "OK",
        "error_stack": "",
        "error_data"...

Hi John, yes we are .

Hi @<1523701070390366208:profile|CostlyOstrich36> , update for you here. I had noticed that the issue was not present for smaller datasets, which led us to discover that the problem was being caused by some nginx (I think) settings with the new server deployment. This was blocking the upload of the "dataset content" object. So our devops team was able to resolve the issue. Thanks very much for your help.

I’m using SDK version 1.10.2 and yes, it’s self-hosted. Here is the version info for the server:

 WebApp: 1.9.1-312 • Server: 1.9.1-312 • API: 2.23 

Thanks!

That could happen with any task when it’s cloned. To be honest, the cron and trigger schedulers probably deserve their own UI panel since they operate differently than other tasks. Ideally, a user would be able to add and remove jobs from the schedulers purely through the UI.

Hi @<1523701205467926528:profile|AgitatedDove14> , sure. I just need to scrape them for any sensitive info then i'll post to this thread. Thanks for your reply.

Here's my example script:

from random import randint

from clearml import Task

if __name__ == "__main__":
    task: Task = Task.init(
        project_name="clearml-examples", task_name="try-to-make-logging-work"
    )
    task.execute_remotely(queue_name="5da90f42dd4c40edab972a4bef8eab04")
    logger = task.get_logger()
    for i in range(10):
        logger.report_scalar("example plot", series="random", value=randint(0, 100), iteration=i)

Hi Martin, I see . That makes sense though I would have expected the behavior to be the same when running remotely the first time as well . In any case, this solved the issue for me . Thanks for looking at it

Correction: it works when I am running the code in my local VSCode session. I still don't get resource logging when I run in an agent. 🤔 . And on a similar topic, I have a separate task that is logging metrics with tensorboard. When running locally, I see the metrics appear in the "scalars" tab in ClearML, but when running in an agent, nothing. Any suggestions on where to look?

Okay, I take it back. os.getenv("CLEARML_TASK_ID") does work. I forgot to rebuild my container after making the change. Thanks for bringing this option to my attention!

I actually have a question about your original code snipped, @<1556450111259676672:profile|PlainSeaurchin97> . I have been trying to figure out a way to access the task object when running remotely so that I can instantiate the logger but when I tried task_id = os.getenv("CLEARML_TASK_ID") , it’s returning None . I also tried Task.current_task() and also got None back. What is the recommended way to access the Task object from within the remote agent?

Hi @<1523701205467926528:profile|AgitatedDove14> , thanks so the code to be executed by the task needs to be provided to the Task.create() method as script=some/path.py or does it work to have something like

def my_node_task_factory(node: PipelineController.Node) -> Task:
  task = Task.create(...)
  my_function()
  return task 

Sorry, i meant the arguments that are supplied to the decorator method, itself @PipelineDecorator.pipeline() and @PipelineDecorator.component() , things like name , project , docker_args , etc.

Ahhh okay, thank you. Perhaps in the future, it would be great to allow this from the UI as well?

Hi Max, thanks very much for your message! I understand what you’re saying now, though I suppose this is not my issue since I’m not setting any of the decorator values with variables. I’ll post a query in the main channel with code snippets to see if anyone has ideas. Thank you!

Thanks very much! Yeah, it tends to fill up the console