Hi @<1798887590519115776:profile|BlushingElephant78>

have multiple steps that run inside a gitlab repo

one thing to make sure that is not missed, "inside a gitlab repo" , notice the actual pipeline is running on agents/workers/k8s gitlab will be the trigger and monitor it, but the actual execution should probably happen somewhere with proper compute

this works fine when i run the pipeline from the script directly but when i run it from the web interface

try to configure your...

Nice catch AverageBee39 🙂

JitteryCoyote63 Is this an Ignite feature ? what is the expectation ? (I guess the ClearML Logger just inherits from the base ignite logger)

JitteryCoyote63 I think there is a ClearML logger , no?

Hmm that is odd, let me see if I can reproduce it.
What's the clearml version you are using ?

JitteryCoyote63 nice hack 😄
how come it is not automatically logged as console output ?

Okay, let me see...

terminal

LethalCentipede31 sure:
task.upload_artifact(object_or_file, name)https://github.com/allegroai/clearml/blob/master/examples/reporting/artifacts.py

So what you’re saying is to first kick off a new run and then rename the underlying Pipeline Task, which will cause that particular run to become a new pipeline name?

Correct, basically you are not changing the "pipeline" per-se but the execution name of the pipeline, if that makes sense

What would be most ideal would be to be able to right-click on a pipeline run and have a “clone” option, like you can with a task, where you can start a new run with a new name in a single step.

...

RoughTiger69 the easiest thing would be to use the override option of Hydra:
parameter_override={'Args/overrides': '[the_hydra_key={}]'.format(a_new_value)})wdyt?

Okay this is a bit tricky (and come to think about it, we should allow a more direct interface):
pipe.add_step(name='train', parents=['data_pipeline', ], base_task_project='xxx', base_task_name='yyy', task_overrides={'configuration.OmegaConf': dict(value=yaml.dump(MY_NEW_CONFIG), name='OmegaConf', type='OmegaConf YAML')} )Notice that if you had any other configuration on the base task, you should add them as well (basically it overwrites the configurati...

Hi

, It works if I dont specify the project name and just give the task name

But now it searches for it globally , which is not very stable:

Let me check why it fails to find the project...

then will have to rerun the pipeline code then manually get the id and update the task.

Makes total sense to me!

Failed auto-generating package requirements: _PyErr_SetObject: exception SystemExit() is not a BaseException subclass

Not sure why you are getting this one?!

ValueError: No projects found when searching for MyProject/.pipelines/PipelineName

hmm, what are you getting with:

task = Task.get_task(pipeline_uid_here)
print(task.get_project_name())

This seems to be okay to me, are you seeing the dataset in the web UI?
Also:

my_local_dataset_folder = Dataset.get(dataset_project=project, dataset_name=name).get_mutable_local_copy()

what exactly are you seeing in " my_local_dataset_folder " directory?
(it should contain the copy of the S3 file)

Hmm, can you send the full log of the pipeline component that failed, because this should have worked
Also could you test it with the latest clearml python version (i.e. 1.10.2)

(you can find it in the pipeline component page)

What do you have in the artifacts of this task id: 4a80b274007d4e969b71dd03c69d504c

None

What is the link you are seeing there?

I think your "files_server" is misconfigured somewhere, I cannot explain how you ended up with this broken link...
Check the clearml.conf on the machines or the env vars ?

Hi @<1751777160604946432:profile|SparklingDuck54>
Actually the Dataset ID on the task can be easily pulled via:

Task.get_task("task_uid_with_dataset").get_parameter("Datasets/<dataset_alias_name>")

or

Task.get_task("task_uid_with_dataset").get_parameters_as_dict().get("Datasets")

Hi @<1523702307240284160:profile|TeenyBeetle18>

and url of the model refers to local file, no to the remote storage.

Do you mean that in the Model tab when you look into the model details the URL points to a local location (e.g. file:///mnt/something/model) ?
And your goal is to get a copy of that model (file) from your code, is that correct ?

Hi @<1523702786867335168:profile|AdventurousButterfly15>

I am running cross_validation, training a bunch of models in a loop like this:

Use the wildcard or disable all together:

task = Task.init(..., auto_connect_frameworks={"joblib": False})

You can also do

task = Task.init(..., auto_connect_frameworks={"joblib": ["realmodelonly.pkl", ]})

should be the full path, or just the file name?

just file name, this is basically fname matching

so firs yes, I totally agree. This is why the clearml-serving has a dedicated statistics module that creates histograms over time, then we push it into Prometheus and connect grafana to it for dashboards and alerts.
To be honest, I would just use it instead of reporting manually, wdyt?

Hi @<1724960475575226368:profile|GloriousKoala29>

Is there a way to aggregate the results, such as defining an iteration as the accuracy of 100 samples

Hmm, i'm assuming what you actually want is to store it with the actual input/output and a score, is that correct?

is it possible to change an existing model's URL?

Edit the DBs ... That's basically the only way 😞

Nice!!!

Hi ThoughtfulElephant4

I was trying to build an image using clearml server dockerfile ,

Are you saying you are rebuilding the docker image for the clearml-server and it fails ?
Can you provide the full console log?