looks like a great idea, I'll make sure to pass it along and that someone reply 🙂

• Components anyway need to be available when you define the pipeline controller/decorator, i.e. same codebaseNo you an specify a different code base, see here:
  None
• The component code still needs to be self-composed (or, function component can also be quite complex)Well it can address the additional repo (it will be automatically added to the PYTHONPATH), and you c...

Scenario 1 & 2 are essentially the same from caching perspective (the face B != B` means they have different caching hashes, but in both cases are cached).
Scenario 3 is the basically removing the cache flag from those components.

Not sure if I'm missing something.

Back to the @<1523701083040387072:profile|UnevenDolphin73>

From decorators - when the pipeline logic is very straightforward ...

Actually I would disagree, the decorators should be used when the pipeline Logic is not a D...

I pass my dataset as parameter of pipeline:

@<1523704757024198656:profile|MysteriousWalrus11> I think you were expecting the dataset_df dataframe to be automatically serialized and passed, is that correct ?
If you are using add_step, all arguments are simple types (i.e. str, int etc.)
If you want to pass complex types, your code should be able to upload it as an artifact and then you can pass the artifact url (or name) for the next step.

Another option is to use pipeline from dec...

mean? Is it not possible that I call code that is somewhere else on my local computer and/or in my code base? That makes things a bit complicated if my current repository is not somehow available to the agent.

I guess you can ignore this argument for the sake of simple discussion. If you need access to extra files/functions, just make sure you point the repo argument to their repo, and the agent will make sure your code is running from the repo root, with all the repo files under i...

In terms of creating dynamic pipelines and cyclic graphs, the decorator approach seems the most powerful to me.

Yes that is correct, the decorator approach is the most powerful one, I agree.

Hi @<1540142641931358208:profile|FancyBaldeagle86>
You mean in the UI? i.e. clone an experiment hover over the Configuration / Hyperparameter section and clicking edit ?

I basically just mean having a date input like you would in excel where it brings up a calendar and a clock if it’s time – and defaults to “now”

I would love that as well, but I kind of suspect the frontend people will say these things tend to start small and grow into a huge effort. At the moment what we do is the UI is basically plain text and the casting is done on the SDK side.
You can however provide type information and help (you can see it when you hover over the arguments on th...

@<1523701079223570432:profile|ReassuredOwl55>

Hey, here’s a quickie – is it possible to specify different “types” of input parameters (“Args/…“) such that they are handled nicely on the front end?

You me cast / checked in the UI ?

At the top there should be the URL of the notebook (I think)

Hi @<1541229818828296192:profile|HurtHedgehog47>

plots we create in the notebook are not saved as it was made.

I'm assuming these are matplotlib plots ?
Notice that ClearML tries to convert the plot into interactive plots, in that process sometimes, colors and legend is being lost (becomes generic).
You can however manually report the plot, and force it to store it as non-interactive:

task.logger.report_matplotlib_figure(
    title="Manual Reporting", series="Just a plot", ite...

to avoid downgrade to clearml==1.9.1
I will make sure this is solved in clearml==1.9.3 & clearml-session==0.5.0 quickly

what's the clearml package version and clearml-session version ?

StorageManager 🙂

The one it is trying to execute, i.e. on the Task it shows as Script Path

Try to add here:
None

server_info['url'] = f"http://{server_info['hostname']}:{server_info['port']}/"

What happens when you call:

from clearml.backend_interface.task.repo import ScriptInfo

print(ScriptInfo._ScriptInfo__legacy_jupyter_notebook_server_json_parsing(None))

Hmm and you are getting empty list for thi one:

server_info['url'] = f"http://{server_info['hostname']}:{server_info['port']}/"

@<1523704157695905792:profile|VivaciousBadger56>

Is the idea here the following? You want to use inversion-of-control such that I provide a function

f

to a component that takes the above dict an an input. Then I can do whatever I like inside the function

f

and return a different dict as output. If the output dict of

f

changes, the component is rerun; otherwise, the old output of the component is used?

Yes exactly ! this way you...

Is the code in this "other" repo downloaded to the agent's machine? Or is the component's code pushed to the machine on which the repository is?

Yes this repo is downloaded into the agent, so your code has access to it

I would like to start off by saying that I absolutely love clearml.

@<1547028031053238272:profile|MassiveGoldfish6> thank you for saying that! 😍

Is is possible to download individual files from a dataset without downloading the entire dataset? If so, how do you do that?

Well by default files are packaged into multiple zip files, you can control the size of the zip file for finer granularity, but at the end when you download, you are downloading the entire packaged ...

I think that by default the zipped package files are 0.5GB
(you can control it None look for --chunk-size)
I think the missing part of the api is understanding which chunk your specific file stored in.
You can do something like:

ds = Dataset.get(...)
the_artifact_chunk_I_need = ds.file_entries_dict["myt/file/here"].artifact_name

wdyt?
maybe worth to add an interface ?

Hi @<1673501397007470592:profile|RelievedDuck3>

how can I configure my alerts to be notified when the distribution of my metrics (variables) changes on my heatmaps?

This can be done inside grafana, here is a simple example:
None
Specifically you need to create a new metric that is the distance of current distribution (i.e. heatmap) from the previous window), then on the distance metric, ...

Hi SmarmyDolphin68
You have two options:
Automatically upload the models when training pass output_uri to Task.init. For example output_uri=True will upload to the clearml-server, output_uri=' s3://bucket/folder ' will upload to S3 etc. Manually upload a model that you have locally: https://github.com/allegroai/clearml/blob/9ff52a8699266fec1cca486b239efa5ff1f681bc/examples/reporting/model_config.py#L37

Will be shortly released with news RC :)

My pleasure

Where again does clearml place the venv?

Usually ~/.clearml/venvs-builds/<python version>/
Multiple agents will be venvs-builds.1 and so on

SmallBluewhale13
And the Task.init registers 0.17.2 , even though it prints (while running the same code from the same venv) 0.17.2 ?

and you have clearml v0.17.2 installed on the "system" packages level, and 0.17.5rc6 installed inside the pyenv venv ?

hen, in the bash console, after some time, I see some messages being logged from clearml

JitteryCoyote63 Hmm that is strange, let me check something