When I use the APIClient to fetch the tags for the project, I get an empty collection of system tags:

<projects.GetProjectTagsResponse: {
    "tags": [],
    "system_tags": []
}>

Ah I see, if the pipeline controller begins in a Task it does not add the tags to it…

I didn't mention code in #340 nor did I mention data here 😄 The idea was to package non git-specific files for remote execution

Nope, no other config files

Is it CLEARML_CONFIG_FILE ? (I had to dig this from the GH code 😅 )

That will come at a later stage

clearml.backend_api.session.defs.ENV_HOST.get() did not work unfortunately 🤔

Let me try that real quick

But it does work on linux 🤔 I'm using it right now and the environment variables are not defined in the terminal, only in the .env 🤔

As in here SuccessfulKoala55 - https://github.com/allegroai/clearml/blob/master/clearml/automation/cloud_driver.py#L169

I've also followed https://clearml.slack.com/archives/CTK20V944/p1628333126247800 but it did not help

Yeah, and just thinking out loud what I like about the numpy/pandas documentation

That gives us the benefit of creating "local datasets" (confined to the scope of the project, do not appear in Datasets tabs, but appear as normal tasks within the project)

AFAICS it's quite trivial implementation at the moment, and would otherwise require parsing the text file to find some references, right?
https://github.com/allegroai/clearml/blob/18c7dc70cefdd4ad739be3799bb3d284883f28b2/clearml/task.py#L1592

BTW AgitatedDove14 following this discussion I ended up doing the regex way myself to sync these, so our code has something like the following. We abuse the object description here to store the desired file path.

` config_path = task.connect_configuration(configuration=config_path, name=config_fname)
included_files = find_included_files_in_source(config_path)
while included_files:
file_to_include = included_files.pop()
sub_config = task.connect_configuration(
configurat...

Yeah that works fine 😄 I just fetch it once to map argparse users to their IDs for later filtering.

The Slack Monitoring example should be updated btw, as they now use slack_sdk instead of slack (in the import statements)

That's fine as well - the code simply shows the name of the environment variable, not it's value, since that's taken directly from the agent listening to the services queue (and who's then running the scaler)

I think you're interested in the Monitor class:)

This seems to be fine for now, if any future lookups finds this thread, btw
with mock.patch('clearml.datasets.dataset.Dataset.create'): ...

I'm using 1.1.6 (upgraded from 1.1.6rc0) - should I try 1.1.7rc0 or smth?

Sorry AgitatedDove14 , forgot to get back to this.
I've been trying to convince my team to drop poetry 😄

Great, thanks!

I think so, it was just missing from the official documentation 🙂 Thanks!

Or the above;)

Yes it would be 🙂
Visualization is always a difficult topic... I'm not sure about that, but a callback would be nice.

One idea that comes to mind (this is of course limited to DataFrames), but think the git diff , where I imagine 3 independent section:
Removed columns (+ truncated preview of removed values) (see below) Added columns (+ truncated preview of removed values)
The middle column is then a bit complicated, but I would see some kind of "shared columns" dataframe, where each ...

Just because it's handy to compare differences and see how the data changed between iterations, but I guess we'll work with that 🙂
We'll probably do something like:
When creating a new dataset with a parent (or parents), look at immediate parents for identically-named files If those exist, load those with matching framework (pyarrow, pandas, etc), and log differences to the new dataset 🙂

Right so this is checksum based? Are there plans to only store delta changes for files (i.e. store the changed byte instead of the entire file)?

Any follow up thoughts SuccessfulKoala55 or CostlyOstrich36 ?

Any updates @<1523701087100473344:profile|SuccessfulKoala55> ? 🙂

Should this be under the clearml or clearml-agent repo?