This is with:
Task.set_offline_mode(True) task = Task.init(..., auto_connect_streams=False)

I'm working on the config object references 😉

I'm trying to decide if ClearML is a good use case for my team 🙂
Right now we're not looking for a complete overhaul into new tools, just some enhancements (specifically, model repository, data versioning).

We've been burnt by DVC and the likes before, so I'm trying to minimize the pain for my team before we set out to explore ClearML.

That's fine for the current use-case I believe.
Once the team is happy with the logging functionality, we'll move on to remote execution and things will update.

If I add the bucket to that (so CLEARML_FILES_HOST= s3://minio_ip:9000/minio/bucket ), I then get the following error instead --

2021-12-21 22:14:55,518 - clearml.storage - ERROR - Failed uploading: SSL validation failed for ... [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1076)

Ah, uhhhh whatever is in the helm/glue charts. I think it’s the allegroai/clearml-agent-k8s-base , but since I hadn’t gotten a chance to try it out, it’s hard to say with certainty which would be the best for us 😛

Is it CLEARML_CONFIG_FILE ? (I had to dig this from the GH code 😅 )

Or if it wasn't clear, that chunk of code is from clearml's dataset.py

I've also followed https://clearml.slack.com/archives/CTK20V944/p1628333126247800 but it did not help

I will TIAS, but maybe worthwhile to also mention if it has to be the absolute path or if relative path is fine too!

Honestly I wouldn't mind building the image myself, but the glue-k8s setup is missing some documentation so I'm not sure how to proceed

I think so, it was just missing from the official documentation 🙂 Thanks!

Any updates @<1523701087100473344:profile|SuccessfulKoala55> ? 🫣

AFAICS it's quite trivial implementation at the moment, and would otherwise require parsing the text file to find some references, right?
https://github.com/allegroai/clearml/blob/18c7dc70cefdd4ad739be3799bb3d284883f28b2/clearml/task.py#L1592

Why not give ClearML read-only access credentials to the repository?

Right so this is checksum based? Are there plans to only store delta changes for files (i.e. store the changed byte instead of the entire file)?

Just because it's handy to compare differences and see how the data changed between iterations, but I guess we'll work with that 🙂
We'll probably do something like:
When creating a new dataset with a parent (or parents), look at immediate parents for identically-named files If those exist, load those with matching framework (pyarrow, pandas, etc), and log differences to the new dataset 🙂

I also tried switching to dockerized mode now, getting the same issue 🤔

I opened a GH issue shortly after posting here. @<1523701312477204480:profile|FrothyDog40> replied (hoping I tagged the right person).

We need to close the task. This is part of our unittests for a framework built on top of ClearML, so every test creates and closes a task.

Yes exactly that AgitatedDove14
Testing our logic maps correctly, etc for everything related to ClearML

• in the second scenario, I might have not changed the results of the step, but my refactoring changed the speed considerably and this is something I measure.
• in the third scenario, I might have not changed the results of the step and my refactoring just cleaned the code, but besides that, nothing substantially was changed. Thus I do not want a rerun.Well, I would say then that in the second scenario it’s just rerunning the pipeline, and in the third it’s not running it at all 😄
  (I ...

It does not 🙂
We started discussing it here - https://clearml.slack.com/archives/CTK20V944/p1640955599257500?thread_ts=1640867211.238900&cid=CTK20V944
You suggested this solution - https://clearml.slack.com/archives/CTK20V944/p1640973263261400?thread_ts=1640867211.238900&cid=CTK20V944
And I eventually found this solution to work - https://clearml.slack.com/archives/CTK20V944/p1641034236266500?thread_ts=1640867211.238900&cid=CTK20V944

Well, -ish. Ideally what we're after is one of the following:
Couple a task with a dataset. Keep it visible in it's destined location. Create a dataset separately from the task. Have control over its visibility and location. If it's hidden, it should not affect normal UI interaction (most annoying is having to click twice on the same project name when there are hidden datasets, which do not appear in the project view)

I’ll also post this on the main channel -->

I'm using 1.1.6 (upgraded from 1.1.6rc0) - should I try 1.1.7rc0 or smth?

Ah I see, if the pipeline controller begins in a Task it does not add the tags to it…

Still failing with 1.2.0rc3 😞 AgitatedDove14 any thoughts on your end?

Yeah I managed to work around those former two, mostly by using Task.create instead of Task.init . It's actually the whole bunch of daemons running in the background that takes a long time, not the zipping.

Regarding the second - I'm not doing anything per se. I'm running in offline mode and I'm trying to create a dataset, and this is the error I get...
There is a data object it, but there is no script object attached to it (presumably again because of pytest?)

Also something we are very much interested in (including the logger-based scatter plots etc)

In which repo?:)