Thanks Alon. In the full/official documentation the clearml-data CLI is not mentioned anywhere, so perhaps it should be refreshed 😉

I think we're referring to different things here.
I won't be using the UI (and neither will my team).
But as mentioned, we've used DVC before and it adds a lot of junk metadata files to each GitHub PR (many dvc.yaml , dvc.lock and .gitignore files). We're trying to avoid that as much as possible, hence my question about GitHub pull...

I see! The Hyper Datasets don't really fit our use case - it seems really focused on CNNs and image-based data, but lacking support for database-oriented tabular data.

So for now we mainly work with parquet and CSV files, and I was hoping there'd be an easy way to view those... I'll make a workaround with a "Datasets" project I suppose!

Btw TimelyPenguin76 this should also be a good starting point:
First create the target directory and add some files:sudo mkdir /data/clearml sudo chmod 777 -R /data/clearml touch /data/clearml/foo touch /data/clearml/bar touch /data/clearml/bazThen list the files using the StorageManager. It shouldn't take more than a few miliseconds.` from clearml import StorageManager

%%timeit
StorageManager.list("/data/clearml")

-> 21.2 s ± 328 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) `

It's self-hosted TimelyPenguin76
SuccessfulKoala55 WebApp: 1.4.0-175 • Server: 1.4.0-175 • API: 2.18

@<1523701087100473344:profile|SuccessfulKoala55> could you provide some instructions?

For example:

my-repo @ git+

I think now there's the following:
Resource type Queue (name) defines resource + max instancesAnd I'm looking for:
Resource type "pool" of resources (type + max instances) A pool can be shared among queues

Added the following line under volumes for apiserver , fileserver , agent-services :
- /data/clearml:/data/clearml

Running a self-hosted server indeed. It's part of a code that simply adds or uploads an artifact 🤔

/subscribe
/takemymoney

Woot! What about clearml-agent 1.2.0? 🙂

Yes, that one shows up. I forgot to mention we also set the version explicitly, but that just creates a duplicate dataset under Datasets and anyway our main Task is now hidden from the original project.
So project project exists, but it is empty.

SuccessfulKoala55 The changelog wrongly cites https://github.com/allegroai/clearml/issues/400 btw. It is not implemented and is not related to being able to save CSVs 😅

Yes, I’ve found that too (as mentioned, I’m familiar with the repository). My issue is still that there is documentation as to what this actually offers.
Is this simply a helm chart to run an agent on a single pod? Does it scale in any way? Basically - is it a simple agent (similiar to on-premise agents, running in the background, but here on K8s), or is it a more advanced one that offers scaling features? What is it intended for, and how does it work?

The official documentation are very spa...

The only thing I could think of is that the output of pip freeze would be a URL?

Couldn't the agent just come with the toml library? Kinda easy to load up and check if poetry is present then... 🤔

But yes it indeed used poetry correctly, though it would fail in other circumstances

This could be relevant SuccessfulKoala55 ; might entail some serious bug in ClearML multiprocessing too - https://stackoverflow.com/questions/45665991/multiprocessing-returns-too-many-open-files-but-using-with-as-fixes-it-wh

An internal project I've accidentally made with a hidden tag while playing around with the ClearML internal code.

The error seems to come from this line:
self._driver = _FileStorageDriver(str(path_driver_uri.root)) (line #353 in clearml/storage/helper.py
Where if the path_driver is a local path, then the _FileStorageDriver starts with a base_path = '/' , and then takes extremely long time at iterating over the entire file system (e.g. in _get_objects , line #1931 in helper.py )

Using the PipelineController with add_function_step

But... Which queue does it listen to, and which type of instances will it use etc

Hm, just a small update - I just verified and it does indeed work on linux:

` import clearml
import dotenv

if name == "main":
dotenv.load_dotenv()
config = clearml.backend_api.Config.load() # Success, parsed with environment variables `

And agent too, I hope..?

I'd be happy to join a #releases channel just for these!

Just randomly decided to check and saw there's a server 1.4 ready 🎉

I have no idea what’s the difference, but it does not log the internal repository 😞 If I knew why, I would be able to solve it myself… hehe

Still; anyone? 🥹 @<1523701070390366208:profile|CostlyOstrich36> @<1523701205467926528:profile|AgitatedDove14>

SuccessfulKoala55 That string was autogenerated by pyhocon and matches their documentation too - https://github.com/lightbend/config/blob/master/HOCON.md#substitutions
The first example won't work (it will treat ${...} as a string literal and won't replace it). The second does work, but as mentioned anyway, these were not hand typed, but rather generated from pyhocon, so I don't think that's the issue 🤔

I can scroll sideways but if I open any of the comparison items, I pretty much can only see one experiment's values

Well you could start by setting the output_uri to True in Task.init .

PricklyRaven28 That would be my fallback, it would make development much slower (having to build containers with every small change)

No, fixed 🙂