My current approach with pipelines basically looks like a GH CICD yaml config btw, so I give the user a lot of control on which steps to run, why, and how, and the default simply caches all results so as to minimize the number of reruns.

The user can then override and choose exactly what to do (or not do).

It could be related to ClearML agent or server then. We temporarily upload a given .env file to internal S3 bucket (cache), then switch to remote execution. When the remote execution starts, it first looks for this .env file, downloads it using StorageManager, uses dotenv, and then continues the execution normally

That's what I found as well, but it did not like it after all (boto is fine with it, but underlying urllib and requests were not?)

It's fine -- I see the added benefit in making sure the users set up their clearml.conf and I've made a script to edit it to our needs as part of the installation process 🙂 Thanks Martin!

Just because it's handy to compare differences and see how the data changed between iterations, but I guess we'll work with that 🙂
We'll probably do something like:
When creating a new dataset with a parent (or parents), look at immediate parents for identically-named files If those exist, load those with matching framework (pyarrow, pandas, etc), and log differences to the new dataset 🙂

It's given as the second form you suggested in the mini config ( http://${...}:8080 ). The quotation marks are added later by pyhocon.

Maybe. When the container spins, are there any identifiers regarding the task etc available? I create a folder on the bucket per python train.py so that the environment variables files doesn't get overwritten if two users execute almost-simultaneously

I'll try with 1.1.5 first, then 1.1.6rc0

Of course now it's not there anymore 😆 If/when it happens again I'll ping you here 🙂

Would be good if that's mentioned explicitly in the docs 😄 Thanks!

Let me test it out real quick.

Since the additional credentials are available to the autoscaler when it boots up (via the config file), I thought it could use those natively?

https://clearml.slack.com/apps/A019MGGA2Q1-around ?

Sorry for the late reply Jake -- I was away on holidays -- it works perfectly now, thanks!

QuaintPelican38 did you have a workaround for this then? Some cleanup service or similar?

The documentation is messy, I’ve complained about it the in the past too 🙈

Will try later today TimelyPenguin76 and report back, thanks! Does this revert the behavior to the 1.3.x one?

Thanks for the reply CostlyOstrich36 !
Does the task read/use the cache_dir directly? It's fine for it to be a cache and then removed from the fileserver; if users want the data to stay they will use the ClearML Dataset 🙂

The S3 solution is bad for us since we have to create a folder for each task (before the task is created), and hope it doesn't get overwritten by the time it executes.

Argument augmentation - say I run my code with python train.py my_config.yaml -e admin.env...

I can indeed.

There's no decorator, just e.g.

def helper(foo: Optional[Any] = None):
    return foo

def step_one(...):
    # stuff

Then the type hints are not removed from helper and the code immediately crashes when being run

We’d be happy if ClearML captures that (since it uses e.g. pip, then we have the git + commit hash for reproducibility), as it claims it would 😅

Any thoughts CostlyOstrich36 ?

Still failing with 1.2.0rc3 😞 AgitatedDove14 any thoughts on your end?

Yes and no SmugDolphin23
The project is listed, but there is no content and it hides my main task that it is attached to.

Then I wonder:

• How to achieve this? The pipeline controller seems to only work with functions, not classes, so running smaller steps remotely seems more difficult then I imagined. I was already prepared to upload artifacts myself etc, but now I’m not sure?
• Do I really need to recreate the pipeline everytime from scratch? Or can I remove/edit steps? It’s mostly used as a… controller for notebook-based executions and experimentations, before the actual pipeline is known. That is, it will ...

Parquet file in this instance (used to be CSV, but that was even larger as everything is stored as a string...)

Yes, that one shows up. I forgot to mention we also set the version explicitly, but that just creates a duplicate dataset under Datasets and anyway our main Task is now hidden from the original project.
So project project exists, but it is empty.

UPDATE: Apparently the quotation type matters for furl ? I switched the ' to \" and it seems to work now

When is the next release expected? 😄

The instance that took a while to terminate (or has taken a while to disappear from the idle workers)

I'll have a look at 1.1.6 then!
And that sounds great - environment variables should be supported everywhere in the config, or then the docs should probably mention where they are and are not supported 🙂

I'll be happy to test it out if there's any commit available?

The thing I don't understand is how come this DOES work on our linux setups 🤔

I have no tried to download the entire file, but yes