I should maybe mention that the security regarding this is low, since this is all behind a private VPN server anyway, I'm mostly just interested in having the credentials used for backtracking purposes

There's code that strips the type hints from the component function, just think it should be applied to the helper functions too :)

It's okay 🙂 I was originally hoping to delete my "initializer" task, but I'll just archive it if someone is interested in the worker data etc. Setting the queue is quite nice.

I think this should get my team excited enough 😄

Will try!

Curious - is there a temporary changelog for 1.2.0? 😁 Always fun to poke at the upcoming features

EDIT: Wait, should the clearml RC be installed outside the venv for the agent as well?

Still happening with 1.4.1 SuccessfulKoala55

Ah it already exists https://github.com/allegroai/clearml-server/issues/134 , so I commented on it

I'm working on the config object references 😉

I see! The Hyper Datasets don't really fit our use case - it seems really focused on CNNs and image-based data, but lacking support for database-oriented tabular data.

So for now we mainly work with parquet and CSV files, and I was hoping there'd be an easy way to view those... I'll make a workaround with a "Datasets" project I suppose!

Sorry for the late reply Jake -- I was away on holidays -- it works perfectly now, thanks!

With clearml-agent 1.5.1

Or if it wasn't clear, that chunk of code is from clearml's dataset.py

It does, but I don't want to guess the json structure (what if ClearML changes it or the folder structure it uses for offline execution?). If I do this, I'm planning a test that's reliant on ClearML implementation of offline mode, which is tangent to the unit test

I think I may have brought this up multiple times in different ways :D
When dealing with long and complicated configurations (whether config objects, yaml, or otherwise), it's often useful to break them down into relevant chunks (think hydra, maybe).

In our case, we have a custom YAML instruction !include , i.e.
` # foo.yaml
bar: baz

bar.yaml

obj: !include foo.yaml
maybe_another_obj: !include foo.yaml `

I realized it might work too, but looking for a more definitive answer 😄 Has no-one attempted this? 🤔

Any updates @<1523701087100473344:profile|SuccessfulKoala55> ? 🙂

Sounds like a nice idea 😁
Follow-up; any ideas how to avoid PEP 517 with the auto scaler? 🤔 Takes a long time to build the wheels

Since the additional credentials are available to the autoscaler when it boots up (via the config file), I thought it could use those natively?

Thanks!

Is there a preferred way to stop the agent?

So caching results for steps with the same arguments is trivial. Ultimately I would say you can combine the task-based pipeline with a function-based pipeline to achieve such dynamic control as you specified in the first two scenarios.

About the third scenario I'm not sure. If the configuration has changed, shouldn't the relevant steps (the ones where the configuration changed and their dependent steps) be rerun?

At any case, I think if you stay away from the decorators, at the cost of a bi...

` # test_clearml.py
import pytest
import shutil
import clearml

@pytest.fixture
def clearml_task():
clearml.Task.set_offline_mode(True)
task = clearml.Task.init(project_name="test", task_name="test")
yield task
shutil.rmtree(task.get_offline_mode_folder())
clearml.Task.set_offline_mode(False)

class ClearMLTests:
def test_something(self, clearml_task):
assert True run with pytest test_clearml.py `

So maybe summarizing (sorry for the spam):

• Pipelines:- Pros: Automatic upload and serialization of input arguments
• Cons: Clutter, does not support classes, cannot inject code, does not recognize environment when run from e.g. IPython- Tasks:- Pros: Tidier and matches original idea, recognizes environment even when run from IPython
• Cons: Does not support classes, cannot inject code, does not automatically upload input arguments

It's removed now, magic of asking for help and doing nothing 😄

I couldn't find it directly in the SDK at least (in the APIClient)... 🤔

But we do use S3

CostlyOstrich36 I'm not sure what is holding it from spinning down. Unfortunately I was not around when this happened. Maybe it was AWS taking a while to terminate, or maybe it was just taking a while to register in the autoscaler.

The logs looked like this:

1. Recognizing an idle worker and spinning down.
   2022-09-19 12:27:33,197 - clearml.auto_scaler - INFO - Spin down instance cloud id 'i-058730639c72f91e1'2. Recognizing a new task is available, but the worker is still idle.
   ` 2022-09...

I also tried adding gent.package_manager.system_site_packages = true to ensure these virtual environments have access btw, still no avail

Literally just as you said it - it happened

Still crashing, I think that may not be the correct virtual environment to edit 🤔
It's the one created later down the line