https://github.com/allegroai/clearml-agent/pull/98 AgitatedDove14 😁

I'm aware, but it would be much cleaner to define them in the worker's clearml.conf and let ClearML expose them locally to running tasks.

EDIT: Also the above is specifically about serving, which is not the target here 🤔 At least not yet 😄

Any simple ways around this for now? @<1523701070390366208:profile|CostlyOstrich36>

So a missing bit of information that I see I forgot to mention, is that we named our packages as foo-mod in pyproject.toml . That hyphen then get’s rewritten as foo_mod.x.y.z-distinfo .

foo-mod @ git+

You can use logger.report_scalar and pass a single value.

And task = Task.init(project_name=conf.get("project_name"), ...) is basically a no-op in remote execution so it does not matter if conf is empty, right?

Yes that's what I thought, thanks for confirming.

BTW AgitatedDove14 following this discussion I ended up doing the regex way myself to sync these, so our code has something like the following. We abuse the object description here to store the desired file path.

` config_path = task.connect_configuration(configuration=config_path, name=config_fname)
included_files = find_included_files_in_source(config_path)
while included_files:
file_to_include = included_files.pop()
sub_config = task.connect_configuration(
configurat...

Now, the original pyhocon does support include statements as you mentioned - https://github.com/chimpler/pyhocon

It misses the repository information of course, but the 'configuration/Args' were logged. So something weird in identifying the repository

Because setting env vars and ensuring they exist on the remote machine during execution etc is more complicated 😁

There are always ways around, I was just wondering what is the expected flow 🙂

` # test_clearml.py
import pytest
import shutil
import clearml

@pytest.fixture
def clearml_task():
clearml.Task.set_offline_mode(True)
task = clearml.Task.init(project_name="test", task_name="test")
yield task
shutil.rmtree(task.get_offline_mode_folder())
clearml.Task.set_offline_mode(False)

class ClearMLTests:
def test_something(self, clearml_task):
assert True run with pytest test_clearml.py `

UPDATE: Apparently the quotation type matters for furl ? I switched the ' to \" and it seems to work now

Since the additional credentials are available to the autoscaler when it boots up (via the config file), I thought it could use those natively?

Does that make sense CostlyOstrich36 ? Any thoughts on how to treat this? For the time being I'm also perfectly happy to include something specific to extra_clearml_conf , but I'm not sure how to set the sdk.aws.s3.credentials to be a list of dictionaries as needed

Would be great if it is 😍 We have few files that change frequently and are quite large in size, and it would be quite a storage hit to save all of them

I also tried setting agent.python_binary: "/usr/bin/python3.8" but it still uses Python 2.7?

TimelyPenguin76 CostlyOstrich36 It seems a lot of manual configurations is required to get the EC2 instances up and running.
Would it not make sense to update the autoscaler (and example script) so that the config.yaml that's used for the autoscaler service is implicitly copied to the EC2 services, and then any extra_clearml_conf are used/overwritten?

I'm not sure I follow, how would that solution look like?

Oh and clearml-agent==1.1.2

I'm running tests with pytest , it consumes/owns the stream

CostlyOstrich36 I'm not sure what is holding it from spinning down. Unfortunately I was not around when this happened. Maybe it was AWS taking a while to terminate, or maybe it was just taking a while to register in the autoscaler.

The logs looked like this:

1. Recognizing an idle worker and spinning down.
   2022-09-19 12:27:33,197 - clearml.auto_scaler - INFO - Spin down instance cloud id 'i-058730639c72f91e1'2. Recognizing a new task is available, but the worker is still idle.
   ` 2022-09...

Is Task.create the way to go here? 🤔

I know, that should indeed be the default behaviour, but at least from my tests the use of --python ... was consistent, whereas for some reason this old virtualenv decided to use python2.7 otherwise 🤨

Will try!

Curious - is there a temporary changelog for 1.2.0? 😁 Always fun to poke at the upcoming features

EDIT: Wait, should the clearml RC be installed outside the venv for the agent as well?

I'd like to remove the hidden system tag from a project

Where do I import this APIClient from AgitatedDove14 ? I meanwhile edited it directly in mongo, but editing a db directly on a Friday is a big nono

Say I have Task A that works with some dataset (which is not hard-coded, but perhaps e.g. self-defined by the task itself).
I'd now like to clone Task A and modify some stuff, but still use the same dataset (no need to recreate it, but since it's not hard-coded, I have to maintain a reference somewhere to the dataset ID).

Since the Dataset SDK offers use_current_task , I would have also expected there to be something like dataset.link(task) or task.register_dataset(ds) ...

Unfortunately not, each task defines and constructs its own dataset. I want cloned task to save that link 🤔

On an unrelated note, when cloning an experiment via the WebUI, shouldn't the cloned experiment have the original experiment as a parent? It seems to be empty