-ish, still debugging some weird stuff. Sometimes ClearML picks ip and sometimes ip2 , and I can't tell why 🤔

FWIW running clearml ==1.9.1 with WebApp: 1.9.2-317 • Server: 1.9.2-317 • API: 2.23

I thought so too - so I added flush calls just in case, but nothing's changed.
This is somewhat weird since it always happens in the above scenario (Ray + ClearML), and always in the last task/job from Ray

I'll try that in a bit (that requires some access control changes). Any idea how can I modify the dynamically created virtualenv?

` Poetry Enabled: Ignoring requested python packages, using repository poetry lock file!
The currently activated Python version 3.10.6 is not supported by the project (~3.8.0).
Trying to find and use a compatible version.
Using python3.8 (3.8.16)
Creating virtualenv ... in /root/.clearml/venvs-builds/3.10/task_repository/...git/.venv
Installing dependencies from ...

AgitatedDove14 yeah I see this now; this was an issue because I later had to "disconnect" the remote task, so it can, itself, create new tasks (using clearml.config.remote.override_current_task_id(None) ). I guess you might remember that discussion? 😁

EDIT: It's the discussion we had here, for reference. https://clearml.slack.com/archives/CTK20V944/p1640955599257500?thread_ts=1640867211.238900&cid=CTK20V944
So probably not needed in JitteryCoyote63 's case, we still have some...

That gives us the benefit of creating "local datasets" (confined to the scope of the project, do not appear in Datasets tabs, but appear as normal tasks within the project)

You mean the host is considered the bucket, as I wrote in my earlier message as the root cause?

This could be relevant SuccessfulKoala55 ; might entail some serious bug in ClearML multiprocessing too - https://stackoverflow.com/questions/45665991/multiprocessing-returns-too-many-open-files-but-using-with-as-fixes-it-wh

Ah, probably https://github.com/allegroai/clearml/pull/534

Will try this out.

SuccessfulKoala55 no that did not solve the issue 😞

Thanks AgitatedDove14 , I'll give it a try. Perhaps additional documentation is needed for that extra_layout

And agent too, I hope..?

I'd be happy to join a #releases channel just for these!

Just randomly decided to check and saw there's a server 1.4 ready 🎉

I believe it is maybe a race condition that's tangent to clearml now...

AFAIK that's the only way right now (see my comment here - https://clearml.slack.com/archives/CTK20V944/p1657720159903739?thread_ts=1657699287.630779&cid=CTK20V944 )
Or then if you have the ClearML paid service, I believe there is a "vaults" service, right AgitatedDove14 ?

Eek. Is there a way to merge a backup from elastic to current running server?

The network is configured correctly 🙂 But the newly spun up instances need to be set to the same VPC/Subnet somehow

It does not 🙂
We started discussing it here - https://clearml.slack.com/archives/CTK20V944/p1640955599257500?thread_ts=1640867211.238900&cid=CTK20V944
You suggested this solution - https://clearml.slack.com/archives/CTK20V944/p1640973263261400?thread_ts=1640867211.238900&cid=CTK20V944
And I eventually found this solution to work - https://clearml.slack.com/archives/CTK20V944/p1641034236266500?thread_ts=1640867211.238900&cid=CTK20V944

This is related to my other thread, so I’ll provide an example there -->

Maybe it's the missing .bashrc file actually. I'll look into it.

Any thoughts SuccessfulKoala55 ?

For now we've monkey-patched it to our usecase:

` Dataset._Dataset__hidden_tag = "active"

    def foo(cls, dataset_project, dataset_name):
        dataset_project = dataset_project or "Datasets"
        return dataset_project, dataset_project.rpartition("/")[0]

    Dataset._build_hidden_project_name = foo `

One last MinIO-related question (sorry for the long thread!)
While I do have the access and secret defined in clearml.conf, and even in the WebUI, I still get similar warnings as David does here - https://clearml.slack.com/archives/CTK20V944/p1640135359125200

Thanks David! I appreciate that, it would be very nice to have a consistent pattern in this!

Still; anyone? 🥹 @<1523701070390366208:profile|CostlyOstrich36> @<1523701205467926528:profile|AgitatedDove14>

What's new in 1.1.6rc0?

Thanks! I'll wait for the release note/docs update 😁

Would it also be loaded on e.g. Task.running_locally() ?

Here's an example where poetry.lock is removed, and still the console reads:
url: .... branch: HEAD commit: 22fffaf8d5f377b7f10140e642a7f6f26b72ffaa root: /.../.clearml/venvs-builds/3.10/task_repository/... Applying uncommitted changes Poetry Enabled: Ignoring requested python packages, using repository poetry lock file! Creating virtualenv ds-platform in /.../.clearml/venvs-builds/3.10/task_repository/.../.venv Updating dependencies Resolving dependencies...

I'll try a hacky-way around it with sed -i 's/include-system-site-packages = false/include-system-site-packages = true/g' clearml_agent_venv/pyvenv.cfg and report back.

Will try later today TimelyPenguin76 and report back, thanks! Does this revert the behavior to the 1.3.x one?