I can scroll sideways but if I open any of the comparison items, I pretty much can only see one experiment's values

I guess it's mixed. If #340 is resolved, then this initializer task will be a no-op: detach, and init-close new tasks as needed.

The instance that took a while to terminate (or has taken a while to disappear from the idle workers)

SuccessfulKoala55 could this be related to the monkey patching for logging platform? We have our own logging handlers that we use in this case

What do you mean 😄 Using logging.config.dictConfig(...)

What's new in 1.1.6rc0?

If that's the case, wouldn't it apply across the board? This happens in a single task within ray - the other tasks (I have many in a single run) are fine

Or do you mean the contents of the configuration, probably :face_palm: ... one moment

I'll try upgrading to 1.1.5, one moment

Ah it is.

I thought so too - so I added flush calls just in case, but nothing's changed.
This is somewhat weird since it always happens in the above scenario (Ray + ClearML), and always in the last task/job from Ray

We just inherit from logging.Handler and use that in our logging.config.dictConfig ; weird thing is that it still logs most of the tasks, just not the last one?

I'll try with 1.1.5 first, then 1.1.6rc0

Another side effect btw is that some of our log files (we add a file handler to the logger) end up at 0 bytes. This specifically happens with Ray and ClearML and does not reproduce locally

1.1.4

TimelyPenguin76 I added pip install --update clearml-agent to the extra_vm_bash_script for the autoscaler, that should at least guarantee the latest clearml agent is used on the instance, right?

Odd; switching to virtual environment results in
fatal: could not read Username for ' ': terminal prompts disabledeven though it does earlier show that:
agent.git_user = xxx

That's enabled; I was aiming if there are flags to add to pip install CLI, such as --no-use-pep517

I can indeed.

I'm guessing that's not on pypi yet?

I'm trying, let's see; our infra person is away on holidays :X Thanks! Uh, which configuration exactly would you like to see? We're running using the helm charts on K8s, so I don't think I have direct access to the agent configuration/update it separately?

Nope, no .netrc defined anywhere, really (+I've abandoned the use of docker for the autoscaler as it complicates things, at least for now)

Sounds like a nice idea 😁
Follow-up; any ideas how to avoid PEP 517 with the auto scaler? 🤔 Takes a long time to build the wheels

That was a good idea, unfortunately did not help too much, but I think I may have a found a work around, thanks!

I'm using some old agent I fear, since our infra person decided to use chart 3.3.0 😕
I'll try with the env var too. Do you personally recommend docker over the simple AMI + virtual environment?

More complete log does not add much information -
Cloning into '/root/.clearml/venvs-builds/3.10/task_repository/xxx/xxx'... fatal: could not read Username for ' ': terminal prompts disabled fatal: clone of ' ` ' into submodule path '/root/.clearml/venvs-builds/3.10/task_repository/...

Then the username and password would be visible in the autoscaler task 😕
But it should work out of the box, as it does work like that out of the box also regardless of ClearML. The user and personal access token are used as is and it propagates down to submodules, since those are simply another git repository.
I've further checks on a different machine and it works as well 🤔

We have a read-only user with personal access token for these things, works seamlessly throughout and in our current on premise servers... So perhaps something missing in the autoscaler definitions?

Hurrah! Added
git config --system credential.helper 'store --file /root/.git-credentials' to the extra_vm_bash_script and now it works
(logs the given git credentials in the store file, which can then be used immediately for the recursive calls)

Different AMI image/installing older Python instances that don't enforce this...
For future reference though, the environment variable should be PIP_USE_PEP517=false

I just set the git credentials in the clearml.conf and it works out of the box