@<1523701070390366208:profile|CostlyOstrich36> I added None btw

2. I guess? 🤔 I mean the same filter option one has for e.g. tags in the table view. In the "all experiments" project I think it would make sense for one to be able to select the projects of interest, or even filter for textual matches.

3. Sorry I meant the cards indeed :)

Managed now 🙂 Thank you for your patience!
I edited the previous post with some suggestions/thoughts

Not sure I understand your comment - why not let the user start with an empty comparison page and add them from "Add Experiment" button as well?

The results from searching in the "Add Experiment" view (can't resize column widths -> can't see project name ...)

Also I can't select any tasks from the dashboard search results 😞

Also (sorry for all of these!) - could be nice to have a direct "task comparison" link in the UI somewhere, that would open a comparison with no tasks and the user can add them manually using the "add experiments" button. :)

I've also followed https://clearml.slack.com/archives/CTK20V944/p1628333126247800 but it did not help

Still failing with 1.2.0rc3 😞 AgitatedDove14 any thoughts on your end?

Oh and clearml-agent==1.1.2

So where should I install the latest clearml version? On the client that's running a task, or on the worker machine?

One way to circumvent this btw would be to also add/use the --python flag for virtualenv

Yes; I tried running it both outside venv and inside a venv. No idea why it uses 2.7?

Any follow up thoughts SuccessfulKoala55 or CostlyOstrich36 ?

I'll have yet another look at both the latest agent RC and at the docker-compose, thanks!

There was no "default" services agent btw, just the queue, I had to launch an agent myself (not sure if it's relevant)

Will try!

Curious - is there a temporary changelog for 1.2.0? 😁 Always fun to poke at the upcoming features

EDIT: Wait, should the clearml RC be installed outside the venv for the agent as well?

https://github.com/allegroai/clearml-agent/pull/98 AgitatedDove14 😁

Same result 😞 This is frustrating, wtf happened :shocked_face_with_exploding_head:
This is also specifically the services queue worker I'm trying to debug 🤔

I also tried switching to dockerized mode now, getting the same issue 🤔

Okay this was a deep dive into clearml-agent code 😁
Took a long time to figure out that there was a specific Python version with a specific virtualenv that was old (Python 3.6.9 and Python 3.8 had latest virtualenv, but Python 3.7.5 had an old virtualenv).
Then the task requested to use Python 3.7, and that old virtualenv version was broken.

As a result -> Could the agent maybe also output the virtualenv version used with setting up the environment for the first time?

AgitatedDove14
I'll make a PR for it now, but the long story is that you have the full log, but the virtualenv version is not logged anywhere (the usual output from virtualenv just says which Python version is used, etc).

I'm using 1.1.6 (upgraded from 1.1.6rc0) - should I try 1.1.7rc0 or smth?

(in the current version, that is, we’d very much like to use them obviously :D)

Any updates @<1523701087100473344:profile|SuccessfulKoala55> ? 🙂

Great to hear @<1523701087100473344:profile|SuccessfulKoala55> ! Is there an estimated timeline for these releases?

Ah, uhhhh whatever is in the helm/glue charts. I think it’s the allegroai/clearml-agent-k8s-base , but since I hadn’t gotten a chance to try it out, it’s hard to say with certainty which would be the best for us 😛

Without knowing anything, I'm assuming maybe ClearML patches plt.title and not Axes.set_title ?

It is

That's up and running and is perfectly fine.

Another side effect btw is that some of our log files (we add a file handler to the logger) end up at 0 bytes. This specifically happens with Ray and ClearML and does not reproduce locally