Reputation
Badges 1
46 × Eureka!Yes, we love the HPO app, and are using it :)
Tagging my colleague @<1529271085315395584:profile|AmusedCat74> who needs this with me 🙂
And yes, I was also referring to tasks ran by the Autoscaler (potentially via the HPO) app, too.
This great tool is worth paying for!
It was a debugging session. We haven’t yet tried a “Standard” non-debugging clearml session.
Thanks @<1523703436166565888:profile|DeterminedCrab71> . Yes, I've seen the three options to plot different things. What I'm trying to do is for the "Iterations" plot to have the same plot but just change the X label, not the time series. In matplotlib that would be a call to xlabel
.
From the doc I seemed to find ways to log 2D scatter plots, but not line plots :/ (found)
It also seems simpler to keep the scalar logging structure, but be able to pass a multiplier (reflecting the eval_n_steps
in for example Torch Lightning)
(actually, that might even be feasible without touching the UI, depending how the plot is rendered, but I'll check)
Brilliant, thanks a lot for the answer Jake, much appreciated and clearer!
@<1529271085315395584:profile|AmusedCat74> @<1548115177340145664:profile|HungryHorse70> here we have the answer :)
No problem 🙂 Once you’ve merged it, what do we need to do to get the updated version please?
Thanks @<1523701070390366208:profile|CostlyOstrich36> !
- I hadn’t found the multiple-resources within the same autoscaler. Could you point me to the right place please? Are they all used interexchangeably based upon availability, rather than based on job needs?
- We thought of using separate queues (we do that for CPU vs GPU queues), but having ClearML automatically dispatch to the right based on a job specification would be more flexible. (for example, we could then think to dispath dynami...
@<1523701205467926528:profile|AgitatedDove14> great! (I'm on the Pro version :) ).
Can the “multiple agents on a single queue” scenario, combined with the autoscaler, spawn multiple agents on a single EC2 instance, by chance, please? (thinking e.g. 8 agents on a 8xGPU machine)
OK, so no way to have an automatic dispatch to different, correctly-sized instances, it’s only achievable by submitting to different queues?
Logging scalars also leverages ClearML automatic logging. One problem is that this automatic logging seems to keep its own internal "iteration" counter for each scalar, as opposed to keeping track of, say, the optimizer's number of steps.
That can be simply fixed on clearML python lib by allowing to set a per-scalar iteration-multiplier.
Great, thanks both! I suspect this might need an extra option to be passed via the SDK, to save the iteration scaling at logging time, which the UI can then use at rendering time.
Hi 🙂 Anyone having any idea on that one please? Or could point me in the right place or the right person to find out? Thanks for any help!
Tagging @<1529271085315395584:profile|AmusedCat74> my colleague with whom we ran into this issue.
Thanks @<1523701070390366208:profile|CostlyOstrich36> ! I'll do - and might even peek under the hood see if I can make a PR. What's the best repo for that? Is it that of the ClearML python package?
(do you welcome PRs?)
Thanks. That would be very helpful. Some of our graphs are logged by optimization steps, whereas some by epochs, so having all called "Iterations" is not ideal.
(apologies for delay @<1523701087100473344:profile|SuccessfulKoala55> , we got called into meetings. Really appreciate your reactivity!)
Do Pipelines work with Hyperparameter search, and with single training jobs?
@<1523701087100473344:profile|SuccessfulKoala55> yes I am 🙂 And thanks, looking forward to it!