
Reputation
Badges 1
195 × Eureka!AgitatedDove14 I believe that worker status doesn't is not considered a metric?
imagine the beautiful PR showing such a feature š
changing the queue order cool, but a bit too limited.. I have 30 jobs I want to multi select and push up to first priority.. this is a lot of manual labor..
The scalars page provides a metric hide/show control - Is this the one you mean? The debug images page also provides a filter by metric - Depending on your naming policy this can easily be used to focus on more sparsely appearing images.
well, it does in a sense, but the poor version. while in the scalars you can browse over full names or headers (full_name= "header / description" format) both from a list or by regexp, in the debug images there is only the header filtering and only from a...
FrothyDog40 Hi, there is a sort of a UI bug related to the mentioned above..
if you choose a single experiment then use filter or sorting such that it is not displayed anymore you can't deselect it.
you have this - sign in the top selection box (select all \ deselect all) but it doesn't do anything.
the workaround is to select an additional experiment and then when the multiple experiment bar pops up choose show experiments selected and deselect them and go back.
its inconvenient, both in th...
Bonus 2: having a way to push experiments up in the queue
AgitatedDove14 Thanks, I am aware of the auto_connect_frameworks and it makes sense for torch.save (and probably other stuff) however if it is called about 10 times during model initialization it seem excessive and I would be happy to avoid this behavior during model initialization if possible. do you know what is actually triggering the uploads?
I'll try to create a minimal example later today if the above is not sufficient
That's good enough for me, I forgot about the all projects option
some arbitrary job.. can be executed in parallel to a running job on the same agent.
directly from the UI from the services queue?
thanks AgitatedDove14 , Not that I know I didn't do any special setup other than the straightforward pip install hydra-core --pre
sounds like an ok option, any recommended reference for how to start?
sounds great, is it part of the release? or latest repo?
CostlyOstrich36 ,I went through tasks and session and couldn't find an equivalent ping
CostlyOstrich36 , I don't think we are talking about the same things..
if I remove the import and use the path I get this:raise MissingConfigException(hydra.errors.MissingConfigException: Cannot find primary config 'config'. Check that it's in your config search path.
parameters = { 'float': 2.2, 'string': 'my string', }
for example.. can be anything really, some dictionary
AgitatedDove14 after a small test run we done, we get some issues with the proposed approach
I think it works as far as what you mentioned is concerned, there is a dynamic link in the debug samples that downloads the html file when we try to view it, and it is updated according to the notebook
However we access our server from a limited networking machine that can only access AWS ips, and the HTML file is not a standalone and has some dependencies that require networking.. so can only be vi...
FrothyDog40 ,ClearML tries to conserve storage by limiting the history length for debug images (seeĀ sdk.metrics.file_history_sizeĀ configuration entry), though the history can indeed grow large by setting a large value or using a metric/variant naming scheme to circumvent this limit.
I am aware, but I am not sure what did you try to say by thisDoes your use case call for accessing a specific iteration for all images or when looking at a specific image?
Yes, for some cases I would like...
TimelyPenguin76 , I generate it manually , clone some task -> adjust config -> enqueue
then when the agent pulls it I get the following behaviorremote_dict = task.connect(local_dict) # matching keys are overridden from remote config
which I can't reproduce as I described above
kind of on the same topic, it would be very useful if some kind of verbosity will be enabled.. some kind of progress bar for get_top_experiments()
if preferred I can open a Github issue about this
but I don't get the same when I try to reproduce it for debug as described above (with task.connect)
AgitatedDove14 ,maybe worth updating the main Readme.md in the github.. if someone try to follow the instructions there it breaks
Another option is to pull Tasks from a dedicated queue and use the LocalClearMLJob to spwan them
This sounds like it can work. we are talking about something like:
` #<Machine 1>
#Init Optimizer with some dedicated queue
<Machine 2>
heavy one time Common Initialization
while True:
# sample queue
# enqueue with LocalClearMLJob
# Execute Something
# report results
<Machine i>
heavy one time Common Initialization
while True:
# sample same queue
# enqueue wi...