Ah fantastic, thanks! Another one for me - is there support for custom python models at all? For example, dummy models that simply return the output of an equation run over the dataframe after transforming some of the input columns. Something similar to mlflows custom pyfunc that allows a standard way of interfacing with custom models as you do with keras/sklearn/pytorch models

Ah fantastic, Ill use that over IDEs suggestions from now on 🙂

Oh btw the set_system_tags we talked about previously to auto-archive by adding the archived tag is throwing me an error.

for t in top_exp: t.set_tags(t.get_tags() + ["trial"]) t.set_system_tags(t.get_system_tags() + ["archived"])With error: ValueError: Task object can only be updated if created or in_progress

I think its because the _edit call happens after the set_task_property call?

To go off the online example, it finds the top 3 performing models and prints out the ID. What would be better would be to take those 3 IDs and - in the python code - grab the model artifacts for each, put them into the parent HPO model as its artifacts, and then go through the archive everything. Doesnt solve the issue if a HPO run is going to take a few days (in which the UI would become unusable), but once its done then the auto archiving would clean it up a lot. Is that possible at all, u...

Yeah I was imagining the artifact, id, link to the child task, etc, would all be saved out. I have the HPO experiment open in the UI at the moment, and yup, I can see in the Results>Plots a table summary, but that wasnt the issue, it was trying to clean up the project wide experiments view without making a large number of projects. Are tagging / archiving available in the API for a task? Also, thanks for the help so far 🙂

Ah yup, found it, I was in the server Tasks doco and not the clearml Task doco, oops!

Hey AgitatedDove14 , another question if I can! Im trying to access this information from the API so I can put it as an artifact as well. Currently this is quite a few lines of code using get_top_experiments and get_last_scalar_metrics()[“evaluate”][“mae”][“last”], again I feel like Im missing something as I assume theres a far simpler way of getting data displayed so easily in the UI 🙂

Awesome. If I end up convincing the team to use ClearML, Ill probably have a ton of small requests to streamline the automation side of orchestration, is the team open to PRs from external people?

Yeah, post execution in a separate script / task

Alas no, apologies. Are you saying that in the global_min case, if a trial returns an MAE of 1.3, but the previous trial got an MAE of 0.5, the optimiser gets told that the MAE of the latest model is 0.5 instead of the truth?

Fantastic. Essentially the example provide just prints out ids to the log file, and Im trying to play around with better things to do so that the top models and similar are saved out in some way I can access without manually reading a log file. Maybe reporting a scalar thats a string which has the task id for the top model? Unsure the best way, hence why I was trying to access the optimiser itself which would naturally contain that info

Ah I see, it does print out the top experiments, you jus thave to make sure the metric and what not agrees. If I was looking to just attach some basic information to the task (after its been rerun, instead of printing it to the log), would the best option be to use the Logger to try and attach it, or set parameters, set comment, or is there a general way to set some metadata that is intended to be used in that capacity.

But ideally yes, the HPO should have a df artifact summarising the HPO itself so I can try and make use of the information properly 🙂

How odd, its not in my task.py file!

Let me check…

Ah. The code inspector is jumping to backend_interface.task.py, not clearml.task.py, how embarrassing for me!

Im trying to do it at the end of the optimisation. The same place in the example where you print the ids to the log. Im just hoping theres a way to get said table simply rather than going through a bunch of api calls to construct it myself

Yup, I see. My apologies for not catching that before posting 🙂

Yes, this is expected, because you are trying to log a matplotlib figure as if it was a plotly figure. Try using report_matplotlib_figure

Thanks Martin, this is super useful. Using the get_top_experiments would be great, but do I actually have access to the controller (an_optimizer) from the Task object itself? I dont see anything like an_optimizer = task.connect(an_optimizer) which seems to be the normal way of connecthing things up?

Yup, thats how Ive been doing it now. Will happily update to a simpler method call whenever one gets made. Trying to make use of the HPO is a big thing Im trying to sell the team on, as its what sets ClearML apart from MLFlow or neptune - useful task orchestration and cloning 🙂

I meant the object instance itself so we can see what class it is and if its initialised. Also that looks like a matplotlib plot, not a plotly plot which you logging call indicates

Compared to the settings I can see when I go through the online portal and use the free hosted version:

The lack of this info might be to do with some API issues shown in the web dev console:

Yeah its trying to plan down the line into model deployment. Whilst its easy to save out a keras SavedModel or similar and have that artifact uploaded into the store, just wanted to check if there was a more generic solution. I could just create a Python class and serialise that out such that it has a standard interface, but good to check. So for example, some artifact representing an arbitrary math function. For better context, the idea is to make deploying any artifact we upload using clear...

Gorgeous!

+1 for autoarchiving. Right now the interface feels incredibly clunky to use once the number of HPO trials starts to increase. I currently have a demo project and have different algos to make predictions (a simple keras model, a RF, etc). Ideally I’d want to see the HPO execution just once with all the trials underneath it, or just the top (few) models. At the moment, I have pages and pages of models, 99% of them I dont care about. Is it possible to archive models and set tags in the code rat...

So Im guessing either return_fig is False when you ran this, or plt.cdf is having issues (which it can). Id always recommend explicitly creating the figure instead of using the plt shortcuts.

fix, ax = plt.subplots()
ax.plot(…)

etc

Ah okay.Probably better than the Logger.report_text I was going to use to dump some json into, but I see a dict gets stores as json in upload_artifact as well. Perfect!

https://github.com/allegroai/clearml/blob/master/examples/optimization/hyper-parameter-optimization/hyper_parameter_optimizer.py#L128

the top models in the example arent saved out in a useful way, just printed out. Im trying to figure out th ebest way of saving these IDs so I can get the tasks/models

Heres the landing page, now with no option for settings or sidebar navigation:

Okay, that makes sense then. Whats still got me scratching my head is the examples printing out the WOOT WOOT for breaking the record, despite it clearly not. Hmm.