AgitatedDove14 for example let’s add to https://github.com/allegroai/clearml/blob/master/examples/frameworks/catboost/catboost_example.py second catboost model training:... catboost_model = CatBoostRegressor(iterations=iterations, verbose=False) catboost_model2 = CatBoostRegressor(iterations=iterations+200, verbose=False) ... catboost_model.fit(train_pool, eval_set=test_pool, verbose=True, plot=False, save_snapshot=True) catboost_model2.fit(train_pool, eval_set=test_pool, verbose=True, plot=False, save_snapshot=True) ...
as result, we have just one learn and one validation plots, but we want to get two. How can I get plots for both models?
AgitatedDove14 hm, I don’t know what is the right expected behaviour, I’ve expected 2 plots. If my assumption looks right, should I make an issue on github?
` import numpy as np
import pandas as pd
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split
from lightautoml.tasks import Task
from lightautoml.automl.presets.tabular_presets import TabularAutoML
import clearml
cml_task = clearml.Task.get_task(clearml.config.get_remote_task_id())
logger = cml_task.get_logger()
data = pd.read_csv("./examples/data/sampled_app_train.csv")
....
automl = TabularAutoML(task=Task('binary'))
cml_task.connect(automl)
oof_pred = automl.fit_predict(train_data, roles={"target": "TARGET"})
logger.report_single_value("ROCAUC test", roc_auc_score(test_data["TARGET"].values, test_pred.data[:, 0]))
logger.flush() `
TabularAutoML is a custom class that uses some popular frameworks deep inside
https://github.com/sb-ai-lab/LightAutoML
EnviousPanda91 please feel free to PR if it works 🙂
https://github.com/allegroai/clearml/blob/86586fbf35d6bdfbf96b6ee3e0068eac3e6c0979/clearml/binding/frameworks/catboost_bind.py#L114
Oh, so is it a bug and you should have seen two series on each graph? (I think it is... not sure how to actually name the second instance other than running number)
EnviousPanda91 , which framework isn't being logged? Can you provide a small code snippet?
EnviousPanda91 'connect' will log the object properties, the automagic logging is controlled in the Task.init call. Specifically Which framework produces metrics that are not logged? Your sample code manually reports some scalars/values, do you these as well?
EnviousPanda91 so which frame works are being missed? Is it a request to support new framework or are you saying there is a bug somewhere?
AgitatedDove14 no, it’s not a request.
I have custom python class, that uses a lot of models from frameworks that supported by ClearML already. I want to enable auto reporting for all models by using command clearml_task.connect(my_custom_class_instance) , but it doesn’t work the way I need it to — there is the only one loss curve, because because this graph is redrawn every time a new instance starts training.
Is there any way to reporting all instances inside my custom class without modifying this class?
AgitatedDove14 done) btw, could you show me the place in the code where scalars are written? I want to make a hotfix