but we run everything in docker containers. Will it still help?

exactly what I’m talking about

The only thing I found is that I need to run flake8, but it fails even without any changes, i.e. it was not enforced before (see my msg in )

clearml.utilities.pigar.main.GenerateReqs.extract_reqs

no, I’m providing the id of task which generated the model as a “hyperparam”

do you want a fully reproducible example or just 2 scripts to illustrate?

“assuming the “catboost_train.py” is in the working directory” - maybe I get this part wrong?

So maybe the path is related to the fact I have venv caching on?

if you import a local package from a different local folder, and that folder is Not in the same repo

need to check with infra engineers

You have two options
I think both can work but too much of a hassle. I think I’ll skip extracting the common code and keep it duplicated for now

“To have the Full pip freeze as “installed packages” - that’s exactly what I’m trying to prevent. Locally my virtualenv has all the dependencies for all the clearml tasks, which is fine because I don’t need to download and install them every time I launch a task. But remotely I want to keep the bare minimum needed for the concrete task. Which clearml successfully does, as long as I don’t import any local modules.

and my problem occurred right after I tried to delete ~1.5K tasks from a single subproject

So I thought, maybe I can tell clearml-session to use hostname from ngrok

Did a small update: added a workaround and renamed the issue to include more client_facing condition
limit_execution_time is presentinstead of an implementation detail condition
timeout_jobs are present

For me - workaround is totally acceptable, thus scheduler is once again usable for me.

https://github.com/martjushev/clearml_requirements_demo

no new unremovable entries have appeared (although I haven’t tried)

looking into the output folder of catboost, I see 3 types of metrics outputs:
tfevents (can be read by tensorboard) catboost_training.json (custom (?) format). Is read here to be shown as an ipython widget: https://github.com/catboost/catboost/blob/c2a6ed0cb85869a73a13d08bf8df8d17320f8215/catboost/python-package/catboost/widget/ipythonwidget.py#L93 learn_error.tsv, test_error.tsv, time_left.tsv which have the same data as json. Apparently they are to be used with this stale metrics viewer pr...

https://github.com/mlflow/mlflow/pull/2417

and this can break a lot of things, when somebody start the scheduler with an older version of clearml, saves the state, then upgrades and new clearml expects the state in another format

Wanted to check if MLFlow supports catboost. Apparently, it does. Pull request was merged 16 hours ago. Nice timing 😃

I guess this is the one https://catboost.ai/docs/concepts/python-reference_catboostipythonwidget.html

https://github.com/catboost/catboost/search?q=tensorboard

as I understand, it uses tensorboard from C++ code

Although it is only for model tracking, autologging is yet to be implemented there

slightly related follow-up question: can I add user properties to a scheduler configuration?

or somehow, we can centralize the storage of S3 credentials (i.e. on clearml-server) so that clients can access s3 through the server

this does not prevent from enqueuing and running new tasks, rather an eyesore

found the code: https://github.com/ipython/ipykernel/blob/master/ipykernel_launcher.py