yes

or I can make comparisons inside some projects but not others

I have 143 GB free in /

are you referring to extra_docker_shell_ scrip t SuccessfulKoala55 ?

that worked, thanks!!

and would it be possible to run it using the normal local agent?

oh right, it will try to use globals from /etc/pip.conf first and then from the virtualenv's pip.conf

Sure, will do 🙂

I can't access the WebAPP nor ssh the server

images?

So what changed?

We changed other bits of code, but not that one..
But maybe we are focusing on the wrong thing, the question now is why is ClearML only detecting these packages (running a different experiment than Diego)

Pillow == 8.0.1
clearml == 0.17.5
google_cloud_storage == 1.40.0
joblib == 0.17.0
numpy == 1.19.5
pandas == 1.3.1
seaborn == 0.11.0
tensorflow_gpu == 2.3.1
tqdm == 4.54.1

we'll create a minimal working example :-)

right, callbacks.py is a file inside the repo, but is not part of the package

but the reason I said the comparison could be an issue is because I'm not being able to do comparisons of experiments

the thing is that this runs before you create the virtual environment, so then in the new environment those settings are no longer there

ok, so ClearML doesn't add all the imported packages needed to run the task to the Installed Packages, only the ones in the main script?

yes, the code is inside a git repository In the main script: from callbacks import function_plot_conf_matrix
and inside callbacks.py of course at the beginning we have from sklearn.metrics import confusion_matrix or something like that

oh I meant now...so after the reboot everything goes back to "normal"..except that I can't make the comparisons

so what we should do is turn pip freeze on in the clearml.conf file?

the main bits are those 🙂

could it be a memory issue triggered by the comparison of 3 experiments?

thanks!!

Worked perfectly, thanks!

can I see an example please?

I should remark that it's been working OK nonstop for 5 months already.. but yesterday and today I'm experiencing theses crashes

exactly 🙂

we are developing a model and I've built a webapp with Streamlit that let's you select the task, and you can see the confusion matrices, splits, data, and predictions on data train/val (all saved in the task), ...and also a model predict function in an image you upload

oh...so is this a bug?

sorry, in my case it's the default mode

that worked, thanks!!