For example I have a DATA_DIR environment variable which points to the directory where disk-data is stored

Depending on where the agent is, the value of DATA_DIR might change

Oh I get it, that also makes sense with the docs directing this at inference jobs and avoiding GPU - because of the 1-N thing

cluster.routing.allocation.disk.watermark.low:

it is a model name

I manually deleted the allegroai/trains:latest image, that didn't help either

a machine that had previous installation, but I deleted the /opt/trains directory beforehand

CostlyOstrich36 ⬆

Still happens

💪

the explicity of the variable queue makes it feels like it runs somewhere else

this?

Sorry I meant this link
https://azuremarketplace.microsoft.com/en-us/marketplace/apps/apps-4-rent.clearml-on-centos8

I think a good idea is to add to the error message when the clearml agent fails due to import error, a suggestion ot try out with pip freeze

sudo curl https://raw.githubusercontent.com/allegroai/trains-server/master/docker-compose.yml -o /opt/trains/docker-compose.yml

Can you lend a few a words about how the not-pip freeze mechanism of detecting packages work?

but shouldn't the :lastest make it redownload the right image?

this is the selection from the column setting menu

But does it disable the agent? or will the tasks still wait for the agent to dequeue?

cool, didn't know about the PAT

Switching to a VPN solves this, just saying 😕

One sec I'll paste the relevant pieces of code

👍

I re-executed the experiemnt, nothing changes

Hi guys, just updated the issue - seems like the new release did fix the color scale, but I notice some data points are missing (the plot is missing data!)

see my comment on the issue
https://github.com/allegroai/clearml/issues/373#issuecomment-894756446

checking and will let you know