AbruptWorm50 - the agent poll the queue, so any free agent can pull tasks. From the graph on the right, it looks like experiments were not waiting in the queue (max experiments is 1, and it was immediately pulled). Can you also check what happens if two experiments are enqueued at the same time?

EnviousStarfish54 - Got it!
I'll look into it and update you. Thanks for reporting!

EnviousStarfish54 - we identified the issue, and will fix it for the next release

Looking at the 2nd image you sent, I see that in addition to "services" queue, you also have queues called "training" and "training*_2" - and the experiments you circled is in the "training" queue. In that image - there are no experiments in the services queue.
If you press on the "services" queue (like you did in the first image) you can view the experiments in the queue and the workers. Can you check if there is a situation where there are tasks pending in that queue while one of the work...

sudo docker logs clearml-webserver

@<1526371986278715392:profile|VivaciousReindeer64> - Yes - please send me the docker-compose and the log of the webserver

Make sure restart the compose ( docker-compose up -d )

NutritiousBear41 - I'm currently checking what can cause this. Will get back to you shortly

AbruptWorm50 - does the issue still occur, or did you manage to resolve it?

VivaciousReindeer64 - you can try to add to the following to the webserver service environment section:
WEBSERVER__fileBaseUrl= https://192.168.1.145:8081

No problem. Glad it's working now

Hi EnviousStarfish54 .
I'm trying to make sure I understand the scenario. What I undestood is that you add a custom column (metric) to the experiments table, sort by it and then refresh with F5. I wasn't able to reproduce this on the Demo site ( https://demoapp.trains.allegro.ai/projects/*/experiments?columns=selected&columns=type&columns=name&columns=tags&columns=status&columns=project.name&columns=users&columns=started&columns=last_update&columns=last_iteration&columns=m.5451af93e0bf68a4ab...

Hi SmallBluewhale13 - can you go to http://<your ip>:8080/version.json and copy what you get?

SmallBluewhale13 - plot comparing only shows the last iteration, where as the individual info screen displayed the last 5 iterations. Could this explain the issue?

NutritiousBear41 - can you go to localhost:8080/version.json (or change to the relevant address of your web app) and see what it outputs?

NutritiousBear41 - If the problem still occurs - I would like to see i this is a caching issue. Can you please try the following:
Open developer tools (F12) and go to network tab Locate the last call to version.json (this occurs every few minutes so you should see it at the bottom) Right-click and select Edit and Resend On the right side - look at the Request Headers and remove the 2 lines starting with "If-Modified-Since:" and "If-None-Match:" (probably at the bottom) Press send

ShinyRabbit94 - The github issue reported by SuperiorPanda77 is a display issue - meaning the files are deleted, and only the "undefined" display is incorrect. This should be fixed in the next release (1.4).
In your case the files are not deleted from the filesystem - so I assume this is a different issue. Can you provide more info so that I can try to reproduce:
How are the artifacts created? Are the links in the popup describing the un-deleted files correct? Can you add a screenshot?

MotionlessMonkey27 - was this resolved?

Hi ThoughtfulGorilla90 - when did you perform the deletion. It might take up to 24 hours for the application to get the updated size

@<1523701087100473344:profile|SuccessfulKoala55> - yes, plots are reported every iteration.
@<1564060248291938304:profile|AmusedParrot89> - the plot comparison indeed compares the latest iteration of the experiments. I will see if this can be better indicated somewhere

@<1564060248291938304:profile|AmusedParrot89> - let me check this and get back to you

The artifacts (debug samples, models, artifacts) deletion is currently performed done from the browser, after the task object is deleted from the backend. So it might take time for all the artifacts to be deleted - please make sure not to close your browser.
Note that only artifacts saved on fileserver are deleted - exteranal artifacts (in the cloud or on a local filesystem) are not deleted.
We are working on moving the deletion to the server side, to avoid these kind of issues. Should be in ...

Hi ThoughtfulGorilla90 , how did you delete the experiments/models - was it from the web app or using the SDK?

Hi DepressedChimpanzee34 .
Currently supporting plotly animations is not planned for the upcoming versions. I would suggest opening a feature request in github, or add it to the UI change request thread: https://github.com/allegroai/clearml/issues/81 .
Thanks,
Oren.

Hi @<1570583227918192640:profile|FloppySwallow46> . We've update the rate limits. Can you please check if the issue is still occurring?

SuperiorPanda77 - thanks for updating. So indeed these may be similar issues. I will re-check this and udpate

We limit the allowed calls per IP - to make sure the server is not blocked accidentally. We enabled over 1000 calls per minute.

Thanks DepressedChimpanzee34 . We'll update if/when it is supported.

NastySeahorse61 - can you check if this was updated? What does you usage page say?

@<1523701790044852224:profile|FreshParrot56> - Indeed a bug. We will fix this in the upcoming version - due in a few weeks