AbruptWorm50 - the agent poll the queue, so any free agent can pull tasks. From the graph on the right, it looks like experiments were not waiting in the queue (max experiments is 1, and it was immediately pulled). Can you also check what happens if two experiments are enqueued at the same time?
Looking at the 2nd image you sent, I see that in addition to "services" queue, you also have queues called "training" and "training*_2" - and the experiments you circled is in the "training" queue. In that image - there are no experiments in the services queue.
If you press on the "services" queue (like you did in the first image) you can view the experiments in the queue and the workers. Can you check if there is a situation where there are tasks pending in that queue while one of the work...
AbruptWorm50 - just to make sure there is no misunderstanding - the last image you sent is on the "training" queue and not on the "services" queue. Are there free agents running on that queue?
AbruptWorm50 - does the issue still occur, or did you manage to resolve it?
@<1750327614469312512:profile|CrabbyParrot75> - can you add your workspace id?
JitteryCoyote63 - thanks for the info. We are trying to reproduce this, and will update
@<1681836303299121152:profile|RoundElk14> - thanks for the info.
app.clear.ml is the correct server
app.allegro,ai is an older demo server that should not be used. Do you know where you found the link to that - so that we can remove it?
@<1566596960691949568:profile|UpsetWalrus59> - please note that if you report the plots as two separate series of the same metric - it should work better
SmallBluewhale13 - plot comparing only shows the last iteration, where as the individual info screen displayed the last 5 iterations. Could this explain the issue?
ShinyRabbit94 - The github issue reported by SuperiorPanda77 is a display issue - meaning the files are deleted, and only the "undefined" display is incorrect. This should be fixed in the next release (1.4).
In your case the files are not deleted from the filesystem - so I assume this is a different issue. Can you provide more info so that I can try to reproduce:
How are the artifacts created? Are the links in the popup describing the un-deleted files correct? Can you add a screenshot?
@<1523701790044852224:profile|FreshParrot56> - Indeed a bug. We will fix this in the upcoming version - due in a few weeks
@<1566596960691949568:profile|UpsetWalrus59> - if you could also paste the payload and response of the call to events.get_multi_task_plots
- from the network tab of the browser's dev-tools (F12), this might also help understand
@<1529271098653282304:profile|WorriedRabbit94> - I assume the issue is related to the to autoscaler instances that were running for a long time and produced a lot of logs. Please try the following:
NutritiousBear41 - I would like to ask you to try the following:
Refresh the web page (control+F5). Check the browser's local storage: F12 -> Application/Storage -> Local Storage - and copy the values of "currentVersion" and "dismissedVersion" (if they exist)
NutritiousBear41 - If the problem still occurs - I would like to see i this is a caching issue. Can you please try the following:
Open developer tools (F12) and go to network tab Locate the last call to version.json (this occurs every few minutes so you should see it at the bottom) Right-click and select Edit and Resend On the right side - look at the Request Headers and remove the 2 lines starting with "If-Modified-Since:" and "If-None-Match:" (probably at the bottom) Press send
NutritiousBear41 - I'm currently checking what can cause this. Will get back to you shortly
Do you still get the new version notification?
NutritiousBear41 - can you go to localhost:8080/version.json (or change to the relevant address of your web app) and see what it outputs?
Hi SmallBluewhale13 - can you go to http://<your ip>:8080/version.json
and copy what you get?
Hi CooperativeFox72 . Thanks for reporting this issue.
Can you add the following information:
Is this on the hosted community server ( http://app.community.clear.ml ) or a server that you installed? If this is a server the you installed locally - which server version are you running Which browser, and what resolution are you using?Thanks,
Oren.
Hi JitteryCoyote63 . Just to make sure we fully understand the scenario - you had a project with several experiments, some of which were archived. Then you went to archive and selected all of the experiments and deleted them using the bulk delete operation. And then you saw that the project was empty even when you were not in archive mode. Is that correct?
@<1529271098653282304:profile|WorriedRabbit94> - I will have someone from the team check which experiments are consuming the storage
Hi ThoughtfulGorilla90 , how did you delete the experiments/models - was it from the web app or using the SDK?
Hi @<1570583227918192640:profile|FloppySwallow46> . We've update the rate limits. Can you please check if the issue is still occurring?
@<1558986867771183104:profile|ShakyKangaroo32> - are you running the server using docker-compose?
If so - please add the following to the .env
file in the same directory as the compose:CLEARML_FILES_HOST=http://
<YOUR IP or HOSTNAME>
:8081
Then restart the service by running sudo docker-compose up -d
Please update if this worked, or if you have any questions/issues
@<1564060248291938304:profile|AmusedParrot89> - I see the logic in displaying the last iteration per metric in the compare screen. We will need to think if this won't cause any other issues.
In the mean time - may I ask you to open a github issue - so it will be easier to track?
Thanks DepressedChimpanzee34 . We'll update if/when it is supported.
@<1585078752969232384:profile|FantasticDuck7> - you can set
WEBSERVER__displayedServerUrls={"apiServer":"<apiserver address>","filesServer":"<filesserver addess>","webServer":"<webserver address>"}
This should affect the display of the credentials screen
HandsomeGiraffe70 - please try to update to clearml 1.6.1 - this should solve the issue. Please update us if this worked
@<1574931903440490496:profile|CrookedBear44> - if you logged in with email - you can also login with one of the other options available in the log in page.
To reset your email's password please do the following:
- Go to the login page ( None )
- Press
Continue with Email
- Go to
Log In
tab - Press
Don't remember your password
- Fill your email and press
SEND EMAIL
Please make sure to check your spam folder
Let me know if you need any help.