Hi JealousSealion33 . We are planning on releasing a clearml k8s package very soon. I hope this is something that you will be able to use. If not, could you please elaborate on the issue you are facing and share the error you are receiving?
HandsomeGiraffe70 - please try to update to clearml 1.6.1 - this should solve the issue. Please update us if this worked
@<1566596960691949568:profile|UpsetWalrus59> - if you could also paste the payload and response of the call to events.get_multi_task_plots
- from the network tab of the browser's dev-tools (F12), this might also help understand
@<1566596960691949568:profile|UpsetWalrus59> - please note that if you report the plots as two separate series of the same metric - it should work better
Oh, OK - I will check that. Thanks!
AbruptWorm50 - does the issue still occur, or did you manage to resolve it?
Hi @<1570583227918192640:profile|FloppySwallow46> . We've update the rate limits. Can you please check if the issue is still occurring?
We limit the allowed calls per IP - to make sure the server is not blocked accidentally. We enabled over 1000 calls per minute.
@<1574931903440490496:profile|CrookedBear44> - if you logged in with email - you can also login with one of the other options available in the log in page.
To reset your email's password please do the following:
- Go to the login page ( None )
- Press
Continue with Email
- Go to
Log In
tab - Press
Don't remember your password
- Fill your email and press
SEND EMAIL
Please make sure to check your spam folder
Let me know if you need any help.
Make sure restart the compose ( docker-compose up -d
)
Hi @<1526371986278715392:profile|VivaciousReindeer64> ,
I replicated your environment, and found the following solution:
- In the docker-compose, add the following to the webserver section:
volumes:
- /opt/clearml/config:/mnt/external_files/configs
- In the host machine, create a file
/opt/clearml/config/configuration.json
containing the following:
{
"displayedServerUrls": {
"filesServer": "
"
}
}
- restart the docker-compose: `sudo docker-...
@<1526371986278715392:profile|VivaciousReindeer64> - please check the following:
- What do you get if you go to http//192.168.1.145:8080/configuration.json?
- Can you check the log of the webserver docker (using
sudo docker logs clearml-webserver
) - especially the beginning? Does it say anything about the fileBaseUrl?
@<1526371986278715392:profile|VivaciousReindeer64> - I added this and it worked:
- WEBSERVER__fileBaseUrl="
"
Maybe the quotes are required.
@<1526371986278715392:profile|VivaciousReindeer64> - I'll check it on my env
VivaciousReindeer64 - you can try to add to the following to the webserver service environment section:
WEBSERVER__fileBaseUrl= https://192.168.1.145:8081
@<1526371986278715392:profile|VivaciousReindeer64> - Yes - please send me the docker-compose and the log of the webserver
Hi @<1526371986278715392:profile|VivaciousReindeer64> - please try this file:
you can re-direct it to a file or just copy with the clipboard
sudo docker logs clearml-webserver
SmallBluewhale13 - plot comparing only shows the last iteration, where as the individual info screen displayed the last 5 iterations. Could this explain the issue?
EnviousStarfish54 - Got it!
I'll look into it and update you. Thanks for reporting!
Hi CooperativeFox72 . Thanks for reporting this issue.
Can you add the following information:
Is this on the hosted community server ( http://app.community.clear.ml ) or a server that you installed? If this is a server the you installed locally - which server version are you running Which browser, and what resolution are you using?Thanks,
Oren.
DrainedHippopotamus42 - once you send more metrics, the calculation will be correct. The problem only occurs when you delete all the metrics.
Great! Thanks for the update
AbruptWorm50 - the agent poll the queue, so any free agent can pull tasks. From the graph on the right, it looks like experiments were not waiting in the queue (max experiments is 1, and it was immediately pulled). Can you also check what happens if two experiments are enqueued at the same time?
AbruptWorm50 - just to make sure there is no misunderstanding - the last image you sent is on the "training" queue and not on the "services" queue. Are there free agents running on that queue?
@<1681836303299121152:profile|RoundElk14> - thanks for the info.
app.clear.ml is the correct server
app.allegro,ai is an older demo server that should not be used. Do you know where you found the link to that - so that we can remove it?
Yes - I found it. Will see how to fix that. Thanks again.
Let us know if you run into any other issues
@<1681836303299121152:profile|RoundElk14> - I'm happy that you managed to resolve the issue. We are trying to reproduce it, but so far didn't manage to do so. As far as I understand - signed up to an account with google, and later tried to login with email - is that correct?
@<1558986867771183104:profile|ShakyKangaroo32> - are you running the server using docker-compose?
If so - please add the following to the .env
file in the same directory as the compose:CLEARML_FILES_HOST=http://
<YOUR IP or HOSTNAME>
:8081
Then restart the service by running sudo docker-compose up -d
Please update if this worked, or if you have any questions/issues