Jake, the only way that I know to run the agents is to run docker

` docker ps | grep clearml
2635cec202d9 allegroai/clearml:latest "/opt/clearml/wrappe…" 3 days ago Up 3 days 8008/tcp, 8080-8081/tcp, 0.0.0.0:8080->80/tcp, :::8080->80/tcp clearml-webserver
f8d307913fe0 allegroai/clearml:latest "/opt/clearml/wrappe…" 3 days ago Up About a minute 0.0....

I did my best in explanation.
You have buffer of tasks, for example 100. When you add task #101 the task under #1 is replaced with new and you keep now tasks from #2 to #101.
Because I have > 100 saved experiment, I don't think that anyone should bother to change it, unless you are running more than 100 experiments at the same time

Hi, I solved this cut out of labels with
fig.tight_layout() return fig

AgitatedDove14 I think Tim wanted to know what is task_log_buffer_capacity and what functionality it provides

You mean I can do Epoch001/ and Epoch002/ to split them into groups and make 100 limit per group?

Thank you, I will try

Thank you very much it worked! I hope I will never see this kind of bug, will be happy to give more feedback if you would like to find a rootcause

do you think if we manually delete folder /opt/clearml/data/ that would solve this problem same way?

You can see the white-gray mesh on background that shows the end of the image. It is cropped in the middle of labels

ReassuredTiger98 why don't you take 5 minutes time and check out source code? https://github.com/allegroai/clearml/blob/701fca9f395c05324dc6a5d8c61ba20e363190cf/clearml/backend_interface/task/log.py

this is pretty obvious, it replaces last task with new task when the buffer is full

yes, I was using only experiments tab to compare scalars and see validation and train images and I can see that information

doesn't fit in 1 message in slack

Yes, I have latest 1.0.5 version now and it gives same result in UI as previous version that I used

AgitatedDove14 orchestration module - what is this and where can I read more about it?

Shay, you are correct, one of the docker is down. But don't they supposed to run as part of docker /opt/clearml/ docker-compose -f docker-compose.yml up ?

so I run /opt/clearml/ docker-compose -f docker-compose.yml up just to make sure and this are errors that I am seeing

CostlyOstrich36 any ideas?

No, even new started experiment is still creating images with 172.,
` cat ~/clearml.conf

ClearML SDK configuration file

api {
# Notice: 'host' is the api server (default port 8008), not the web server.
api_server:
web_server:
files_server: `

Results > Debug Samples

and more logs 🙂 nice warning about dev server in production

` clearml-apiserver | /usr/local/lib/python3.6/site-packages/elasticsearch/connection/base.py:208: ElasticsearchWarning: Legacy index templates are deprecated in favor of composable templates.
clearml-apiserver | warnings.warn(message, category=ElasticsearchWarning)
clearml-apiserver | [2022-06-09 13:28:03,875] [9] [INFO] [clearml.initialize] [{'mapping': 'events_plot', 'result': {'acknowledged': True}}, {'mapping': 'events_tra...

yes, all runs on same machine on different dockers

When I save plot in jupyter notebook it is not cropped

TimelyPenguin76 Thank you for posting this. I just realized that I changed wrong config. I changed the one on server, but I needed to change the one inside the docker container. Now all works. Thanks for help!

Hi Shay, thanks for reply
I just went by old path remembered in browser. Last week we updated client and server, they are both running on our physical server

No, I was always shutting down server. But if you can give me step by step how to clean install I will be happy to do it

To be more precise - how to clean up and install 🙂

docker has access to all 4 GPUs with --gpus all flag and we specify in config on what cuda device(s) to run, in pytorch we can run more than 2 gpus

I save it to PC and it is not only UI issue. My guess is that it plt.fig is cropped or by SummaryWriter or by trains. Help me debug where is the problem, I will meanwhile try to see what this SummaryWriter does to plt plots

I have restarted docker with docker down several times and nothing changes

SweetBadger76 sorry to tag you but I dont know where to find logs. Do I have elasticsearch logs on my server that I installed the Clearml-server?