so I run /opt/clearml/ docker-compose -f docker-compose.yml up just to make sure and this are errors that I am seeing

Thank you. I've changed clearml.conf, but url are remain with old ip. Do I need to restart ClearML or run any command to apply config changes?

TimelyPenguin76 Thank you for posting this. I just realized that I changed wrong config. I changed the one on server, but I needed to change the one inside the docker container. Now all works. Thanks for help!

You mean I can do Epoch001/ and Epoch002/ to split them into groups and make 100 limit per group?

Thank you, I will try

I have restarted docker with docker down several times and nothing changes

now it is empty and I don't know where to find credentianl to connect one more docker client

To be more precise - how to clean up and install 🙂

and more logs 🙂 nice warning about dev server in production

` clearml-apiserver | /usr/local/lib/python3.6/site-packages/elasticsearch/connection/base.py:208: ElasticsearchWarning: Legacy index templates are deprecated in favor of composable templates.
clearml-apiserver | warnings.warn(message, category=ElasticsearchWarning)
clearml-apiserver | [2022-06-09 13:28:03,875] [9] [INFO] [clearml.initialize] [{'mapping': 'events_plot', 'result': {'acknowledged': True}}, {'mapping': 'events_tra...

all dockers seems to be up but running experiment fails to communicate with API server

` clearml-apiserver | [2022-06-09 13:27:33,737] [9] [INFO] [clearml.app_sequence] ################ API Server initializing #####################
clearml-apiserver | [2022-06-09 13:27:33,737] [9] [INFO] [clearml.database] Initializing database connections
clearml-apiserver | [2022-06-09 13:27:33,737] [9] [INFO] [clearml.database] Using override mongodb host mongo
clearml-apiserver | [2022-06-09 13:27:33,737] [9] [INFO] [clearml.database] Using override mongodb port 27017
clearml-apiserver | [2...

CostlyOstrich36 any ideas?

` Retrying (Retry(total=239, connect=240, read=239, redirect=240, status=240)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))': /auth.login
Retrying (Retry(total=238, connect=240, read=238, redirect=240, status=240)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))': /auth.login
Retrying (Retry(total=237, connect=240, read=237, redirect=240, status=24...

Hi, I solved this cut out of labels with
fig.tight_layout() return fig

Results > Debug Samples

curl: (7) Failed to connect to localhost port 9200: Connection refused

Hi Shay, thanks for reply
I just went by old path remembered in browser. Last week we updated client and server, they are both running on our physical server

yes, I was using only experiments tab to compare scalars and see validation and train images and I can see that information

ReassuredTiger98 why don't you take 5 minutes time and check out source code? https://github.com/allegroai/clearml/blob/701fca9f395c05324dc6a5d8c61ba20e363190cf/clearml/backend_interface/task/log.py

this is pretty obvious, it replaces last task with new task when the buffer is full

AppetizingMouse58 all is Linux. Or idea was to run docker on same server to initiate tasks from UI but it was taking to much time so we give up and still do "python train.py experiment=myexpname"

Thank you very much it worked! I hope I will never see this kind of bug, will be happy to give more feedback if you would like to find a rootcause

Hi David, where can I get these logs?

AgitatedDove14 I think Tim wanted to know what is task_log_buffer_capacity and what functionality it provides

here are requirements from the repository that I was able to run hydra_example.py and that I have crash with my custom train.py

Previously I had general tab in Hyper Parameters, but now without this line I don't have it.

Ok, let me check it later today and come back with the results of the example app

AgitatedDove14 orchestration module - what is this and where can I read more about it?

Python 3.8.8 (default, Feb 24 2021, 21:46:12)
[GCC 7.3.0] :: Anaconda, Inc. on linux
clearml.version
'1.0.5'
Ubuntu 20.04.1 LTS

We have physical server in server farm that we configure with 4 GPUs, so we run all on this hardware without cloud rent

I have firewall installed on the server and not all ports are open

Couple of words about our hydra config
it is located in root with train.py file. But the default config points to experiment folder with other configs and this is what I need to specify on every run