AgitatedDove14
yes, IP-based access, with DNS for some reason took 5 times longer and we abandoned it (MD is about our DNS and its settings).

AgitatedDove14
I suspect that it will be difficult to assign the old IP - I will consult with the admin guys.

Architecturally it sounds right to work with the base, though I don't have that much experience. I correctly understand, What then it is necessary to replace the address in a line of a way for a file?

docker-compose file

ah, I get it, I use (pytorch) lightning, and that's where it all comes from.

docker-compose -f /opt/clearml/docker-compose.yml down docker-compose -f /opt/clearml/docker-compose.yml pull docker-compose -f /opt/clearml/docker-compose.yml up -d

we run in containers without venv, in the main section, and then delete it or use it for similar experiments Sounds like something very similar, I'll try to use it, thanks a lot! Can this be configured in the UI by simply adding a docker image to the launch options?

great, point 2 sounds like the right thing!)

please tell me, is it possible to somehow make it so that costomous fakets, which are not in the public domain, would be used?

for example, if I somehow start the execution of an agent task in a specific docker container?)

Thank you very much for your help and for such a convenient product!)
I haven't figured out the alents yet, but it already looks amazing!)

1. specify container from UI

2. libraries in the ubuntu repository have not yet reached their pip / pypi repository

sounds great)

AgitatedDove14
if I had to choose between logging or not logging, I would choose logging
If you choose between logging as 0 or as nan, then I would choose as nan
If you choose between skipping or logging like nan, then here I find it difficult, it seems that it is better to log than skip, but you need to think.

to a greater extent, we are used to the tensorboard, where nan is logged in a special way, and this behavior seems to be natural.

AgitatedDove14 yes, that's right, he's changed

@<1523701087100473344:profile|SuccessfulKoala55>
I'm talking about something like OPTUNA

wow, that's interesting, please let me know. Are there screenshots or a demo video somewhere where you can see how the enumeration parameters are set.

i use the local free version of clearml

About migration - we saved the data archives and copied them to a new server, extracting them to the appropriate folders and setting the necessary rights, and rebuilding the docker image and launching the container

Before all this, we migrated to the new version according to the instructions and everything went well, all the data after the restart was displayed correctly.

And only after that we began the process of switching to new hardware - with a large disk.

And why can it be that the displayed time is zero?
although the experiments were considered for several days

CostlyOstrich36
*If the agent did not perform a certain action, then its average reward per episode for this action will be nan , not 0

Does this only work for the completed status?
and does not take into account failed and ABORTED experiments?

all code

class LitMNIST(LightningModule): ... self.log('test/test_nan', np.nan, prog_bar=False, logger=True, on_step=True, on_epoch=False) ...

CostlyOstrich36 ^

up error

/usr/local/lib/python3.7/dist-packages/urllib3/util/retry.py:86: DeprecationWarning: Using 'Retry.BACKOFF_MAX' is deprecated and will be removed in v2.0. Use 'Retry.DEFAULT_BACKOFF_MAX' instead DeprecationWarning,

use local hosting by changing only 1 port (8080->8099) in the original docker compose file

ClearML does not log images when in lighting use TensorBoardLogger

2022-03-29 15:15:52,031 - clearml.metrics - WARNING - Failed uploading to http://10.151.32.18:8091 (HTTPConnectionPool(host='10.151.32.18', port=8091): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd4ec8f2310>: Failed to establish a new connection: [Errno 111] Connection refused')))
2022-03-29 15:15:52,034 - clearml.metrics - ERROR - Not uploading 1/4 events because the data upload failed

This only happens when I try to block pictures.
When I disable image logging, this error does not occur.

Perhaps someone has already encountered this and knows how to solve it?
help please

requirements.txt

tb image

clearml