interestingly, it works on one machine, but not on another one

I cannot share the file itself, but here are some potential helpful points:
Multiple lines empty One line is empty but has spaces (6 to be exact) The last line of the file is empty

line 13 is empty 🤔

using Trains Agent 🙂

so most likely one hard requirement installs the version 2 of pyjwt while setting up the experiment

Hi AgitatedDove14 , I don’t see any in the https://pytorch.org/ignite/_modules/ignite/handlers/early_stopping.html#EarlyStopping but I guess I could overwrite it and add one?

clearml doesn't change the matplotlib backend under the hood, right? Just making sure 😄

Sorry both of you, my problem was actually lying somewhere else (both buckets are in the same region) - thanks for you time!

Will the from clearml import Task raise an error if no clearml.conf exists? Or only when actual features requiring to define the server (such as Task.init ) will be called

Ok, I won't have time to venture to check the different database components, the first option (shuting down the server) sounds like the easiest option for me, I would then run manually the script once a month or so

The only thing that changed is the new auth.fixed_users.pass_hashed field, that I don’t have in my config file

AgitatedDove14 Yes I have the xpack security disabled, as in the link you shared (note that its xpack.security.enabled: "false" with brackets around false), but this command throws:

{"error":{"root_cause":[{"type":"parse_exception","reason":"request body is required"}],"type":"parse_exception","reason":"request body is required"},"status":400}

Setting it after the training correctly updated the task and I was able to store artifacts remotely

Does what you suggested here > https://github.com/allegroai/trains-agent/issues/18#issuecomment-634551232 also applies for containers used by the services queue?

I made sure before deleting the old index that the number of docs matched

That gave me

Running in Docker mode (v19.03 and above) - using default docker image: nvidia/cuda running python3

Building Task 94jfk2479851047c18f1fa60c1364b871 inside docker: ubuntu:18.04

Starting docker build
docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].
ERRO[0000] error waiting for container: context canceled

with 1.1.1 I get
User aborted: stopping task (3)

Make sure the cloned task is in Draft mode, if not, reset it
Then in the Execution tab of th task, in the Source Code section (first one), you can edit the values

AgitatedDove14 So what you are saying is that since I have trains-server 0.16.1, I should use trains>=0.16.1? And what about trains-agent? Only version 0.16 is released atm, this is the one I use

I can probably have a python script that checks if there are any tasks running/pending, and if not, run docker-compose down to stop the clearml-server, then use boto3 to trigger the creating of a snapshot of the EBS, then wait until it is finished, then restarts the clearml-server, wdyt?

Hi SuccessfulKoala55 , not really wrong, rather I don't understand it, the docker image with the args after it

btw, I tried with alpine instead of ubuntu:18.04, got :

Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
df20fa9351a1: Pulling fs layer
df20fa9351a1: Verifying Checksum
df20fa9351a1: Download complete
df20fa9351a1: Pull complete
Digest: sha256:185518070891758909c9f839cf4ca393ee977ac378609f700f60a771a2dfe321
Status: Downloaded newer image for alpine:latest
docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting containe...

Alright, thanks for the answer! Seems legit then 🙂

Sorry, I was actually able to fix it (using 1.1.3) not sure what was the problem 😄

Hi CostlyOstrich36 , this weekend I took a look at the diffs with the previous version ( https://github.com/allegroai/clearml-server/compare/1.1.1...1.2.0# ) and I saw several changes related to the scrolling/logging:
apiserver/bll/event/ http://log_events_iterator.py apiserver/bll/event/ http://events_iterator.py apiserver/config/default/services/_mongo.conf apiserver/database/model/ http://base.py apiserver/services/ http://events.pyI suspect that one of these changes might be responsible ...

I am sorry to give infos that are not very precise, but it’s the best I can do - Is this bug happening only to me?

I think it comes from the web UI of the version 1.2.0 of clearml-server, because I didn’t change anything else

CostlyOstrich36 I updated both agents to 1.1.2 and still go the same problem unfortunately. Since I can download the full log file from the Web UI, I guess the agents are reporting correctly?
Could it be that the elasticsearch does not return all the requested logs when it is queried from the WebUI to display it in the console?
Now that I think about it, I remember that on the changelog of the clearml-server 1.2.0 the following is listed:
` Fix UI Workers & Queues and Experiment Table pages ...

No, they have different names - I will try to update both agents to the latest versions

yes, 1.2 indeed