Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Profile picture
AppetizingMouse58
Moderator
0 Questions, 132 Answers
  Active since 10 January 2023
  Last activity 2 years ago

Reputation

0
0 Hi Clearml, I Tried To Upgrade The Clearml Server Following This

Hi SarcasticSparrow10 , I am trying to understand whether we have some gaps in the instructions. In the upgrade process did you perform the steps 3-10 of the below instruction? Were there any errors when performing these steps?
https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_mongo44_migration

2 years ago
0 Hey There Have The Following Issue After Upgrading Server And Trains To 0.16:

SubstantialBaldeagle49 This should collect the logs: 'sudo docker logs trains-apiserver >& apiserver.logs'

4 years ago
0 Greetings! Could You Help Me? I’Ve Just Tried Delete Old Experiment (Year Ago) But Got The Following Error:

Hi ResponsiveCamel97 , the shards and indices stats look fine. Can you please try the async delete of the task data? You can run the following line in the shell inside the apiserver container. Just replace <task_id> with your actual task id
curl -XPOST -H "Content-Type: application/json" " " -d'{"query": {"term": {"task": "<task_id>"}}}'You should get in response something like this:
{"task":"p6350SG7STmQALxH-E3CLg:1426125"}Then you can periodically ping ES on the status of the r...

2 years ago
0 Hi All, I’M Running Experiments Using Clearml. The Training Is Very Slow, And I’M Getting The Following Errors And Warnings:

It seems that index events-log-d1bd92a3b039400cbafc60a7a5b1e52b got corrupted. In case there are no backups the only choice would be to delete this index from elasticsearch

2 years ago
0 Hi All, I’M Running Experiments Using Clearml. The Training Is Very Slow, And I’M Getting The Following Errors And Warnings:

Actually the task logs will be lost. The tasks themselves and their reported metrics and plots would stay. The command is the following:
curl -XDELETE localhost:9200/events-log-d1bd92a3b039400cbafc60a7a5b1e52b

2 years ago
0 Hi, I Am Having Problem With Clearml Running On Our Private Server. This Error Occured On Older Version On Clearml And Server. Now After Update And Purge Of All Old Database With

The index events-training_stats_scalar-d1bd92a3b039400cbafc60a7a5b1e52b status is red. Meaning that the data for this index got corrupted. Since there are no replicas the only feasible option would be to delete this index. All the training scalars events for the old taskd would be lost then. But the newly created tasks should start working fine.
curl -XDELETE

2 years ago
2 years ago
0 Hi Clearml, I Tried To Upgrade The Clearml Server Following This

Are you sure that it was performed fully according to the suggested sequence? The error that you posted says that v3.6 data is incompatible with v4.4 and suggests version 4.2 or earlier. Step 3 starts with mongo 4.0 that should be able to open v3.6 data. And then a number of gradual updates through versions 4.0->4.2->4.4 is performed

2 years ago
0 Hi Everyone, I Am Just Wondering Whether The Bugs Regarding The Deletion Of Tasks Is Fixed In The Current Version? E.G. This Happening When You Want To Delete A Lot Of Tasks.

@<1523701868901961728:profile|ReassuredTiger98> Strange:( in 1.10 we already had the code for clearing ES scrolls created during the task deletion. I would recommend upgrading to the latest release v1.12.1 anyway. In addition you can instruct ES to allow more open scrolls like below. By default it is limited to 500.

one year ago
0 Hi All, I Am Creating Sub Project, For Experiment, But It Seems There Is

Hi QuaintJellyfish58 , thanks for the feedback. I am trying to compare what you send and receive for team's view with what you get in My-work view. Can you please also send the data for the same requests and responses in the My work view structured in the same way like you sent for the team view now?

2 years ago
0 Hi Everyone, I Am Just Wondering Whether The Bugs Regarding The Deletion Of Tasks Is Fixed In The Current Version? E.G. This Happening When You Want To Delete A Lot Of Tasks.
curl -X PUT localhost:9200/_cluster/settings -H 'Content-Type: application/json' -d'{"persistent" : {"search.max_open_scroll_context": 1000}}'
one year ago
0 Hi Everyone! I'M Using Minios3 As A File Server And As Default Output Uri. I'Ve Faced The Following Problem. When I Delete Tasks From Web Ui (And Also From Archive) Their Artifacts Didn'T Get Deleted From S3. I'M Using Self Hosted Clearml==1.11. What Shou

Hi @<1585078752969232384:profile|FantasticDuck7> , there is an apiserver configuration file apiserver->config->default->services->storage_credentials.conf
It contains the parameters for accessing files on the external storages like s3, google or azure. Please provide the same minio server access parameters as you do for the SDK configuration.
The actual deletion is performed by the async_delete service. You can inspect its logs with "sudo docker logs async_delete" command. Before configuring...

one year ago
0 Hi, I Am Having Problem With Clearml Running On Our Private Server. This Error Occured On Older Version On Clearml And Server. Now After Update And Purge Of All Old Database With

Ok, I see. Then you can enter the apiserver container:
sudo docker exec -it clearml-apiserver /bin/bashAnd run the following commands inside the container
curl -XGET curl -XGET

2 years ago
0 Hi, I Would Like To Follow-Up In This

Hi JitteryCoyote63 , you mentioned that download task logs brings all the events. It would be interesting to compare the events that are in the download log but not in the task log screen with those that are returned in the screen too. Can you please share the download task logs file and the request and response that you get from the events.get_task_log for the same task?

2 years ago
0 Hi, I Would Like To Follow-Up In This

Thanks, looking into it:)

2 years ago
0 Hi All, I Am Creating Sub Project, For Experiment, But It Seems There Is

Hi QuaintJellyfish58 , I am investigating the issue. Can you please also send the request and response from projects.get_all when you are in the Team's Work view (the case where there is no undefined project)?

2 years ago
0 Greetings! Could You Help Me? I’Ve Just Tried Delete Old Experiment (Year Ago) But Got The Following Error:

With what memory setting do you run ES? How much memory and cpu is currently occupied by ES container?

2 years ago
0 Hi All, I’M Running Experiments Using Clearml. The Training Is Very Slow, And I’M Getting The Following Errors And Warnings:

Hi RattyFish27 , it seems that there is some issue with Elasticsearch cluster. Can you please run the following commands on the server and paste here their output?
curl -XGET curl -XGET

2 years ago
0 Hi, I Am Having Problem With Clearml Running On Our Private Server. This Error Occured On Older Version On Clearml And Server. Now After Update And Purge Of All Old Database With

It seems that elasticsearch is failing on any search request. Can you please run the following commands and share the results?
curl -XGET curl -XGET

2 years ago
0 Hi All, I Am Creating Sub Project, For Experiment, But It Seems There Is

Hi QuaintJellyfish58 , it seems that we identified the problem. The undefined project that you see is not a real project. It is a placeholder where the statistics of ex-1 project should be shown. We found a bug in the apiserver that under these particular conditions fails to return the ex-1 project data so the placeholder remains empty (undefined). If I understand correctly it should only cause the inconvenience but not influence your workflow. Is it correct? We are fixing the issue in the n...

2 years ago
0 Hi Clearml, I Tried To Upgrade The Clearml Server Following This

I mean it is not possible to open v3.6 data in version 4.4. That's why the steps 3-10 are there

2 years ago
0 Hi All, I’M Running Experiments Using Clearml. The Training Is Very Slow, And I’M Getting The Following Errors And Warnings:

Are you running them on the computer that hosts the server docker containers. What is the port binding for elasticsearch in your docker compose?

2 years ago
0 Trying To Enqueue A Task Through The Ui, Getting This Error - What Could It Be? (Running On Aws, On The Official Trains Ami)

Hi Elior, chances are that you do not have enough space for Elasticsearch on your storage. Please check the ES logs and increase the available disk space.

4 years ago
Show more results compactanswers