Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
I Installed My Own Clearml-Server And Things Seems To Work Fine, But When I Delete An Experiment (I First Archived It Using Web Interface Popup Menu) Then Delete It Using The Web Inferface Popup Menu In Archive View)... Experiment Disapears From Web Inter

I installed my own clearml-server and things seems to work fine, but when i delete an experiment (i first archived it using web interface popup menu) then delete it using the web inferface popup menu in archive view)... Experiment disapears from Web interface but i have a warning telling me that some files (artifacts associated with this experiment) are not deleted. And if i look in the directory mounted by docker-compose in which file server stores all these data, files are indeed not deleted. Do i miss something in configuration of file server ?

  
  
Posted 2 years ago
Votes Newest

Answers 30


i tried so far but it was not so easy, because there is a python executable "update_from_env" that empties the configuration.json file. So i create a file in /mnt/external_files/configs and my configuration.json was read.

  
  
Posted 2 years ago

It seems that i should define this variable by the use of an environment variable in ConfigurationService.globalEnvironment.

  
  
Posted 2 years ago

What i don't understant is the list of artifacts that were not deleted

  
  
Posted 2 years ago

sudo docker logs clearml-fileserver

This gives no info at all. May be i should increase the log level to debug. The only message i got is about "werkzeug" the default server module of flash that shouldn't be use as production deployement (by the way, why not use gunicorn as entrypoint in docker-compose ?)

  
  
Posted 2 years ago

When i deleted the Experiment, i obtain a the following window :

  
  
Posted 2 years ago

I take a look a src/app/webapp-common/shared/entity-page/entity-delete/base-delete-dialog.effects.ts.

I see that an error is raised in the mergeMap at line 125, but i'm not familiar enough with Typescript to find why.

  
  
Posted 2 years ago

My files (fs) are deleted but i have the same issue as reported by SuperiorPanda77 , with some undefined value that is said not to be deleted. I guess that as my command deleteFileServerSources works but exit with some strange return value, other commands in the row addFaieldDeletedFiles and deleteProjectFromRoot are not executed (file src/app/webapp-common/shared/entity-page/entity-delete/base-delete-dialog.effects.ts )

  
  
Posted 2 years ago

MotionlessCoral18 , I think there is a new version out - 1.4, can you try upgrading to that?

  
  
Posted 2 years ago

The addresses seems strange, is this the hostname?

I use the nip services to have subdomains: clearml.domain api.domain and file.domain that points to the same host.

  
  
Posted 2 years ago

My artifacts are now deleted but the directories where the artifacts are stored are not deleted.

  
  
Posted 2 years ago

DeterminedCrab71 You right, if i understand correctly HTTP.FILE_BASE_URL is undefined, then file to delete is describe as "misc" instead of "fc" then i guess system is unable to launch the delete of the file

  
  
Posted 2 years ago

I was unable to define FILE_BASE_URL inside the docker container. I modify the HTTP constant in app.constants.ts with hard code values, compile the webapp again (npm) and replace it in my docker container and now it works....

  
  
Posted 2 years ago

you can manually override this configuration by adding or updating (if you have 1) the configuration.json file next to the index.html
with fileBaseUrl

  
  
Posted 2 years ago

Also, MotionlessCoral18 , you can take a look at the fileserver and see if you detect any errors, using sudo docker logs clearml-fileserver

  
  
Posted 2 years ago

in your setup you should probably configure docker webserver container to mount a config file to /usr/share/nginx/html/configuration.json

  
  
Posted 2 years ago

Usually the location is on the file server at opt/clearml/data/fileserver

The addresses seems strange, is this the hostname?

It seems like a mix between hostname and IP?

  
  
Posted 2 years ago

MotionlessCoral18 if you build the UI, instead of overriding stuff with hard coded values, you can set fileBaseUrl in src/environments/environment.prod.ts

  
  
Posted 2 years ago

Just fixing a typo - /opt/clearml/data/fileserver

  
  
Posted 2 years ago

were you able to confirm in the file server whether the files were actually deleted and the UI error message is wrong?

  
  
Posted 2 years ago

Files are stored on the same box where the docker is running. And there is a mounting point between file server docker and the host itself.

  
  
Posted 2 years ago

No particular information in console( no error), no network error too.

  
  
Posted 2 years ago

the UI should then send delete requests to the fileserver, do you see those in the network?

  
  
Posted 2 years ago

In docker-compose, image was latest allegroai/clearml:latest when i pull docker images. When i launch it, after installation i have in WebApp following informations : "WebApp: 1.3.1-169 • Server: 1.3.1-169 • API: 2.17"

  
  
Posted 2 years ago

No, no "delete" request 😞 Is there a way to put a breakpoint just before ?

  
  
Posted 2 years ago

Or whether the files are still there?

  
  
Posted 2 years ago

it might be an issue in the UI due to this unconventional address or network settings

I think this is related to an https://github.com/allegroai/clearml-server/issues/112#issue-1149080358 that seems to be a reoccurring issue across many different setups

  
  
Posted 2 years ago

My configuration.json is { "fileBaseUrl": " http://file.10.68.0.250.nip.io "}, but HTTP.FILE_BASE_URL still remains undefined . Something is missing ?

  
  
Posted 2 years ago

it might be an issue in the UI due to this unconventional address or network settings, before you try this again, can you open the browser devtools?
the console and network tabs should give us more insight

  
  
Posted 2 years ago

MotionlessCoral18 I thought no this is expected since the webapp sends a delete request for each object, but does not handle the directory. That something we need to improve

  
  
Posted 2 years ago

you can take a look at src/app/webapp-common/shared/entity-page/entity-delete/base-delete-dialog.effects.ts using the devtools debugger

  
  
Posted 2 years ago