Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi Everyone, How Can I Add A New Local Storage Location. Clearml Is In /Opt/Clearml And I Have Now Added A Second Hard Driver To My Server. How Can I Make Clearml Use The Additional Storage Space?

Hi everyone,
how can I add a new local storage location. clearml is in /opt/clearml and I have now added a second hard driver to my server. How can I make clearml use the additional storage space?

  
  
Posted 2 years ago
Votes Newest

Answers 30


Can you give me an example how I can add a second fileserver?

  
  
Posted 2 years ago

And how to specify this fileserver as output_uri ?

  
  
Posted 2 years ago

Well, you can simply duplicate the fileserver's docker-compose entry and expose it using a different port (8082 for example?)

  
  
Posted 2 years ago

And how do I specify this in the output_uri ? The default file server is specified by passing True . How would I specify to use the second?

  
  
Posted 2 years ago

Just use the full URL for the server (i.e. http://<server-address>:8082 )

  
  
Posted 2 years ago

Thank you very much 🙂

  
  
Posted 2 years ago

Btw: It is weird that the fileservers are directly exposed, so no authentication through the webserver is needed. Is this something that is different in the paid version or why is it like that in the open-source version?

  
  
Posted 2 years ago

The security scheme is indeed covered in the enterprise version

  
  
Posted 2 years ago

(and in the SaaS Hosted version as well, of course)

  
  
Posted 2 years ago

Okay, thank you 🙂

  
  
Posted 2 years ago

Any idea why deletion of artifacts on my second fileserver does not work?

fileserver_datasets: networks: - backend - frontend command: - fileserver container_name: clearml-fileserver-datasets image: allegroai/clearml:latest restart: unless-stopped volumes: - /opt/clearml/logs:/var/log/clearml - /opt/clearml/data/fileserver-datasets:/mnt/fileserver - /opt/clearml/config:/opt/clearml/config ports: - "8082:8081"
ClearML successfully stores artifacts on there, I can download them, but when I try to delete I get .... not deleted as error message, but 100% deletion and they get removed from the webserver, but remain on disk.

  
  
Posted 2 years ago

I guess that's because the UI doesn't recognize it as a fileserver and does not attempt to send the requests there

  
  
Posted 2 years ago

But would storing then work?

  
  
Posted 2 years ago

Should work since that's done by the SDK

  
  
Posted 2 years ago

Ah, I see. Any way to make the UI recognize it as a file server?

  
  
Posted 2 years ago

I'll ask them. but I'm not sure regarding priority 😞

  
  
Posted 2 years ago

I guess the supported storage mediums (e.g. S3, ceph, etc...) dont have this issue, right?

  
  
Posted 2 years ago

In these mediums, the UI currently does not handle deleting

  
  
Posted 2 years ago

Ah, okay, that's weird 🙂 Thank you for answering!

  
  
Posted 2 years ago

BTW, deleting using the SDK ( Task.delete() should handle all cases)

  
  
Posted 2 years ago

Ah, thanks a lot. So for example the CleanUp Service ( https://github.com/allegroai/clearml/blob/master/examples/services/cleanup/cleanup_service.py ) should have no troubles deleting the artifacts.

  
  
Posted 2 years ago

Btw: Is it intented that the folder structures in the fileserver directories is not deleted?

  
  
Posted 2 years ago

If there are models used by other tasks, they will be kept there under the existing structure

  
  
Posted 2 years ago

One thing I want to add: Maybe you disabling deletion of artifacts if file-server deletion fails. Doesn't make sense that we cannot track existing files if something goes wrong.

  
  
Posted 2 years ago

What do you mean by "disable deletion of artifacts"?

  
  
Posted 2 years ago

Hi SuccessfulKoala55
I meant that in the WebUI deletion should only be allowed for artifacts for which deletion actually works.
For example I now have a lot of lingering artifacts that exist on the fileservers, but not on the clearml-api-server (I think).
Another example: I delete a task via WebUI. ClearML-server tries to delete the task and the artifacts belonging to the task. However, it will show that the task has been successfully deleted but some artifacts have not. Now there is no way anymore to delete the artifacts except for asking the admin to manually look for files on the fileserver.

In my opinion the task should only be deleted after the artifacts have been deleted successfully . And the artifacts should only be deleted from the server when the fileserver successfully deletes the files behind the artifacts.

  
  
Posted 2 years ago

I fully understand, the thing is it's not the server deleting the files right now, but the UI - once we add this option in the server, we will be able to provide this flow. Still, I'm sure that it would still be a viable option to delete a task even if the associated artifacts can't be deleted for some reason (for example, what if you have an artifact weighing a few KBs that's somehow corrupted?)

  
  
Posted 2 years ago

Yea, when the server handles the deletes everythings fine and imo, that is how it should always have been.

I don't think it is a viable option. You are looking at the best case, but I think you should expect the worst from the users 🙂 Also I would rather know there is a problem and have some clutter than to hide it and never be able to fix it because I cannot identify which artifacts are still in use without spending a lot of time comparing artifact IDs.

  
  
Posted 2 years ago

Well, it's certainly in our workplan 🙂

  
  
Posted 2 years ago

Perfect and thank you for your efforts! :)

  
  
Posted 2 years ago
1K Views
30 Answers
2 years ago
one year ago
Tags