Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Is There A Way To Save The Models Completely On The Clearml Server? It Seems That Clearml Server Does Not Store The Models Or Artifacts Itself, But They Are Stored Somewhere Else (E.G., Aws S3-Bucket) Or On My Local Machine And Clearml Server Is Only Sto

Is there a way to save the models completely on the ClearML server?

It seems that ClearML Server does not store the models or artifacts itself, but they are stored somewhere else (e.g., AWS S3-bucket) or on my local machine and ClearML Server is only storing configuration parameters and previews (e.g., when the artifact is a pandas dataframe). Is that right?

  
  
Posted one year ago
Votes Newest

Answers 45


@<1523701083040387072:profile|UnevenDolphin73> : Thanks, but it does not mention the File Storage of "ClearML Hosted Server".

  
  
Posted one year ago

Heh, well, John wrote that in the first reply in this thread ๐Ÿ™‚
And in Task.init main documentation page (nowhere near the code), it says the following -
image

  
  
Posted one year ago

Exactly ๐Ÿ™‚

  
  
Posted one year ago

Hi all, sorry for not being so responsive today ๐Ÿ™

  
  
Posted one year ago

Unbelievable! That worked.

  
  
Posted one year ago

But we do use S3

  
  
Posted one year ago

We have the following, works fine (we also use internal zip packaging for our models):

model = OutputModel(task=self.task, name=self.job_name, tags=kwargs.get('tags', self.task.get_tags()), framework=framework)
model.connect(task=self.task, name=self.job_name)
model.update_weights(weights_filename=cc_model.save())
  
  
Posted one year ago

@<1523701087100473344:profile|SuccessfulKoala55> : I referenced this conversation in the issue None

  
  
Posted one year ago

The documentation is messy, Iโ€™ve complained about it the in the past too ๐Ÿ™ˆ

  
  
Posted one year ago

@<1523701087100473344:profile|SuccessfulKoala55> : That is the link I posted as well. But this should be mentioned also at places where it is about about the external or non-external storage. Also it should be mentioned everywhere we talk about models or artifacts etc. Not necessarily in details, but at least with a sentence and a link.

  
  
Posted one year ago

@<1523701083040387072:profile|UnevenDolphin73>

  
  
Posted one year ago

@<1523701083040387072:profile|UnevenDolphin73> : I see. I did not make the connection that output_uri=True is what I was missing. I thought this was the default. But the default is actually "None", which is different than "True".

  
  
Posted one year ago

@<1523701087100473344:profile|SuccessfulKoala55> I think I might have made a mistake earlier - but not in the code I posted before. Now, I have the following situation:

  • In my training Python process on my notebook I train the custom made model and put it on my harddrive as a zip file. Then I run the code
output_model = OutputModel(task=task, config_dict={...}, name=f"...")
output_model.update_weights(weights_filename=r"C:\path\to\mymodel.zip", is_package=True)
  1. I delete the "C:\path\to\mymodel.zip", because it would not be available on my colleagues' computers.

  2. In a second process, the model-inference process, I run

mymodel = task.models['output'][-1]
mymodel = mymodel.get_local_copy(extract_archive=True, raise_on_error=True)

and get the error

ValueError: Could not retrieve a local copy of model weights 8ad4db1561474c43b0747f7e69d241a6, failed downloading

I do not have an aws S3 instance or something like that. This is why I would like to store my mymodel.zip file directly on the ClearML Hosted Service. The model is around 2MB large.

How should I proceed?

  
  
Posted one year ago

@<1523701083040387072:profile|UnevenDolphin73> : I do not see any way to download the model manually from the web app either. All I see is the link to the file on my harddrive (see shreenshot).

The second process says there is not file at all. I think, all that happened is that the update_weights only uploaded the location of the .zip file (which we denote as a .model file) on my harddrive, but not the file itself.
image

  
  
Posted one year ago

But, I guess @<1523701070390366208:profile|CostlyOstrich36> wrote that in a different chat, right?

  
  
Posted one year ago

Heh, good @<1523704157695905792:profile|VivaciousBadger56> ๐Ÿ˜
I was just repeating what @<1523701070390366208:profile|CostlyOstrich36> suggested, credits to him

  
  
Posted one year ago

I can only say Iโ€™ve found ClearML to be very helpful, even given the documentation issue.
I think theyโ€™ve been working on upgrading it for a while, hopefully something new comes out soon.
Maybe @<1523701205467926528:profile|AgitatedDove14> has further info ๐Ÿ™‚

  
  
Posted one year ago

@<1523704157695905792:profile|VivaciousBadger56> regrading: None
Is this a discussion or PR ?
(general ranting is saved for our slack channel ๐Ÿ™‚ )

  
  
Posted one year ago

I am not sure if it the fact the name of the file ends with .model is an issue - but that would be somewhat crazy design...

  
  
Posted one year ago

It is documented at None ... super deep in the code. If you don't know that output_uri in TASK's (!) init is relevant, you would never know...

  
  
Posted one year ago

FWIW, we prefer to set it in the agentโ€™s configuration file, then itโ€™s all automatic

  
  
Posted one year ago

Either? ๐Ÿ™‚

  
  
Posted one year ago

FWIW Itโ€™s also listed in other places @<1523704157695905792:profile|VivaciousBadger56> , e.g. None says:

In order to make sure we also automatically upload the model snapshot (instead of saving its local path), we need to pass a storage location for the model files to be uploaded to.
For example, upload all snapshots to an S3 bucketโ€ฆ

  
  
Posted one year ago

@<1523701083040387072:profile|UnevenDolphin73> : I do not get this impression, because during update_weights I get the message

2023-02-21 13:54:49,185 - clearml.model - INFO - No output storage destination defined, registering local model C:\Users..._Demodaten_FF_2023-02-21_13-53-51.624362.model

  
  
Posted one year ago

It should store it on the fileserver, perhaps you're missing a configuration option somewhere?

  
  
Posted one year ago

We'll try to add referenced to that in other places as well ๐Ÿ‘

  
  
Posted one year ago

Yes, you're correct, I misread the exception.
Maybe it hasn't completed uploading? At least for Datasets one needs to explicitly wait IIRC

  
  
Posted one year ago

@<1523701083040387072:profile|UnevenDolphin73> : If I do, what should I configure how?

  
  
Posted one year ago

Hi @<1523704157695905792:profile|VivaciousBadger56> , you can configure Task.init(..., output_uri=True) and this will save the models to the clearml file server

  
  
Posted one year ago

"Messy" is putting it nicely.

  
  
Posted one year ago
10K Views
45 Answers
one year ago
one year ago
Tags