Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Is There A Way To Save The Models Completely On The Clearml Server? It Seems That Clearml Server Does Not Store The Models Or Artifacts Itself, But They Are Stored Somewhere Else (E.G., Aws S3-Bucket) Or On My Local Machine And Clearml Server Is Only Sto

Is there a way to save the models completely on the ClearML server?

It seems that ClearML Server does not store the models or artifacts itself, but they are stored somewhere else (e.g., AWS S3-bucket) or on my local machine and ClearML Server is only storing configuration parameters and previews (e.g., when the artifact is a pandas dataframe). Is that right?

  
  
Posted 2 years ago
Votes Newest

Answers 45


We have the following, works fine (we also use internal zip packaging for our models):

model = OutputModel(task=self.task, name=self.job_name, tags=kwargs.get('tags', self.task.get_tags()), framework=framework)
model.connect(task=self.task, name=self.job_name)
model.update_weights(weights_filename=cc_model.save())
  
  
Posted 2 years ago

I have already been trying to contribute (have three pull requests), but honestly I feel it is a bit weird, that I need to update a documentation about something I do not understand, while I actually try to evaluate if ClearML is the right tool for our company...

  
  
Posted 2 years ago

@<1523701087100473344:profile|SuccessfulKoala55> : That is the link I posted as well. But this should be mentioned also at places where it is about about the external or non-external storage. Also it should be mentioned everywhere we talk about models or artifacts etc. Not necessarily in details, but at least with a sentence and a link.

  
  
Posted 2 years ago

@<1523704157695905792:profile|VivaciousBadger56> I'm not sure I'm following you - is the issue not being able to upload to the ClearML server or to load the downloaded file?

  
  
Posted 2 years ago

FWIW, we prefer to set it in the agent’s configuration file, then it’s all automatic

  
  
Posted 2 years ago

@<1523704157695905792:profile|VivaciousBadger56> It seems like whatever you pickled in the zip file relies on some additional files that are not pickled.

  
  
Posted 2 years ago

We're certainly working hard on improving the documentation (and I do apologize for the frustrating experience)

  
  
Posted 2 years ago

@<1523704157695905792:profile|VivaciousBadger56> regrading: None
Is this a discussion or PR ?
(general ranting is saved for our slack channel 🙂 )

  
  
Posted 2 years ago

Yes, you're correct, I misread the exception.
Maybe it hasn't completed uploading? At least for Datasets one needs to explicitly wait IIRC

  
  
Posted 2 years ago

@<1523701083040387072:profile|UnevenDolphin73> : Thanks, but it does not mention the File Storage of "ClearML Hosted Server".

  
  
Posted 2 years ago

I can only say I’ve found ClearML to be very helpful, even given the documentation issue.
I think they’ve been working on upgrading it for a while, hopefully something new comes out soon.
Maybe @<1523701205467926528:profile|AgitatedDove14> has further info 🙂

  
  
Posted 2 years ago

Unbelievable! That worked.

  
  
Posted 2 years ago

The documentation is messy, I’ve complained about it the in the past too 🙈

  
  
Posted 2 years ago

Heh, good @<1523704157695905792:profile|VivaciousBadger56> 😁
I was just repeating what @<1523701070390366208:profile|CostlyOstrich36> suggested, credits to him

  
  
Posted 2 years ago

Heh, well, John wrote that in the first reply in this thread 🙂
And in Task.init main documentation page (nowhere near the code), it says the following -
image

  
  
Posted 2 years ago

Hi all, sorry for not being so responsive today 🙏

  
  
Posted 2 years ago

By the way, output_uri is also documented as part of the Task.init() docstring ( None )

  
  
Posted 2 years ago

Exactly 🙂

  
  
Posted 2 years ago

@<1523701083040387072:profile|UnevenDolphin73> : If I do, what should I configure how?

  
  
Posted 2 years ago

But, I guess @<1523701070390366208:profile|CostlyOstrich36> wrote that in a different chat, right?

  
  
Posted 2 years ago

missing a configuration option

Which one, where? Any idea? I did not set output_uri - do I have to do that?

I am refering to

  
  
Posted 2 years ago

"Messy" is putting it nicely.

  
  
Posted 2 years ago

@<1523701083040387072:profile|UnevenDolphin73>

  
  
Posted 2 years ago

@<1523701087100473344:profile|SuccessfulKoala55> Also, I think that - in this case, but also in other cases - the issue is not just the documentation, but also the design of the SDK.

  
  
Posted 2 years ago

@<1523701083040387072:profile|UnevenDolphin73> : I do not get this impression, because during update_weights I get the message

2023-02-21 13:54:49,185 - clearml.model - INFO - No output storage destination defined, registering local model C:\Users..._Demodaten_FF_2023-02-21_13-53-51.624362.model

  
  
Posted 2 years ago

Hi @<1523704157695905792:profile|VivaciousBadger56> , you can configure Task.init(..., output_uri=True) and this will save the models to the clearml file server

  
  
Posted 2 years ago

@<1523701083040387072:profile|UnevenDolphin73> : From which URL is your most recent screenshot?

  
  
Posted 2 years ago

FWIW It’s also listed in other places @<1523704157695905792:profile|VivaciousBadger56> , e.g. None says:

In order to make sure we also automatically upload the model snapshot (instead of saving its local path), we need to pass a storage location for the model files to be uploaded to.
For example, upload all snapshots to an S3 bucket…

  
  
Posted 2 years ago

@<1523701083040387072:profile|UnevenDolphin73> : I do not see any way to download the model manually from the web app either. All I see is the link to the file on my harddrive (see shreenshot).

The second process says there is not file at all. I think, all that happened is that the update_weights only uploaded the location of the .zip file (which we denote as a .model file) on my harddrive, but not the file itself.
image

  
  
Posted 2 years ago

It should store it on the fileserver, perhaps you're missing a configuration option somewhere?

  
  
Posted 2 years ago
155K Views
45 Answers
2 years ago
2 years ago
Tags