Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Trying To Create A Data Pipeline On My Own. Wanted To Ask, For Each Batch Of Data, Do I Have To Create A New Dataset Object Or Do I Just Create One Dataset Object And Add Batches To It. If Its The Latter, Then How.

Trying to create a data pipeline on my own. Wanted to ask, for each batch of data, do I have to create a new Dataset Object or do I just create one Dataset Object and add batches to it. If its the latter, then how.

  
  
Posted 2 years ago
Votes Newest

Answers 9


Can you guys let me know what finalize and publish methods do?

  
  
Posted 2 years ago

VexedCat68 , thanks for the update 🙂

  
  
Posted 2 years ago

Understandable.

  
  
Posted 2 years ago

Yep this will work. BTW check the new pipeline it might have a more flexible solution
https://github.com/allegroai/clearml/blob/master/examples/pipeline/full_custom_pipeline.py

  
  
Posted 2 years ago

I'm kind of new to developing end to end applications so I'm also learning how the predefined pipelines work as well. I'll take a look at the clear ml custom pipelines

  
  
Posted 2 years ago

Basically lock the Task (so you cannot reset it or change it). Usually it also marks "ready to use" etc. It also will publish the models the Task created.

  
  
Posted 2 years ago

Finalizes locks the model and publish I assume publishes it to the server

  
  
Posted 2 years ago

I'm assuming

  
  
Posted 2 years ago

For anyone reading this. I think I've gotten an understanding. I can add folders to a dataset so I'll be creating single dataset and will just keep adding folders to it. Then keep records of it in a database

  
  
Posted 2 years ago