Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi Again, Hoping Someone Can Shed Some Light On My Workflow Planning. Im Rebuilding My Project From Scratch In Clearml, Using Remote Gitlab, I Have My Personal Pc (Dev) And A Local Linux Machine (Staging), No Prod Machine Yet. The Process Involves Pullin

hi again, hoping someone can shed some light on my workflow planning.

Im rebuilding my project from scratch in clearml, using remote gitlab, i have my personal pc (dev) and a local linux machine (staging), no prod machine yet.
the process involves pulling data from an API, storing in postgres, and running an ETL pipeline for feature creation (most likely to be stored back in postgres unless i settle on a better solution)

im trying to think my way through setting up my project to develop on my pc:

  • push changes to gitlab dev branch and have clearml-agent (dev) execute pipeline on my local dev machine
  • merge changes into staging branch on gitlab and have clearml-agent execute pipeline on staging machine
    is this the right approach? im just trying to get the workflow to fit into the clearml way of doing things.
    Is anyone else doing something similar and could give me some tips or direct me to a walkthrough somewhere. I havent been able to join all the dots reading the docs so far.

Ive scoured the internet but have not found any resources on data pipelines or ETL using clearml

  • can i have a clearml agent listed to a specific git branch?
  • how are environment variables passed to the clearml agent for different branches?
  • is there a full end to end project anywhere on the internet using clearml?
    i have a few blanks to fill in but i will say clearml looks like an awesome platform so far, i just need to align my dev workflow to get it up and running properly
  
  
Posted 3 days ago
Votes Newest

Answers 2


Hi @<1773158059758063616:profile|PanickyParrot17> ,

  • can i have a clearml agent listed to a specific git branch?The git branch is part of the repository info stored on each ClearML task - when creating the task (or when editing it), you can set this branch information. This is also automatically detected when you're running your code from a locally-cloned git repository - if you're using a branch there, it will be automatically picked up and stored in the resulting task's repository info

  • how are environment variables passed to the clearml agent for different branches?Environment variables can be pre-set for the agent, or, if you're using the agent in the docker mode (recommended), these can be additionally specified in the task's container section (using the extra arguments field) in the form of docker environment variables ( -e ENV_VAR=VALUE )

  
  
Posted 3 days ago

thanks @<1523701087100473344:profile|SuccessfulKoala55> that clears that up a bit for me. ive been going back through all the docs and videos trying to get a better plan together, there seems to be a big focus on distributed training and manually injecting params via the UI, but lacking a bit in the earlier stages of a data project.

id really love clearml to be the solution to self hosted ML, otherwise moving to AWS will be the next step for me

  
  
Posted 3 days ago
27 Views
2 Answers
3 days ago
3 days ago
Tags