Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi Everyone! Is There A Way To Specify The Working Directory In A Pipeline Component? I’M Using Pipelines From Decorators, I Can Set The Repo Url Just Fine, But I’M Running Everything From A Subfolder, And The Working Dir Is Set To

Hi everyone! Is there a way to specify the working directory in a pipeline component? I’m using pipelines from decorators, I can set the repo url just fine, but I’m running everything from a subfolder, and the working dir is set to . . Is there a way to specify the working dir from the decorator? Now I have something like this:

@PipelineDecorator.component(
    name='dataset'
    repo='
',
    repo_branch='main',
)
def generate_dataset() -> str:
  ...
  
  
Posted 2 months ago
Votes Newest

Answers 10


Hi @<1523701205467926528:profile|AgitatedDove14> , in my case all the code is in a subfolder, like projects/main , so if I run from the git root it can’t find the local modules

  
  
Posted 2 months ago

Is there any other way to specify it besides directly in the component?

  
  
Posted 2 months ago

Oh nice thanks, will try with that combination

  
  
Posted 2 months ago

Hmm yes, @<1570220858075516928:profile|SlipperySheep79> I think you are right in your case it make sense to do add this option.
Could you add GH issue with the feature request? it should be fairly easy to add and we use GH to make sure we track those requests
wdyt?

  
  
Posted 2 months ago

This would work to load the local modules, but I’m also using poetry and the

pyproject.toml

is in the subdirectory, so the agent won’t install any dependency if I don’t set the

work_dir

hmmm true, in terms of requirements, you can list them in the decorator (see packages argument)

  
  
Posted 2 months ago

Sure will do!

  
  
Posted 2 months ago

Hi @<1570220858075516928:profile|SlipperySheep79>

Is there a way to specify the working dir from the decoratoe

not directly, but why would that change anything? I mean the coponent code will be created in the git root, and you can still access files inside the subfolders

from .subfolder import something

what am I missing?

  
  
Posted 2 months ago

In the meantime, any suggestion on how to set the working_dir in any other way? We are moving to this new code structure and I’d like to have clearml up and running

  
  
Posted 2 months ago

Okay this is a bit hacky but will work

@PipelineDecorator.component(...)
def step(...)
  import sys
  import os
  sys.path.append(os.path.join(os.path.abspath(os.path.dirname(__file__)), "projects", "main" ))
  
  from file import something
  
  
Posted 2 months ago

This would work to load the local modules, but I’m also using poetry and the pyproject.toml is in the subdirectory, so the agent won’t install any dependency if I don’t set the work_dir

  
  
Posted 2 months ago
154 Views
10 Answers
2 months ago
2 months ago
Tags