Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi! To Make My Script Work Inside A Task, I Need To Add

Hi!

To make my script work inside a task, I need to add working_dir to the PYTHONPATH . According to the ClearML agent console log, here are the relevant lines:

Current configuration (clearml_agent v1.8.1, location: /tmp/clearml.conf):
----------------------
<...>
working_dir = .

Additionally, I found the CLEARML_TASK_WORKING_DIR variable in the ClearML agent's source code here .

To include this in my PYTHONPATH , I tried adding the following test lines to my custom shell script:

echo $CLEARML_TASK_WORKING_DIR
echo $CLEARML_TASK_SCRIPT_ENTRY
echo AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

However, I get empty strings as output (see the screenshot).

Please help!

  
  
Posted one month ago
Votes Newest

Answers 8


Hi @<1714451218161471488:profile|ClumsyChimpanzee54> ! We will automatically add the cwd of the pipeline controller to the python path when running locally in a future version.
If running remotely, you can approach this in a few ways:

  • add the whole project to a git repo and specify that repo in the pipeline steps
  • have a prebuilt docker image that contains your project's code. you may then set the working directory to the path of your project
  • if the agent running the docker is running on the same machine that contains your project, you may mount the directory containing your project using -v and extend the python path to point to that directory using the -e docker argument. You may add these using docker_args
  
  
Posted one month ago

@<1523701070390366208:profile|CostlyOstrich36> I've added clearml to my project. Everything works fine locally: all metrics and other metadata are sent to clearml. Now I want to run my experiments through agent and at this step I got problems.

My project has a complex structure, so I need to modify the PYTHONPATH variable. Otherwise agent will fail with import error. Specifically, I want to add CLEARML_TASK_WORKING_DIR there. How and where should I do this?

  
  
Posted one month ago

Thanks

  
  
Posted one month ago

Thanks @<1523701435869433856:profile|SmugDolphin23> I will try the first one

  
  
Posted one month ago

You can export it in the same shell you run the agent in and that should work for example

export FOO=bar clearml-agent daemon ... 
  
  
Posted one month ago

image

  
  
Posted one month ago

Hey, having very similar issue where my pipeline requires imports from other files in my repository that require the root of the directory to be on the pythonpath.
e.g.
from src.pipelines.pipeline_function_dependencies import parse_total_dataset
src/pipelines/nested_cv_pipeline.py

So folder on level of src needs to be on pythonpath. Should I be using docker_args for this? Or maybe docker_bash_setup_script?

And how do I access the path to the working dir in the docker container so I can set the pythonpath?

  
  
Posted one month ago

Hi @<1691983266761936896:profile|AstonishingOx62> , I'm not sure I understand what you're trying to do. You have some python code unrelated to ClearML. Does it run without issues? Did you afterwards add Task.init() to that code?

  
  
Posted one month ago
217 Views
8 Answers
one month ago
one month ago
Tags
Similar posts