Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
When My Remote Task Is Installing The Python Dependencies

When my remote task is installing the python dependencies --packages requests for example, is there any caching "magic" that is done by the k8 agent? Or is it always the equivalent of pip install <deps> --no-cache-dir ?

  
  
Posted 8 months ago
Votes Newest

Answers 30


PYTHONPATH: /home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-py/sfi:/home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-py:/home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-py/sfi/imagery/models/training::/home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-py/sfi:/usr/lib64/python37.zip:/usr/lib64/python3.7:/usr/lib64/python3.7/lib-dynload:/home/npuser/.clearml/venvs-builds/3.7/lib64/python3.7/site-packages:/home/npuser/.clearml/venvs-builds/3.7/lib/python3.7/site-packages

  
  
Posted 8 months ago

note

/home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-py

is the correct pat

So how come it is failing?
Can you also print sys.path just to be sure ?

  
  
Posted 8 months ago

yea IDK, the git repo is a python library. Is it possible to run something like pip install -e . so I can utilize the setup.py?

  
  
Posted 8 months ago

When I run from sfi.imagery import models. It works fine local. So the repo is setup for proper imports. But fails in clearML tasks

  
  
Posted 8 months ago

note /home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-py is the correct path

  
  
Posted 8 months ago

The config change worked but my PYTHONPATH is still not working as expected

  
  
Posted 8 months ago

basically, can I do local installs vs supplying a requirements.txt

  
  
Posted 8 months ago

Yea I've done that already but I can do it again

  
  
Posted 8 months ago

Traceback (most recent call last): File "sfi/imagery/models/training/ldc_train_end_to_end.py", line 26, in <module> from sfi.imagery.models.chip_classifier.eval import eval_chip_classifier ModuleNotFoundError: No module named 'sfi.imagery.models'

  
  
Posted 8 months ago

Notice you have in the Path:
/home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-py/sfiBut you should have:
/home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-py/

  
  
Posted 8 months ago

If you look lower, it is there '/home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-py'

  
  
Posted 8 months ago

When I run this line locally, it works fine
from sfi.imagery.models.chip_classifier.eval import eval_chip_classifier

  
  
Posted 8 months ago

Seems like it has everything I would need

  
  
Posted 8 months ago

it has sfi suffix and regular root

  
  
Posted 8 months ago

What's the log you are seeing?

  
  
Posted 8 months ago

which should cover both cases

  
  
Posted 8 months ago

/home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-pyYep I see it now, could you simulate locally (i.e have the other folders in the path as well)?
could it be you also have a file somewhere that is called sfi or imagery or models or chip_classifier that it accidently tries to import first from ?

  
  
Posted 8 months ago

For instance, In my repo, I have a setup.py, how would I run pip install -e .

  
  
Posted 7 months ago

AgitatedDove14 How would I install using a setup.py in a clearML task?

  
  
Posted 7 months ago

Not yet AgitatedDove14 Perhaps we can pair on this Monday.

  
  
Posted 8 months ago

BoredHedgehog47 were you able to locate the issue ?

  
  
Posted 8 months ago

This is to address the PYTHONPATH issues

  
  
Posted 7 months ago

I was hoping to use docker_bash_setup_script but it didn't work when I ran pip install -e . in the respective script

  
  
Posted 7 months ago

Would be great if the docker_bash_setup_script had output I could see

  
  
Posted 7 months ago

Okay, this worked --packages '-e .'

  
  
Posted 7 months ago

SysPath: ['/home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-py/sfi/imagery/models/training', '/home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-py/sfi', '/home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-py', '/usr/lib64/python37.zip', '/usr/lib64/python3.7', '/usr/lib64/python3.7/lib-dynload', '/home/npuser/.clearml/venvs-builds/3.7/lib64/python3.7/site-packages', '/home/npuser/.clearml/venvs-builds/3.7/lib/python3.7/site-packages']

  
  
Posted 8 months ago

PYTHONPATH is still not working as expected

inside your code if you do :
import os print("PYTHONPATH", os.environ["PYTHONPATH"])what are you getting?

  
  
Posted 8 months ago

Could it be something else is missing and hence the import fails ?

  
  
Posted 8 months ago
107 Views
30 Answers
8 months ago
4 months ago
Tags