For instance, In my repo, I have a setup.py, how would I run pip install -e .
When I run from sfi.imagery import models. It works fine local. So the repo is setup for proper imports. But fails in clearML tasks
Not yet AgitatedDove14 Perhaps we can pair on this Monday.
Could it be something else is missing and hence the import fails ?
The config change worked but my PYTHONPATH is still not working as expected
basically, can I do local installs vs supplying a requirements.txt
This is to address the PYTHONPATH issues
Seems like it has everything I would need
When I run this line locally, it works finefrom sfi.imagery.models.chip_classifier.eval import eval_chip_classifier
BoredHedgehog47 were you able to locate the issue ?
If you look lower, it is there '/home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-py'
Would be great if the docker_bash_setup_script
had output I could see
PYTHONPATH: /home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-py/sfi:/home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-py:/home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-py/sfi/imagery/models/training::/home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-py/sfi:/usr/lib64/python37.zip:/usr/lib64/python3.7:/usr/lib64/python3.7/lib-dynload:/home/npuser/.clearml/venvs-builds/3.7/lib64/python3.7/site-packages:/home/npuser/.clearml/venvs-builds/3.7/lib/python3.7/site-packages
I was hoping to use docker_bash_setup_script
but it didn't work when I ran pip install -e .
in the respective script
Seems related to this https://github.com/allegroai/clearml/issues/241
AgitatedDove14 How would I install using a setup.py in a clearML task?
yea IDK, the git repo is a python library. Is it possible to run something like pip install -e .
so I can utilize the setup.py?
Traceback (most recent call last): File "sfi/imagery/models/training/ldc_train_end_to_end.py", line 26, in <module> from sfi.imagery.models.chip_classifier.eval import eval_chip_classifier ModuleNotFoundError: No module named 'sfi.imagery.models'
/home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-py
Yep I see it now, could you simulate locally (i.e have the other folders in the path as well)?
could it be you also have a file somewhere that is called sfi or imagery or models or chip_classifier that it accidently tries to import first from ?
SysPath: ['/home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-py/sfi/imagery/models/training', '/home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-py/sfi', '/home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-py', '/usr/lib64/python37.zip', '/usr/lib64/python3.7', '/usr/lib64/python3.7/lib-dynload', '/home/npuser/.clearml/venvs-builds/3.7/lib64/python3.7/site-packages', '/home/npuser/.clearml/venvs-builds/3.7/lib/python3.7/site-packages']
Yea I've done that already but I can do it again
PYTHONPATH is still not working as expected
inside your code if you do :import os print("PYTHONPATH", os.environ["PYTHONPATH"])
what are you getting?
note /home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-py
is the correct path
BoredHedgehog47
is this ( https://clearml.slack.com/archives/CTK20V944/p1665426268897429?thread_ts=1665422655.799449&cid=CTK20V944 ) the same issue (or solution) ?
Notice you have in the Path:/home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-py/sfi
But you should have:/home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-py/
note
/home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-py
is the correct pat
So how come it is failing?
Can you also print sys.path
just to be sure ?