Reputation
Badges 1
212 × Eureka!the API url works fine, returns 200
` * Serving Flask app 'fileserver' (lazy loading)
- Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead. - Debug mode: off
[2022-09-08 13:24:25,822] [8] [WARNING] [werkzeug] * Running on all addresses.
WARNING: This is a development server. Do not use it in a production deployment. `
I think if I use the local service URL this problem is fixed
perhaps I need to use localhost
I can see this log message in the nginx controller"GET / HTTP/1.1" 405 178 "-" "curl/7.79.1" 95 0.003 [clearml-clearml-fileserver-8081] [] 10.36.1.61:8081 178 0.004 405 b4f5caf7665ffa1e8823a195ae41ec26
I don't see any requests
I think the issue is the pod to pod comms can't resolve my route53 dns records
I used the values from the dashboard/configuration/api keys
perhaps the 405 is from nginx
Made some progress getting the gpu nodes to provision, but got this error on my task K8S glue status: Unschedulable (0/4 nodes are available: 1 node(s) had taint { http://nvidia.com/gpu : true}, that the pod didn't tolerate, 3 node(s) didn't match Pod's node affinity/selector.)
I dont know how to do that
Is this a config file on your side or something I can change, if we had enterprise version?
Are you able to do screenshare to discuss this? I'm not sure I understand the k8 glue agent purpose.
yea IDK, the git repo is a python library. Is it possible to run something like pip install -e . so I can utilize the setup.py?
I was hoping to use docker_bash_setup_script but it didn't work when I ran pip install -e . in the respective script
When I run this line locally, it works finefrom sfi.imagery.models.chip_classifier.eval import eval_chip_classifier
Yea the "-e ." seems to fit this problem the best. On another note, It seems like whatever I add to docker_bash_setup_script is having no effect.
` PYTHONPATH: /home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-py/sfi:/home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-py:/home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-py/sfi/imagery/models/training::/home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-py/sfi:/usr/lib64/python37.zip:/usr/lib64/python3.7:/usr/lib64/python3.7/lib-dynload:/home/npuser/.clearml/venvs-builds/3.7/lib6...
` SysPath: ['/home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-py/sfi/imagery/models/training', '/home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-py/sfi', '/home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-py', '/usr/lib64/python37.zip', '/usr/lib64/python3.7', '/usr/lib64/python3.7/lib-dynload', '/home/npuser/.clearml/venvs-builds/3.7/lib64/python3.7/site-packages', '/home/npuser/.clearml/venvs-builds/3.7/l...
Traceback (most recent call last): File "sfi/imagery/models/training/ldc_train_end_to_end.py", line 26, in <module> from sfi.imagery.models.chip_classifier.eval import eval_chip_classifier ModuleNotFoundError: No module named 'sfi.imagery.models'
AgitatedDove14 How would I install using a setup.py in a clearML task?
Seems like it has everything I would need
For instance, In my repo, I have a setup.py, how would I run pip install -e .
Seems related to this https://github.com/allegroai/clearml/issues/241
The config change worked but my PYTHONPATH is still not working as expected
Yea I've done that already but I can do it again
When I run from sfi.imagery import models. It works fine local. So the repo is setup for proper imports. But fails in clearML tasks
basically, can I do local installs vs supplying a requirements.txt
note /home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-py is the correct path
Good question. The repo I'm using requires nvidia GPU and I can't get the code to run locally