` PYTHONPATH: /home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-py/sfi:/home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-py:/home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-py/sfi/imagery/models/training::/home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-py/sfi:/usr/lib64/python37.zip:/usr/lib64/python3.7:/usr/lib64/python3.7/lib-dynload:/home/npuser/.clearml/venvs-builds/3.7/lib6...

I don't know how to get past this? My k8 pods shouldn't need to reach out to the public file server URL.

This is to address the PYTHONPATH issues

okay makes sense now, thanks!

yes but its limited

The pods should be able to use internal DNS names

Does that make sense

Traceback (most recent call last): File "sfi/imagery/models/training/ldc_train_end_to_end.py", line 26, in <module> from sfi.imagery.models.chip_classifier.eval import eval_chip_classifier ModuleNotFoundError: No module named 'sfi.imagery.models'

Okay, makes sense. So there is no copying of the data to the pod, it is simply references via the EFS

Okay I will try the CLI thanks

So this is an additional config file with enterprise? Is this new config file deployable via helm charts?

When I exec into the pod, it says I need sudo, but wondering if extra_docker_shell_script is executed as sudo already?

I don't recognize that file?

Okay so I just tried this and immediately I'm getting errors Failed to establish a new connection: because the file server URL in my clearml.conf is the k8 dns name. So I'm sort of stuck because if I revert it to the public DNS name, then upon Dataset.get I will get same failure.

Yes, as an example: My task starts up and checks the mounted EFS volume for x data, if x data does not exist there, it then pulls x data from S3.

Python 3.7.10 and clearml==1.7.0

okay great thanks

Can you fix this or should I open a PR. I'm blocked by this.

basically, can I do local installs vs supplying a requirements.txt

hmm that does look really helpful! Let me see if I can fix this using that info! thankyou!

kubernetes cluster

Basically, when I have the clearml sdk imported into my python module and I Task.init I'm getting that error only when running the experiment in the task pod.

Per my question above, what is the base path where the git repo is cloned?

Are there any work arounds to this issue? Our team is evaluating this product to potentially buy enterprise license. If we can't fetch data this is a problem.

yes, I see in the UI how to create a new queue. How do I associate that queue with a nodeSelector though?

Okay, seems like there are ways to do it, just need to be a bit clever

yea, does the enterprise version have more functionality like this?

Okay fixed that taint restriction

Made some progress getting the gpu nodes to provision, but got this error on my task K8S glue status: Unschedulable (0/4 nodes are available: 1 node(s) had taint { http://nvidia.com/gpu : true}, that the pod didn't tolerate, 3 node(s) didn't match Pod's node affinity/selector.)

How do I setup the clearml k8s glue?