Yea this is a bug

Hey triggering the tasks from the CLI resolved the python pathing issues!

AgitatedDove14 How would I install using a setup.py in a clearML task?

note /home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-py is the correct path

I typically install latest

Would that go under arguments ?

During the task pod runtime

https://github.com/allegroai/clearml-helm-charts/blob/main/charts/clearml-agent/values.yaml#L35-L101

I don't recognize that file?

When I deployed the webserver, I changed the value https://github.com/allegroai/clearml-helm-charts/blob/main/charts/clearml/values.yaml#L36 to be the public file server URL. Then in the UI, I copied the blob from the settings/API keys. Which had the public URLs. After that I did my data uploads which worked fine as they used public URLs. The problem is due to tight security on this k8 cluster, the k8 pod cannot reach the public file server url which is associated with the dataset.

My next question, how do I add more queues?

👍

Seems like it has everything I would need

If you look lower, it is there '/home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-py'

For instance, if I wanted the default queue and gpu queue that I create, how do I do that?

Yes that fixed it, thanks!

Can you fix this or should I open a PR. I'm blocked by this.

Or is there an easier way?

hey there, it happens in the task pod

kubernetes cluster

It seems like the clearml python sdk might have issues when a subprocess is opened?

{"asctime": "2022-09-28 18:45:55,353", "levelname": "INFO", "name": "root", "module": "ldc_train_end_to_end", "threadName": "MainThread", "message": "Training classifier with command:\npython -m sfi.imagery.models.bbox_predictorv2.train ./sfi/imagery/models/training/train_config.json", "filename": "ldc_train_end_to_end.py", "funcName": "train_model"} File "/usr/lib64/python3.7/site.py", line 177 file=sys.stderr) ^ SyntaxError: invalid syntax

hmm that does look really helpful! Let me see if I can fix this using that info! thankyou!

` PYTHONPATH: /home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-py/sfi:/home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-py:/home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-py/sfi/imagery/models/training::/home/npuser/.clearml/venvs-builds/3.7/task_repository/commons-imagery-models-py/sfi:/usr/lib64/python37.zip:/usr/lib64/python3.7:/usr/lib64/python3.7/lib-dynload:/home/npuser/.clearml/venvs-builds/3.7/lib6...

"additionalInfo": { "inBytes": "438", "localPort": "9134", "outBytes": "401", "unusual": "80", "value": "{\"inBytes\":\"438\",\"localPort\":\"9134\",\"outBytes\":\"401\",\"unusual\":\"80\"}", "type": "default" },

Could I simply just reference the files by name and pass in a string such as ~/.clearml/my_file.json

https://github.com/allegroai/clearml-helm-charts/blob/main/charts/clearml-agent/values.yaml#L61

Also I'd like to create the queues pragmatically, is that possible?

Okay so I just tried this and immediately I'm getting errors Failed to establish a new connection: because the file server URL in my clearml.conf is the k8 dns name. So I'm sort of stuck because if I revert it to the public DNS name, then upon Dataset.get I will get same failure.

Yes I will try that