All right got it, will try it out. Thanks for the quick response.

Thanks!

AlertBlackbird30 - got it running. Few comments:

Nodeport is set by default despite being parameter in values.yml. For example:` webserver:
extraEnvs: []

service:
type: NodePort
port: 80 `2. Ingress was using 8080 for webserver but service was 80
3. Had to change path in ingress to “/*” instead of “/” to get it working for me

The helm chart installs a agentservice, how is that related if at all?

AgitatedDove14 - is my understanding right that we have to call pipe.wait() ?

Only one. Will replicate it in detail and see what’s actually up

Fix - use shutil.copy instead of shutil.copy2 - verifying now.

Ok that wasn’t it 😞

Will try it out. A weird one this.

AgitatedDove14 - this was an interesting one. I think I have found the issue, but verifying the fix as of now.

One of the devs was using shutil.copy2 to copy parts of dataset to a temporary directory in a with block - something like:

with TemporaryDirectory(dir=temp_dir) as certificates_directory: for file in test_paths: shutil.copy2(f"{dataset_local}/{file}", f"{certificates_directory}/file")
My suspicion is since copy2 copies with full data and symlin...

AgitatedDove14 - worked with mutable copy! So was definitely related to the symlinks in some form

AgitatedDove14 you had mentioned some pipeline related changes coming, are they in?

+1. The k8s helper needs to be more k8s native.

Can makes some PRs around this. Playing with some changes already for my setup

Ah ok. Kind of getting it, will have to try the glue mode

I would like to create a notebook instance and start using it without having to do anything on a dev box

There is kubectl exec 🙂

AlertBlackbird30 :

--remote-gateway [REMOTE_GATEWAY] Advanced: Specify gateway ip/address to be passed to interactive session (for use with k8s ingestion / ELB)I see this in clearml-session - what’s the intent here?

But ok the summary is I guess it doesn’t work in a k8s env

IF there’s a post-task script, I can add a way to zip and upload pip cache etc to s3 - as in do any caching that I want without having first class support in clearml

Initially thought use_current_task in Dataset.create does it, but that seems to set the DataprocessingTask itself to the current task

Ah ok got it.

I would prefer controlled behavior than some available version being used. Here triggered a bunch of jobs that all went fine and even evaluations were fine and then when we triggered a inference deploy it failed

Ah thanks for the pointer AgitatedDove14

That’s about what i do

yeah

Lot of time had passed though

Sounds great!

Is nested projects in the release? I see it in the community server but no mention in the blog or the release notes?

Yes, for datasets where we need GDPR compliance

Here’s an example - https://github.com/allegroai/clearml-serving/blob/b5f5d72046f878bd09505606ca1147d93a5df069/clearml_serving/serving_service.py#L58