Hi,
It did, nvidia/cuda:10.1-runtime-ubuntu18.04.

So if i need to set this every time, what is the following config for? And how do i pass in new env parameters?
` default_docker: {
# default docker image to use when running in docker mode
image: "dockerrepo/mydocker:custom"

    # optional arguments to pass to docker image
    # arguments: ["--ipc=host", ]
    arguments: ["--env GIT_SSL_NO_VERIFY=true",]
} `

Ok thanks, that worked.

which clearml.conf is it refering to? I'm executing on my client, which is then remotely executed by the agent. Both of them has ~/clearml.conf.

does the bash script need clearml-agent to be able to communicate to the https clearml-server first? If yes, there's a chicken/egg problem here.

Thanks this would be a good alternative before the enterprise version comes in. How is this different from argparser btw?

Yup that works. Thanks.

thanks SuccessfulKoala55 . I verified your last comment and it works.

what feature on this paid roadmap are you referring to? I am indeed communicating with Noem on paid features.

Hi,
I'm running on Dell ECS storage appliance, which offers S3 compatibility.
yes http://ECS.ai is the DNS name of the server.
ClearML-models is the bucket.
Let me try with ip:port.

I meant the dataset id.

Hi, i changed it, but it still point to https://files.pythonhosted.org/packages .

No, i can't see the files. But i can see if i don't use ':port' in the URL when uploading. I can't access the machine today, i'll try to check the S3 logs when i'm back.

the hackathon is 3 days.

And any roadmap on this? The organisation's on ssh auth is firm. This can end up not possible to use ClearML for remote execution.

Hi ResponsiveHedgehong88 , I was trying to do the same thing but the loggerhook doesn't seem to work. The console log and scalar logs didn't come out when I registered via init.py and load via log_config. Are you able to share how you configure it?

thanks. That seems to work. I got a question, does it save the best model or the model in the last epoch?

Thanks SuccessfulKoala55 . I can try my hand on a patch. But the pod spinning is handled by the k8s glue, which has no link to the client side. How should the client pass the key over to k8s glue during runtime via clearml server?

Hi, scenario as follows.

client.py runs task.execute_remotely(queue='myqueue', exit_process=True) The API section of clearml.conf at client side is read in. client side calls clearml server and insert task into queue. K8S glue retrieves task from queue. Spawn a K8S pod. K8S pod performs git clone Error. ssh keys not found.
Each individual has their own key in the gitlab profile and gitlab is configured to only work via ssh.
We can't place the key in the image as this is as good as ...

Yes of cos, its a long one.

Hi yes, still getting the SSLs. It looks like some incompatibility with the OS ssl libraries.

I thought of another potential way but not sure if the SDK supports it.
We will perform manual save and upload of model using vanilla boto3 and credentials passed in as env var. Use ClearML SDK to update the Model Repo on the location of the model, without ClearML uploading it explicitly.Would the above work?

Any idea where i can find the relevant API calls for this?

Sorry AgitatedDove14 can you bump me to that thread?

Setting the credentials on agent machine means the users cannot use their own credentials since an k8s glue agent serves multiple users.

Referencing your suggestion, we can configure output_uri on task.set_base_docker() but how should we do this for the credentials?

Going back to the open source, I think that adding the credentials as part of the source code might allow to have "credentials" auto populate as part of the remote execution, wdyt?

Not sure how this will work when i can't supply the credentials to ClearML programatically.

I didn't track the version on this change in behaviour. But last I tried it was able to download the content after I provide the credentials.

My assumption is that the agent will have pulled that off the client's clearml.conf.

Can i dig into the mongodb or ES to pull these data?

f you can directly access the machine running the agent, yes you could. If not reverse proxy is in the working 

Hi AgitatedDove14 , i might have misunderstood your previous comment above. Do you mean that clearml-session can only work regardless of whether xforwarding is configured, if we have direct access to the Kubernetes worker when we run K8S glue?

We did some testing today and clearml-session tried to tunnel with a k8s cluster ip, and thus failed.

If we setup a ingress with Me...

Unfortunately due to security, clients can't have direct access to the nodes. Is there any possible workarounds at the moment?