Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
When I Do

When I do Dataset.get why does the SDK use clearml.storage - ERROR - Could not download http://files.clearml.myhost.com vs using what is defined on the agent such as http://clearml-fileserver.clearml.svc.cluster.local:8081 ?

  
  
Posted one year ago
Votes Newest

Answers 30


BoredHedgehog47 that actually depends on the container, are you running as root inside the container ?
if not I think the easiest hack is to always map /etc/hosts as a k8s secret file?

  
  
Posted one year ago

When I exec into the pod, it says I need sudo, but wondering if extra_docker_shell_script is executed as sudo already?

  
  
Posted one year ago

AgitatedDove14 Will I need sudo permissions if I add this script to extra_docker_shell_script
echo "192.241.xx.xx venus.example.com venus" >> /etc/hosts

  
  
Posted one year ago

file and redirect the public url to k8 dns url?

Yes! that would work, Nice!
You can add it into the extra_docker_shell_script it will be executed in any pod the clearml-glue will spin (obviously this needs to be configured on the pod running the clearml k8s glue)
https://github.com/allegroai/clearml-agent/blob/ba2db4e727b90e595df2b13f458d9580659bf12e/docs/clearml.conf#L152

  
  
Posted one year ago

I suppose a short term hack would to just edit the /etc/hosts file and redirect the public url to k8 dns url?

  
  
Posted one year ago

he problem is due to tight security on this k8 cluster, the k8 pod cannot reach the public file server url which is associated with the dataset.

Understood, that makes sense, if this is the case then the path_substitution feature is exactly what you are looking for

  
  
Posted one year ago

When I deployed the webserver, I changed the value https://github.com/allegroai/clearml-helm-charts/blob/main/charts/clearml/values.yaml#L36 to be the public file server URL. Then in the UI, I copied the blob from the settings/API keys. Which had the public URLs. After that I did my data uploads which worked fine as they used public URLs. The problem is due to tight security on this k8 cluster, the k8 pod cannot reach the public file server url which is associated with the dataset.

  
  
Posted one year ago

Just curious, if

is a value I can set, where is it used?

It is used when Creating a dataset from inside the cluster (i.e. when launching using the clearml k8s glue),
it will have No effect on what users have on their local machines
i.e. they can always point to a diff server.

That said, when users create their initial clearml.conf and copy paste the info from the web UI, this value (or it might be another one, I'll double check later) will set the initial configuration the copy paste into their local clearml.conf.

Which I think what happened is you had a diff one set when you spinned it initially, this was the configuration you used to create your local clearml.conf, then you changed it but your clearml.conf stayed the same.
wdyt?

  
  
Posted one year ago

Just curious, if https://github.com/allegroai/clearml-helm-charts/blob/19a6785a03b780c2d22da1e79bcd69ac9ffcd839/charts/clearml-agent/values.yaml#L50 is a value I can set, where is it used? It would be great if it overrides the Dataset.get embedded url parsed from my clearml conf file

  
  
Posted one year ago

So this is an additional config file with enterprise?

Extension to the "clearml.conf" capabilities

Is this new config file deployable via helm charts?

Yes, you can also set it company/user wide using the clearml Vault feature (again enterprise, sorry 😞 )

  
  
Posted one year ago

So this is an additional config file with enterprise? Is this new config file deployable via helm charts?

  
  
Posted one year ago

is this a config file on your side or something I can change, if we had enterprise version?

Yes, this is one of the things you can configure

  
  
Posted one year ago

Is this a config file on your side or something I can change, if we had enterprise version?

  
  
Posted one year ago

I don't recognize that file?

  
  
Posted one year ago

So you could change it down the road if infra/hosting changes.

Internally this is doable and Enterprise edition supports it, at the end this is stored in DBs πŸ™‚

Also in this case, I'm uploading the data to the public file server URL, but my k8 pod can't reach that for security reasons.

Yes, this is solvable as well (again sorry for pointing it, but only in the enterprise version), where you can specify per client or globally:
` path_substitution = [
# Replace registered links with local prefixes,
# Solve mapping issues, and allow for external resource caching.

        # {
        #     registered_prefix = " ` ` "
        #     local_prefix = "file:///mnt/shared/bucket/research
        # },
        # {
        #     registered_prefix = "file:///mnt/shared/folder/"
        #     local_prefix = "file:///home/user/shared/folder"
        # }
    ] `Wdyt?
  
  
Posted one year ago

Are there any work arounds to this issue? Our team is evaluating this product to potentially buy enterprise license. If we can't fetch data this is a problem.

  
  
Posted one year ago

I think the best change would to respect the value set https://github.com/allegroai/clearml-helm-charts/blob/19a6785a03b780c2d22da1e79bcd69ac9ffcd839/charts/clearml-agent/values.yaml#L50 so you could change it down the road if infra/hosting changes. Also in this case, I'm uploading the data to the public file server URL, but my k8 pod can't reach that for security reasons.

  
  
Posted one year ago

You could change infrastructure or hosting, and now your data is associated with the wrong URL

Yeah that makes sense, so have it on a specific dns name? (this is usually the case with k8s deployments)

  
  
Posted one year ago

The pods should be able to use internal DNS names

  
  
Posted one year ago

I don't know how to get past this? My k8 pods shouldn't need to reach out to the public file server URL.

  
  
Posted one year ago

You could change infrastructure or hosting, and now your data is associated with the wrong URL

  
  
Posted one year ago

URLs that it was uploaded with, as that URL could change.

How would that change, the actual files are there ?

  
  
Posted one year ago

IMO, the dataset shouldnt be tied to the clearml.conf URLs that it was uploaded with, as that URL could change. It should respect the file server URL the agent has.

  
  
Posted one year ago

Does that make sense

  
  
Posted one year ago

Okay so I just tried this and immediately I'm getting errors Failed to establish a new connection: because the file server URL in my clearml.conf is the k8 dns name. So I'm sort of stuck because if I revert it to the public DNS name, then upon Dataset.get I will get same failure.

  
  
Posted one year ago

Yes exactly !

  
  
Posted one year ago

ahhh its possible my clearml.conf was using the public urls when I made it. Let me try this

  
  
Posted one year ago

When a remote task runs

Dataset.get()

it is not using the correct URL

BoredHedgehog47 it will get the link the data was Registered with, when creating the Dataset.
This has Nothing to do with the local configuration, it can point to any arbitrary file location on the internet.
It was created there, because at the time of the dataset creation someone (manually or via the config) set a specific host as the file location, and to that host the files were uploaded (again you can have a mixture of hosts, the backend is not aware of them, they just keep it as string)
Make sense ?

  
  
Posted one year ago

If you submit your task using the CLI does it still work?

  
  
Posted one year ago

I guess I am out of ideas. The config is wrong somewhere. Maybe double check all the configs? It’s taking the value from somewhere!

  
  
Posted one year ago
548 Views
30 Answers
one year ago
one year ago
Tags