Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi, I'M Trying To Run

Hi,
I'm trying to run clearml-agent in docker mode - however I'm having trouble saving the artifacts.
2022-08-25 05:06:21,820 - clearml.storage - ERROR - Failed creating storage object Reason: [Errno 2] No such file or directory: '/path/to/creds.json' 2022-08-25 05:06:21,820 - clearml.metrics - WARNING - Failed uploading to ('NoneType' object has no attribute 'upload_from_stream')In the console I can see the following lines:
sdk.google.storage.credentials.0.bucket = clearml-storage sdk.google.storage.credentials.0.project = my-project sdk.google.storage.credentials.0.credentials_json = /path/to/creds.jsonMy base image is nvidia/cuda:11.7.1-cudnn8-runtime-ubuntu20.04
is the problem that the config files reads the creds as sdk.google.storage.credentials.0.credentials_json and not sdk.google.storage.credentials_json Or maybe since i'm running in a docker mode - the creds are not copied into the container - if so how can I configure the task to always copy the creds file into the container?

  
  
Posted one year ago
Votes Newest

Answers 19


Just to make sure I get your use-case, the agent itself is started on the host machine with the --docker command, right?

  
  
Posted one year ago

You do need to restart the agent

  
  
Posted one year ago

Btw -after updating clearml.conf do I need to restart the agent?
I can't see the additional tab under https://clearml.slack.com/archives/CTK20V944/p1658199530781499?thread_ts=1658166689.168039&cid=CTK20V944 , and I reran the task and got the same error

  
  
Posted one year ago

BTW SuccessfulKoala55 we are running with
CLEARML-AGENT version 1.3.0

  
  
Posted one year ago

How are you configuring it? Can you share the storage related section of the clearml.conf?

  
  
Posted one year ago

but it is on the host server

  
  
Posted one year ago

google.storage { credentials = [ { bucket: "clearml-storage" project: "my-project" credentials_json: "/path/to/creds.json" }, ] }No - just emulating - it is more of /home/... /creds.json

  
  
Posted one year ago

Note that I mounted the file into the container in a different location (/root/creds.json) since I'm not sure it will be a good idea to keep the original path, but you can try it out

  
  
Posted one year ago

Well, if the path is correct, than what you're missing is the volume mount since obviously the file is not present inside the docker container

  
  
Posted one year ago

If you keep it as I wrote it, you'll need to modify the sdk configuration as well so it will know to look for it in the new place

  
  
Posted one year ago

How do you do so?
can't find it under
clearml-agent daemon -h

  
  
Posted one year ago

Thx - it worked!
BTW - maybe it worth while to add this comment in the ClearML Agent daemon documentation - that when ever you update the clearml.conf you need to
clearml-agent daemon --stop recreate all the daemonclearml-agent daemon ....

  
  
Posted one year ago

You actually have a file on the host called "/path/to/creds.json"?

  
  
Posted one year ago

Shut it down, and start it up again

  
  
Posted one year ago

If so, you'll need to add a volume mount that will be applied to any docker container the agent spins up. You can do that by adding this in your clearml.conf file:
agent.extra_docker_arguments: ["-v", "/path/to/creds.json:/root/creds.json"]

  
  
Posted one year ago

Hi OutrageousSheep60 , the first problem is that clearml is trying to look for /path/to/creds.json which I assume is not a correct file path

  
  
Posted one year ago

i've doubled checked the Path- and it is correct

  
  
Posted one year ago

or, you can just try:
agent.extra_docker_arguments: ["-v", "/path/to/creds.json:/path/to/creds.json"]

  
  
Posted one year ago

yes - the agent is running with --docker
Great - where do I define the volume mount?
Should I build a base image that runs on the server and then use it as the base image in the container?

  
  
Posted one year ago