Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi, I'M Trying To Run

Hi,
I'm trying to run clearml-agent in docker mode - however I'm having trouble saving the artifacts.
2022-08-25 05:06:21,820 - clearml.storage - ERROR - Failed creating storage object Reason: [Errno 2] No such file or directory: '/path/to/creds.json' 2022-08-25 05:06:21,820 - clearml.metrics - WARNING - Failed uploading to ('NoneType' object has no attribute 'upload_from_stream')In the console I can see the following lines:
sdk.google.storage.credentials.0.bucket = clearml-storage sdk.google.storage.credentials.0.project = my-project sdk.google.storage.credentials.0.credentials_json = /path/to/creds.jsonMy base image is nvidia/cuda:11.7.1-cudnn8-runtime-ubuntu20.04
is the problem that the config files reads the creds as sdk.google.storage.credentials.0.credentials_json and not sdk.google.storage.credentials_json Or maybe since i'm running in a docker mode - the creds are not copied into the container - if so how can I configure the task to always copy the creds file into the container?

  
  
Posted 2 years ago
Votes Newest

Answers 19


but it is on the host server

  
  
Posted 2 years ago

You do need to restart the agent

  
  
Posted 2 years ago

If you keep it as I wrote it, you'll need to modify the sdk configuration as well so it will know to look for it in the new place

  
  
Posted 2 years ago

Hi OutrageousSheep60 , the first problem is that clearml is trying to look for /path/to/creds.json which I assume is not a correct file path

  
  
Posted 2 years ago

google.storage { credentials = [ { bucket: "clearml-storage" project: "my-project" credentials_json: "/path/to/creds.json" }, ] }No - just emulating - it is more of /home/... /creds.json

  
  
Posted 2 years ago

Just to make sure I get your use-case, the agent itself is started on the host machine with the --docker command, right?

  
  
Posted 2 years ago

How do you do so?
can't find it under
clearml-agent daemon -h

  
  
Posted 2 years ago

Note that I mounted the file into the container in a different location (/root/creds.json) since I'm not sure it will be a good idea to keep the original path, but you can try it out

  
  
Posted 2 years ago

Shut it down, and start it up again

  
  
Posted 2 years ago

You actually have a file on the host called "/path/to/creds.json"?

  
  
Posted 2 years ago

i've doubled checked the Path- and it is correct

  
  
Posted 2 years ago

BTW SuccessfulKoala55 we are running with
CLEARML-AGENT version 1.3.0

  
  
Posted 2 years ago

yes - the agent is running with --docker
Great - where do I define the volume mount?
Should I build a base image that runs on the server and then use it as the base image in the container?

  
  
Posted 2 years ago

Btw -after updating clearml.conf do I need to restart the agent?
I can't see the additional tab under https://clearml.slack.com/archives/CTK20V944/p1658199530781499?thread_ts=1658166689.168039&cid=CTK20V944 , and I reran the task and got the same error

  
  
Posted 2 years ago

If so, you'll need to add a volume mount that will be applied to any docker container the agent spins up. You can do that by adding this in your clearml.conf file:
agent.extra_docker_arguments: ["-v", "/path/to/creds.json:/root/creds.json"]

  
  
Posted 2 years ago

Well, if the path is correct, than what you're missing is the volume mount since obviously the file is not present inside the docker container

  
  
Posted 2 years ago

or, you can just try:
agent.extra_docker_arguments: ["-v", "/path/to/creds.json:/path/to/creds.json"]

  
  
Posted 2 years ago

How are you configuring it? Can you share the storage related section of the clearml.conf?

  
  
Posted 2 years ago

Thx - it worked!
BTW - maybe it worth while to add this comment in the ClearML Agent daemon documentation - that when ever you update the clearml.conf you need to
clearml-agent daemon --stop recreate all the daemonclearml-agent daemon ....

  
  
Posted 2 years ago