Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi, I'M Having Some Trouble With Trains-Agent In Docker Mode With A Local Trains Server. I Pulled Allegroai/Trains-Agent:Latest And Spun It Up In A Container, Set The Appropriate Environment Variables To Point To My Trains Server, And Bind Mounted /Var/Ru

Hi, I'm having some trouble with trains-agent in docker mode with a local trains server. I pulled allegroai/trains-agent:latest and spun it up in a container, set the appropriate environment variables to point to my trains server, and bind mounted /var/run/docker.sock:/var/run/docker.sock so it can spin up docker containers for each experiment. From the web ui I can successfully queue experiments, but they fail with task not found errors because their containers don't seem to inherit the the local server configuration from the trains-agent container. I've done some debugging using the 'extra_docker_shell_script' option, and I can see that experiment containers end up with an empty directory mounted to /root/trains.conf. I have the strong suspicion that I'm missing some obvious configuration and that when trains-agent spins up each experiment's container there's a value missing for doing a bind mount to /root/trains.conf. When I run trains-agent outside of docker, I never have any problems and everything works great.

  
  
Posted 3 years ago
Votes Newest

Answers 9


I can see the trains-agent listed as a machine in the UI. I can also send experiments to the queue and the agent picks them up.

  
  
Posted 3 years ago

Just having TRAINS_AGENT_DOCKER_HOST_MOUNT defined for the agent docker wasn't enough, but after making sure the source directory was mounted from somewhere outside of docker everything works.

  
  
Posted 3 years ago

I added TRAINS_AGENT_DOCKER_HOST_MOUNT="/root/.trains:/root/.trains" to the trains-agent docker. I also started that docker with a bind-mount to make sure /root/.trains inside the agent docker pointed to somewhere on the house outside of any container.

  
  
Posted 3 years ago

RobustGoldfish9
I think you need to set the trains-agent docker to be aware of the host, so it knows how to mount data/cache/configurations into the sibling docker

It should look something like:
TRAINS_AGENT_DOCKER_HOST_MOUNT="/mnt/host/data:/root/.trains"So if running a docker:
docker run -e TRAINS_AGENT_DOCKER_HOST_MOUNT="/mnt/host/data:/root/.trains" ...

  
  
Posted 3 years ago

RobustGoldfish9 do you see the trains-agent listed as a machine in the UI (under workers)

  
  
Posted 3 years ago

house == host.

  
  
Posted 3 years ago

Hi RobustGoldfish9 Kudos on the mount, and my apologies for forgetting to mention it.
You are absolutely right, I'll make sure we have it in the documentation, there is no way to know that obscure env variable 🙂

  
  
Posted 3 years ago

Thank you very much for the assistance! I don't think I would have known about that environment variable without asking here on the slack channel.

  
  
Posted 3 years ago

That did the trick.

  
  
Posted 3 years ago
565 Views
9 Answers
3 years ago
one year ago
Tags