Hey Guys. I Tried Running The Pytorch Mnist Example On A Train-Agent By Running It Locally And Then Resetting The Experiment And Then Enqueue-Ing It To The Default Queue. All Works Well But It Seems The Environment Building Process Gets Stuck On A Manual

Answered

Hey guys.
I tried running the pytorch mnist example on a train-agent by running it locally and then resetting the experiment and then enqueue-ing it to the default queue.
All works well but it seems the environment building process gets stuck on a manual flag triggered by an install (tzdata). Attached is a screen shot of the log.
Looking online brought up the following link ( https://rtfm.co.ua/en/docker-configure-tzdata-and-timezone-during-build/ ). It might be a problem with the docker that the agent builds for the experiment (if i understood how the agent works correctly). How do i go about resolving this?

Thanks again for all you help.

  				
Posted 
	4 years ago

					More
				  		
  Report
		
					ColossalAnt7
				
					0
					 × 1

Votes Newest

Answers 6

Im using the helm chart, but it might be 0.16.1 Ill try upgrading and get back to you.

  				
Posted 
	4 years ago

					More
				  		
  Report
		
					ColossalAnt7
				
					0
					 × 1

Hi ColossalAnt7 , I think we run into it on a few dockers, I believe the bug was fixed in the latest trains-agent RC. Could you verify please ?

  				
Posted 
	4 years ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

Looking at your helm repo, it seems that the 0.16.2 version has not been pushed yet.
Is this something you can resolve on your end, or should i download the source of the helm chart and change some version flags myself?

  				
Posted 
	4 years ago

					More
				  		
  Report
		
					ColossalAnt7
				
					0
					 × 1

You can try overriding the following in your values.yaml under agent section:
agentVersion: "==0.16.2rc1"

  				
Posted 
	4 years ago

					More
				  		
  Report
		
					FriendlySquid61
				
					0

This solved the problem. Thank you very much!

  				
Posted 
	4 years ago

					More
				  		
  Report
		
					ColossalAnt7
				
					0
					 × 1

Hey ColossalAnt7 ,
What version of trains-agent are you using?
You can try upgrading to the latest RC version, this issue should be fixed there:
pip install trains-agent==0.16.2rc1

  				
Posted 
	4 years ago

					More
				  		
  Report
		
					FriendlySquid61
				
					0

Write your answer

1K Views

6 Answers

4 years ago

one year ago