Wherever running an experiment, it will install all required packages in a virtual environment to make sure the experiment is executed exactly as expected
And how do I change agent configuration file?
I just checked there are some uncommitted changes that I see in execution
I know, they represent the changes you made to the example script
However will it always install all the packages again and again? Is there any workaround for that?
I can see you failed experiments in the demo server, but I can't see any completed experiment from which they were cloned...
I ran it outside the examples folder and it works
Good ๐ I see you still have issues with your CUDA installation
But then how the normal one I.e without cloning worked well
Is there a difference in the uncommitted changes section before your changes and after?
However all packages are cached so it won't download again
It would be great if these issues are well elaborated in the documentation. Though the documentation is pretty good.
Anyways from what I see in the logs it shows agent.default_python = 3.7, cuda = 100, cudnn=75
I did that with the agent without making changes it works
Are you running the agent on the same machine?
the agent uses the same configuration file
You can first try to run your experiment again (not by cloning and running in the agent but by executing it again locally). If you like, you can copy the example and run it from another folder which is not located inside a git repo
Did you make sure the agent's default python version and cuda / cudnn are configured correctly?
I need to showcase this to my senior tomorrow
Yes. You can see the agent's configuration in the experiment's log - all values are printed there