I just checked there are some uncommitted changes that I see in execution

I know, they represent the changes you made to the example script

So now what do I do?

I need to showcase this to my senior tomorrow

You can first try to run your experiment again (not by cloning and running in the agent but by executing it again locally). If you like, you can copy the example and run it from another folder which is not located inside a git repo

I did that with the agent without making changes it works

It fails only when I make any changes

Is there a difference in the uncommitted changes section before your changes and after?

I can see you failed experiments in the demo server, but I can't see any completed experiment from which they were cloned...

I ran it outside the examples folder and it works

Good 🙂 I see you still have issues with your CUDA installation

However will it always install all the packages again and again? Is there any workaround for that?

Yes 😂

But then how the normal one I.e without cloning worked well

Wherever running an experiment, it will install all required packages in a virtual environment to make sure the experiment is executed exactly as expected

However all packages are cached so it won't download again

Are you running the agent on the same machine?

Yes very much

Did you make sure the agent's default python version and cuda / cudnn are configured correctly?

In the conf file?

Yes. You can see the agent's configuration in the experiment's log - all values are printed there

And how do I change agent configuration file?

I only see trains conf file

It would be great if these issues are well elaborated in the documentation. Though the documentation is pretty good.

the agent uses the same configuration file

Anyways from what I see in the logs it shows agent.default_python = 3.7, cuda = 100, cudnn=75

https://www.allegro.ai/docs/deploying_trains/trains_agent_install_configure/

and are these the correct values?

No

It worked finally 🙂