When initializing your task, simply make sure to turn off the auto-connect for pytorch, using Task.init(..., auto_connect_frameworks={'pytorch': False})