agent.package_manager.system_site_packages

 can be used to inherit packages

Correct, it is basically venv with --system-site-packages

I do not think virtualenv nesting is support, if it was then in theory you could have executed the clearml-agent from virtual environment with system_site_packages turned on and then it would inherit from it. But again I'm not sure virtualenv supports it.
BTW: the latest clearml-agent RC already have venv caching (both pip/conda) 🙂