if in the "installed packages" I have all the packages installed from the requirements.txt than I guess I can clone it and use "installed packages"
After the agent finished installing the "requirements.txt" it will put back the entire "pip freeze" into the "installed packages", this means that later we will be able to fully reproduce the working environment, even if packages change (which will eventually happen as we cannot expect everyone to constantly freeze versions)
My problem right now is that Pytorch Lightning need the s3fs package to store model checkpoint into s3 buckets, but in my "installed packages" is not imported and I get an import error
You can always manually add a package from code if it is missing, call Task.add_requirements
before the Task.init call, of just add import
🙂 "Pytorch Lightning need the s3fs " s3fs is not needed, let PL store the model locally and use "output_uri" to automatically upload the model to your S3 bucket. This way not only you will later be able to switch to any other object storage, you will also have the ability to log it and download the model from the UI, and have better control over the S3 credentials / security LovelyHamster1 WDYT?