and I install the tar
I think the only way to do that is add it into the docker bash setup script (this is a bash script executed before Task)
If i were to push the private package to, say artifactory, is it possible to use that do the install?
If i were to push the private package to, say artifactory, is it possible to use that do the install?
Yes that's the recommended way 🙂
You add the private repo here, for the agent to use:
https://github.com/allegroai/clearml-agent/blob/e93384b99bdfd72a54cf2b68b3991b145b504b79/docs/clearml.conf#L65
Could you send the "installed packages" section of the Task that was created in the notebook ?
# Python 3.6.13 | packaged by conda-forge | (default, Feb 19 2021, 05:36:01) [GCC 9.3.0] argparse == 1.4.0 boto3 == 1.17.70 minerva == 0.1.0 torch == 1.7.1 torchvision == 0.8.2
When you install using pip <filename> you should end up with something like:minerva @ file://... or minerva @ https://...
The minerva one is my custom package AgitatedDove14
AgitatedDove14 - apologies for late reply. So to give context this in a Sagemaker notebook which has conda envs.
I use a lifecycle like this to pip install a package (a .tar.gz downloaded from s3) in a conda env- https://github.com/aws-samples/amazon-sagemaker-notebook-instance-lifecycle-config-samples/blob/master/scripts/install-pip-package-single-environment/on-start.sh
In the notebook I can do things like create experiments and so on. Now the problem is in running the cloned experiment and also for pipelines running remotely etc.
TrickySheep9 is this a conda package or a wheel you are installing manually ?
Very bad at python packaging terms - but I use pyproject.toml and python -m build that generates a wheel and a tar and I install the tar