I'm also beginning to think this is related to https://clearml.slack.com/archives/CTK20V944/p1620664770492400 . Previously when i set force_repo_requirements_txt=true and system_site_packages: true , it seems to work. upgrading to v1.02 seems to change things.

Hi AgitatedDove14 , i dug a bitt deeper. I saw this in installed packages in the original completed task. When the task is cloned, this is copied over and thus the problem. Can i ask, how ClearML create the list of installed packages? Why is it that some of them (E.g. attr is being pulled from @ file:///tmp/build/80754af9/attrs_1604765588209/work)

absl-py==0.11.0 alabaster==0.7.12 antlr4-python3-runtime==4.8 apex==0.1 appdirs==1.4.4 argon2-cffi==20.1.0 ascii-graph==1.5.1 async-generator==1.10 attrs @ file:///tmp/build/80754af9/attrs_1604765588209/work audioread==2.1.9 Babel==2.9.0 backcall @ file:///home/ktietz/src/ci/backcall_1611930011877/work beautifulsoup4 @ file:///home/linux1/recipes/ci/beautifulsoup4_1610988766420/work black==21.4b2 bleach==3.3.0 blis==0.4.1 boto3==1.17.10 botocore==1.20.10 brotlipy==0.7.0 catalogue==1.0.0 certifi==2020.12.5 cffi @ file:///tmp/build/80754af9/cffi_1605538068321/work chardet @ file:///tmp/build/80754af9/chardet_1605303185383/work clearml==1.0.2 click==7.1.2 cloudpickle==1.6.0 cmake==3.18.4.post1 codecov==2.1.11 conda==4.9.2 conda-build==3.21.4 conda-package-handling @ file:///tmp/build/80754af9/conda-package-handling_1603018141399/work coverage==5.4 cryptography @ file:///tmp/build/80754af9/cryptography_1605544487601/work cxxfilt==0.2.2 cycler==0.10.0 cymem @ file:///tmp/build/80754af9/cymem_1613319259039/work Cython==0.28.4 DataProperty==0.50.0 decorator @ file:///home/ktietz/src/ci/decorator_1611930055503/work defusedxml==0.6.0 -e git+ distlib==0.3.1 DLLogger @ docutils==0.16 entrypoints==0.3 filelock @ file:///home/linux1/recipes/ci/filelock_1610993975404/work flake8==3.7.9 Flask==1.1.2 furl==2.1.2 future==0.18.2 fvcore @ git+ glob2 @ file:///home/linux1/recipes/ci/glob2_1610991677669/work graphsurgeon @ file:///workspace/TensorRT-7.2.2.3/graphsurgeon/graphsurgeon-0.4.5-py2.py3-none-any.whl grpcio==1.35.0 h5py==3.1.0 html2text==2020.1.16 humanfriendly==9.1 hydra-core==1.1.0.dev7 hypothesis==4.50.8 idna @ file:///tmp/build/80754af9/idna_1593446292537/work imageio==2.9.0 imagesize==1.2.0 importlib-metadata @ file:///tmp/build/80754af9/importlib-metadata_1602276842396/work importlib-resources==5.1.2 inflect==5.0.2 iniconfig==1.1.1 iopath==0.1.8 ipdb==0.13.4 ipykernel==5.4.3 ipython @ file:///tmp/build/80754af9/ipython_1612212923938/work ipython-genutils @ file:///tmp/build/80754af9/ipython_genutils_1606773439826/work itsdangerous==1.1.0 jedi==0.17.0 Jinja2 @ file:///tmp/build/80754af9/jinja2_1612213139570/work jmespath==0.10.0 joblib==1.0.1 json5==0.9.5 jsonschema @ file:///tmp/build/80754af9/jsonschema_1594303806266/work jupyter-client==6.1.11 jupyter-core==4.7.1 jupyter-tensorboard==0.2.0 jupyterlab==1.2.14 jupyterlab-pygments==0.1.2 jupyterlab-server==1.2.0 jupytext==1.10.2 kiwisolver==1.3.1 libarchive-c @ file:///home/linux1/recipes/ci/python-libarchive-c_1610974153025/work librosa==0.8.0 llvmlite==0.34.0 lmdb==1.1.1 Mako==1.1.4 Markdown==3.3.3 markdown-it-py==0.6.2 MarkupSafe==1.1.1 maskrcnn-benchmark @ file:///opt/pytorch/examples/maskrcnn/pytorch matplotlib==3.3.4 mbstrdecoder==1.0.1 mccabe==0.6.1 mdit-py-plugins==0.2.5 mistune==0.8.4 mlperf-compliance==0.0.10 mock @ file:///tmp/build/80754af9/mock_1607622725907/work msgfy==0.1.0 murmurhash @ file:///tmp/build/80754af9/murmurhash_1607456108764/work mypy-extensions==0.4.3 nbclient==0.5.2 nbconvert==6.0.7 nbformat==5.1.2 nest-asyncio==1.5.1 networkx==2.0 nltk==3.5 notebook==6.2.0 numba @ file:///tmp/build/80754af9/numba_1600100669015/work numpy==1.20.3 nvidia-dali-cuda110==0.29.0 nvidia-pyprof @ git+ nvidia-tensorboard @ file:///nvidia/opt/tensorboard_install/nvidia_tensorboard-1.15.0%2Bnv21.02-py3-none-any.whl nvidia-tensorboard-plugin-dlprof @ file:///nvidia/opt/tensorboard_install/nvidia_tensorboard_plugin_dlprof-0.11-py3-none-any.whl omegaconf==2.1.0.dev27 onnx @ file:///opt/pytorch/pytorch/third_party/onnx onnxruntime==1.6.0 opencv-python==4.4.0.46 orderedmultidict==1.0.1 packaging==20.9 pandas==1.1.4 pandocfilters==1.4.3 parso @ file:///tmp/build/80754af9/parso_1607623074025/work pathlib2==2.3.5 pathspec==0.8.1 pathvalidate==2.3.2 pexpect @ file:///tmp/build/80754af9/pexpect_1605563209008/work pickleshare @ file:///tmp/build/80754af9/pickleshare_1606932040724/work Pillow==8.2.0 Pillow-SIMD @ file:///tmp/pillow-simd pkginfo==1.7.0 plac @ file:///tmp/build/80754af9/plac_1594259967336/work pluggy==0.13.1 polygraphy==0.23.4 pooch==1.3.0 portalocker==2.3.0 preshed==3.0.2 progressbar==2.5 prometheus-client==0.9.0 prompt-toolkit @ file:///tmp/build/80754af9/prompt-toolkit_1602688806899/work protobuf==3.14.0 psutil @ file:///tmp/build/80754af9/psutil_1612298023621/work ptyprocess @ file:///tmp/build/80754af9/ptyprocess_1609355006118/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl py==1.10.0 pybind11==2.6.2 pycocotools==2.0.2 pycodestyle==2.5.0 pycosat==0.6.3 pycparser @ file:///tmp/build/80754af9/pycparser_1594388511720/work pycuda==2020.1 pydot==1.4.2 pyflakes==2.1.1 Pygments @ file:///tmp/build/80754af9/pygments_1610565767015/work pyhocon==0.3.57 PyJWT==2.0.1 pynvml==8.0.4 pyOpenSSL @ file:///tmp/build/80754af9/pyopenssl_1605545627475/work pyparsing==2.4.7 pyrsistent @ file:///tmp/build/80754af9/pyrsistent_1600141720057/work PySocks @ file:///tmp/build/80754af9/pysocks_1605305779399/work pytablewriter==0.47.0 pytest==6.2.2 pytest-cov==2.11.1 pytest-pythonpath==0.7.3 python-dateutil==2.8.1 python-hostlist==1.21 python-nvd3==0.15.0 python-slugify==4.0.1 pytools==2021.1 pytorch-transformers==1.1.0 pytz @ file:///tmp/build/80754af9/pytz_1612215392582/work PyWavelets==1.1.1 PyYAML==5.3.1 pyzmq==22.0.3 regex==2020.11.13 requests @ file:///tmp/build/80754af9/requests_1592841827918/work resampy==0.2.2 revtok @ ruamel-yaml==0.15.87 s3transfer==0.3.4 sacrebleu==1.2.10 sacremoses==0.0.35 scikit-image==0.15.0 scikit-learn==0.24.1 scipy @ file:///tmp/build/80754af9/scipy_1612469469862/work Send2Trash==1.5.0 sentencepiece==0.1.95 six @ file:///tmp/build/80754af9/six_1605205327372/work snowballstemmer==2.1.0 SoundFile==0.10.3.post1 soupsieve @ file:///tmp/build/80754af9/soupsieve_1607965878077/work sox==1.4.1 spacy @ file:///tmp/build/80754af9/spacy_1608321098157/work Sphinx==3.5.1 sphinx-rtd-theme==0.5.1 sphinxcontrib-applehelp==1.0.2 sphinxcontrib-devhelp==1.0.2 sphinxcontrib-htmlhelp==1.0.3 sphinxcontrib-jsmath==1.0.1 sphinxcontrib-qthelp==1.0.3 sphinxcontrib-serializinghtml==1.1.4 srsly @ file:///tmp/build/80754af9/srsly_1607548537638/work subword-nmt @ tabledata==1.1.3 tabulate==0.8.8 tensorboard @ tensorrt @ file:///workspace/TensorRT-7.2.2.3/python/tensorrt-7.2.2.3-cp38-none-linux_x86_64.whl termcolor==1.1.0 terminado==0.9.2 testpath==0.4.4 text-unidecode==1.3 thinc @ file:///tmp/build/80754af9/thinc_1607710152385/work threadpoolctl==2.1.0 toml==0.10.2 torch @ file:///opt/pytorch/pytorch torchtext @ file:///opt/pytorch/text torchvision @ file:///opt/pytorch/vision tornado==6.1 tqdm==4.53.0 traitlets @ file:///home/ktietz/src/ci/traitlets_1611929699868/work typepy==1.1.4 typing==3.7.4.3 typing-extensions==3.7.4.3 uff @ file:///workspace/TensorRT-7.2.2.3/uff/uff-0.6.9-py2.py3-none-any.whl Unidecode==1.2.0 urllib3 @ file:///tmp/build/80754af9/urllib3_1603305693037/work virtualenv==20.4.6 wasabi @ file:///tmp/build/80754af9/wasabi_1612219178408/work wcwidth @ file:///tmp/build/80754af9/wcwidth_1593447189090/work webencodings==0.5.1 Werkzeug==1.0.1 wrapt==1.10.11 yacs==0.1.8 zipp @ file:///tmp/build/80754af9/zipp_1604001098328/work

SubstantialElk6
Regrading cloning the executed Task:
In the pip requirements syntax, "@" is a hint that tells pip where to find the package if it is not preinstalled.
Usually when you find the @ /tmp/folder It means the packages was preinstalled (usually pre installed in the docker).
What is the exact scenario that caused it to appear (this was always the case, before v1 as well).
For example zipp package is installed from pypi be default and not from local temp file.
Your fix basically means the "installed section" is ignored, regardless of what it contains.
Notice that when you clone the Task, you can always clear the "Installed Packages" section and it will just default to the "requirements.txt" from the repository.

Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/tmp/build/80754af9/attrs_1604765588209/work'Seems like pip failed creating a folder
Could it be you are out of space ?

Space is way above nominal. What created this folder that it's trying to process? What processing is this?
Processing /tmp/build/80754af9/attrs_1604765588209/workIs there any paths in the agent machine that i can clear out to remove any possible issues from previous versions?