Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
***** Solved ***** It Appears That Clearml Hasn'T Automatically Captured The Requirements.Txt File In The Cloned Repository. Adding The Following Line Of Code At The Beginning Of My Script Fixed That:

***** solved *****
it appears that ClearML hasn't automatically captured the requirements.txt file in the cloned repository.
adding the following line of code at the beginning of my script fixed that:
Task.add_requirements('../../../requirements.txt')
if you can point out why it is not able to auto capture the requirements file, pls share 🙂

Hi, for some reason my requirements are not installed as specified in my requirements.txt file.
my requirements:

GitPython>=3.1.20
h5py==3.8.0
holoviews[extras]>=1.15.4
joblib==1.2.0
lightgbm==3.3.5
mat73==0.60
matplotlib==3.7.1
numpy>=1.23.5
pytest==7.3.1
scikit_learn==1.3.0
scipy==1.10.1
setuptools>=66.0.0
Shapely==2.0.1
sktime>=0.19.2
statsmodels>=0.13.5
tqdm==4.65.0
tsfel==0.1.5
tsfresh==0.20.0
pandas>=2.0.1
torch>=2.0.1
torchdata>=0.6.1
clearml>=1.11.1
tensorboard>=2.6.0
pyarrow>=11.0.0
fastparquet>=2023.7.0

the installed req (taken from the execution tab in clearml ui):

absl-py==1.4.0
alabaster==0.7.13
asttokens==2.2.1
attrs==23.1.0
Babel==2.12.1
backcall==0.2.0
bleach==6.0.0
bokeh==2.4.3
boto3==1.26.158
botocore==1.29.158
cachetools==5.3.1
certifi==2023.7.22
charset-normalizer==3.2.0
clearml==1.11.1
click==8.1.6
cloudpickle==2.2.1
colorcet==3.0.1
contourpy==1.1.0
cycler==0.11.0
Cython==3.0.0
dask==2023.8.0
decorator==5.1.1
Deprecated==1.2.14
distributed==2023.8.0
docutils==0.20.1
exceptiongroup==1.1.2
executing==1.2.0
filelock==3.12.2
fonttools==4.42.0
fsspec==2023.6.0
furl==2.1.3
gitdb==4.0.10
GitPython==3.1.31
google-auth==2.22.0
google-auth-oauthlib==1.0.0
grpcio==1.56.2
gspread==5.10.0
h5py==3.8.0
holoviews==1.16.0
httplib2==0.22.0
idna==3.4
imagesize==1.4.1
importlib-metadata==6.8.0
iniconfig==2.0.0
ipython==8.14.0
jedi==0.19.0
Jinja2==3.1.2
jmespath==1.0.1
joblib==1.2.0
jsonschema==4.19.0
jsonschema-specifications==2023.7.1
kiwisolver==1.4.4
lightgbm==3.3.5
llvmlite==0.40.1
locket==1.0.0
Markdown==3.4.4
MarkupSafe==2.1.3
mat73==0.60
matplotlib==3.7.1
matplotlib-inline==0.1.6
mpmath==1.3.0
msgpack==1.0.5
networkx==3.1
numba==0.57.1
numpy==1.23.5
oauth2client==4.1.3
oauthlib==3.2.2
orderedmultidict==1.0.1
packaging==23.1
pandas==1.5.3
panel==0.14.4
param==1.13.0
parso==0.8.3
partd==1.4.0
pathlib2==2.3.7.post1
patsy==0.5.3
pexpect==4.8.0
pickleshare==0.7.5
Pillow==10.0.0
pluggy==1.2.0
prompt-toolkit==3.0.39
protobuf==4.24.0
psutil==5.9.5
ptyprocess==0.7.0
pure-eval==0.2.2
pyasn1==0.5.0
pyasn1-modules==0.3.0
pyct==0.5.0
Pygments==2.16.1
PyJWT==2.4.0
pyparsing==3.1.1
pytest==7.3.1
python-dateutil==2.8.2
pytz==2023.3
pyviz-comms==2.3.2
PyYAML==6.0.1
referencing==0.30.2
requests==2.31.0
requests-oauthlib==1.3.1
rpds-py==0.9.2
rsa==4.9
s3transfer==0.6.1
scikit-base==0.5.0
scikit-learn==1.3.0
scipy==1.10.1
shapely==2.0.1
six==1.16.0
sktime==0.21.0
smmap==5.0.0
snowballstemmer==2.2.0
sortedcontainers==2.4.0
Sphinx==7.1.2
sphinxcontrib-applehelp==1.0.5
sphinxcontrib-devhelp==1.0.3
sphinxcontrib-htmlhelp==2.0.2
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.4
sphinxcontrib-serializinghtml==1.1.6
stack-data==0.6.2
statsmodels==0.13.5
stumpy==1.11.1
sympy==1.12
tblib==2.0.0
tensorboard==2.13.0
tensorboard-data-server==0.7.1
threadpoolctl==3.2.0
tomli==2.0.1
toolz==0.12.0
torch==2.0.1 # 

torchdata==0.6.1
tornado==6.3.2
tqdm==4.65.0
traitlets==5.9.0
tsfel==0.1.5
tsfresh==0.20.0
typing_extensions==4.7.1
urllib3==1.26.16
wcwidth==0.2.6
webencodings==0.5.1
Werkzeug==2.3.6
wrapt==1.15.0
zict==3.0.0
zipp==3.16.2

as you can see I'm missing some packages (i noticed pyarrow and fastparquet) and pandas version is not as specified (1.5.3 instead of 2.0.1 and above).
what might be the cause of it? I tried updating the req file in my main branch and it didn't helped.

  
  
Posted 8 months ago
Votes Newest

Answers 16


@<1576381444509405184:profile|ManiacalLizard2> I am not sure that's accurate - the thread here shows every requirement on the task was installed, the issue might be with the requirements file, which in this case was not installed since the requirements were specified on the task itself

  
  
Posted 8 months ago

This will automatically happen if the Installed Packages section is empty. Alternatively, you can use agent.package_manager.force_repo_requirements_txt: true in the agent configuration

  
  
Posted 8 months ago

so how can i make clearml to install my req file? i thought that it is automatically detects the req file in the cloned repository..

  
  
Posted 8 months ago

And before you enqueue the task, what shows up in the Tasks "Installed Packages" section in the UI?

  
  
Posted 8 months ago

my repo structure is:
|----my_package
|----scripts
|----some_folder
|----some_folder
|----task_script.py
|----requirements.txt

  
  
Posted 8 months ago

from the consule logs i can see that:
entry_point = preprocessing.py
working_dir = Scripts/Volvo/volvo_data_alignment

  
  
Posted 8 months ago

since it is specifying the pandas==1.5.3 i thought it might be pulling the req file from my main branch so i updated the req file there as well (since i am working on a different branch) and it didn't changedanything (as you might expect)

  
  
Posted 8 months ago

following this thread as it happen every now and then that clearml miss some package for some reason ...

  
  
Posted 8 months ago

These will include all dependency packages installed by the main packages

  
  
Posted 8 months ago

Note in the log Collecting pandas==1.5.3 which means the requirements as pulled by the agent specify this pandas version.

  
  
Posted 8 months ago

The complete list following install is the result of pip freeze after the agent has used the requirements embedded on the task (the ones that were deduced from the local run)

  
  
Posted 8 months ago

installed packages before the task is running:

# Python 3.10.10 | packaged by Anaconda, Inc. | (main, Mar 21 2023, 18:39:17) [MSC v.1916 64 bit (AMD64)]

GitPython == 3.1.31
bokeh == 2.4.3
boto3 == 1.26.158
botocore == 1.29.158
clearml == 1.11.1
h5py == 3.8.0
holoviews == 1.16.0
joblib == 1.2.0
lightgbm == 3.3.5
mat73 == 0.60
matplotlib == 3.7.1
numpy == 1.23.5
pandas == 1.5.3
pytest == 7.3.1
scikit_learn == 1.3.0
scipy == 1.10.1
shapely == 2.0.1
sktime == 0.21.0
statsmodels == 0.13.5
torch == 2.0.1+cu117
torchdata == 0.6.1
tqdm == 4.65.0
tsfel == 0.1.5
tsfresh == 0.20.0
  
  
Posted 8 months ago

Also what is the structure of your git repository, what is the path to the task's entry point script in the repository and where in the repository is your requirements.txt located?

  
  
Posted 8 months ago

What you see is not pulled from your requirements file, but is deduced by ClearML when you're running the code locally (before enqueueing it to run remotely).

  
  
Posted 8 months ago

Hi @<1594863230964994048:profile|DangerousBee35> , can you attach the full remote execution log for the task, as well as the original requirements?

  
  
Posted 8 months ago

it seems like it does finds the requirments file otherwise why would it install tsfel tsfresh and some other "non-basic" packages?

  
  
Posted 8 months ago
583 Views
16 Answers
8 months ago
8 months ago
Tags