@<1576381444509405184:profile|ManiacalLizard2> I am not sure that's accurate - the thread here shows every requirement on the task was installed, the issue might be with the requirements file, which in this case was not installed since the requirements were specified on the task itself
Also what is the structure of your git repository, what is the path to the task's entry point script in the repository and where in the repository is your requirements.txt located?
The complete list following install is the result of pip freeze after the agent has used the requirements embedded on the task (the ones that were deduced from the local run)
my repo structure is:
|----my_package
|----scripts
|----some_folder
|----some_folder
|----task_script.py
|----requirements.txt
since it is specifying the pandas==1.5.3
i thought it might be pulling the req file from my main branch so i updated the req file there as well (since i am working on a different branch) and it didn't changedanything (as you might expect)
it seems like it does finds the requirments file otherwise why would it install tsfel
tsfresh
and some other "non-basic" packages?
What you see is not pulled from your requirements file, but is deduced by ClearML when you're running the code locally (before enqueueing it to run remotely).
from the consule logs i can see that:entry_point = preprocessing.py
working_dir = Scripts/Volvo/volvo_data_alignment
following this thread as it happen every now and then that clearml miss some package for some reason ...
installed packages before the task is running:
# Python 3.10.10 | packaged by Anaconda, Inc. | (main, Mar 21 2023, 18:39:17) [MSC v.1916 64 bit (AMD64)]
GitPython == 3.1.31
bokeh == 2.4.3
boto3 == 1.26.158
botocore == 1.29.158
clearml == 1.11.1
h5py == 3.8.0
holoviews == 1.16.0
joblib == 1.2.0
lightgbm == 3.3.5
mat73 == 0.60
matplotlib == 3.7.1
numpy == 1.23.5
pandas == 1.5.3
pytest == 7.3.1
scikit_learn == 1.3.0
scipy == 1.10.1
shapely == 2.0.1
sktime == 0.21.0
statsmodels == 0.13.5
torch == 2.0.1+cu117
torchdata == 0.6.1
tqdm == 4.65.0
tsfel == 0.1.5
tsfresh == 0.20.0
This will automatically happen if the Installed Packages section is empty. Alternatively, you can use agent.package_manager.force_repo_requirements_txt: true
in the agent configuration
And before you enqueue the task, what shows up in the Tasks "Installed Packages" section in the UI?
so how can i make clearml to install my req file? i thought that it is automatically detects the req file in the cloned repository..
Hi @<1594863230964994048:profile|DangerousBee35> , can you attach the full remote execution log for the task, as well as the original requirements?
These will include all dependency packages installed by the main packages
Note in the log Collecting pandas==1.5.3
which means the requirements as pulled by the agent specify this pandas version.