it seems that only the packages that are on the script are getting installed
I'm really confused, I'm not sure what is wrong and what is the relationship between the templates the agent and all of those thing
For the meantime, I'm giving up on the pipeline thing and I'll write a bash script to orchestrate the execution, because I need to deliver and I'm not feeling this is going anywhere
On an end note I'd love for this to work as expected, I'm not sure what you need from me. A fully reproducible example will be hard because obviously this is proprietary code. What I can tell you about my environment is that I manage it with Poetry, and works well under all other cases. If you'll give me a list of the things you need to figure this out I'll try my best to help so let's have all those combined in one message and see how we can continue from there
For future reference, I'll tell the story from my point of view:
I have written two tasks, and successfully ran them both on my local machine. Looking at those tasks' requiremetns section after they are completed, in teh installed packages part there seem to be packages missing, but it works so I don't really care.
Then, I've written a pipeline script using pipeline controller, to execute those tasks one after the other. It just inserted them into a queue and they never ran. From this slack channel I learned that in order to use pipelines I must have a clearml-agent on my machine which I msut say is a necessity which should not be there for the case of local execution. After I had an agent running, it started consuming the tasks but they all fail because he can't find a specific package.
` # Python 3.8.10 (default, Jun 2 2021, 10:49:15) [GCC 9.4.0]
clearml == 1.0.5
hyperopt == 0.2.5
matplotlib == 3.4.3
numpy == 1.21.2
pandas == 1.3.2
plotly == 5.3.0
python_dateutil == 2.8.2
scikit_learn == 0.24.2
statsmodels == 0.12.2
tqdm == 4.62.2
Detailed import analysis
**************************
IMPORT PACKAGE clearml
tasks/data_projection.py: 9
tasks/hp_optimization.py: 6
tasks/hpo_n_best_evaluation.py: 6
tasks/pipelines/monthly_predictions.py: 4
IMPORT PACKAGE hyperopt
src/models/hpo/init.py: 5
src/models/hpo/arima_hpo.py: 6,7
src/models/hpo/dummy.py: 6,7
tasks/hp_optimization.py: 10
IMPORT PACKAGE matplotlib
tasks/hpo_n_best_evaluation.py: 7
IMPORT PACKAGE numpy
src/metrics.py: 3
src/models/arima/arima_model.py: 3
src/models/feature_engineering.py: 4
src/models/hpo/dummy.py: 5
tasks/data_projection.py: 10
tasks/hp_optimization.py: 7
tasks/hpo_n_best_evaluation.py: 8
IMPORT PACKAGE pandas
src/data/data_io.py: 7
src/data/data_processing.py: 3
src/models/arima/arima_model.py: 4
src/models/feature_engineering.py: 5
tasks/data_projection.py: 3
tasks/hp_optimization.py: 8
tasks/hpo_n_best_evaluation.py: 9
IMPORT PACKAGE plotly
tasks/hp_optimization.py: 9
tasks/hpo_n_best_evaluation.py: 10
IMPORT PACKAGE python_dateutil
src/models/arima/arima_model.py: 5
src/models/hpo/arima_hpo.py: 5
tasks/data_projection.py: 12
tasks/hpo_n_best_evaluation.py: 11
tasks/pipelines/monthly_predictions.py: 5
IMPORT PACKAGE scikit_learn
src/debugging.py: [3]
src/models/arima/arima_model.py: [6]
src/models/feature_engineering.py: [6]
src/models/hpo/arima_hpo.py: [8]
src/models/hpo/dummy.py: [8]
tasks/data_projection.py: [11]
IMPORT PACKAGE statsmodels
src/models/arima/arima_model.py: 7
IMPORT PACKAGE tqdm
tasks/hpo_n_best_evaluation.py: 12
tasks/pipelines/monthly_predictions.py: 6 `
no this is from the task execution that failed
the output above is what the agent has as it seems... obviously on my machine I have it installed
what do you have under installed packages for this task?
In the installed package you have trains==0.16.4 , do you import it somewhere in your script?
back to the main subject, can you try adding it and re run it?
I never installed trains on this environment
pgrep -af trains shows that there is nothing running with that name
There are many ohter packages in my environment which are not listed
this is the one from the original (template) task? I can’t see the package that raise the error, can you try adding it and re-run? do you have the imports analysis?
BTW you have both trains and clearml , can you try with clearml only? it should support all the trains imports
👍
I can not see google package, can you try clone and add it manually? You can always add any package you like to any task with Task.add_requirements('package name', 'package version') ,
Do you have a toy example so I can reproduce it my side (using google.cloud but package is not listed in task)?
btw my site packages is false - should it be true? You pasted that but I’m not sure what it should be, in the paste is false but you are asking about true
false by default, when you change it to true it should use the system packages, do you have this package install in the system? what do you have under installed packages for this task?
alabaster==0.7.12 appdirs==1.4.4 apturl==0.5.2 attrs==21.2.0 Babel==2.9.1 bcrypt==3.1.7 blinker==1.4 Brlapi==0.7.0 cachetools==4.0.0 certifi==2019.11.28 chardet==3.0.4 chrome-gnome-shell==0.0.0 clearml==1.0.5 click==8.0.1 cloud-sptheme==1.10.1.post20200504175005 cloudpickle==1.6.0 colorama==0.4.3 command-not-found==0.3
I'm quite confused... The package is not missing, it is in my environment and executing tasks normally ( python my_script.py.... ) works
This is the pip freeze of the environment I don't know why it differs from what the agent has... the agent only has a subset of these google libs
I set it to true, I have more packages installed now, but it still fails.. here is the log TimelyPenguin76
` Successfully installed clearml-1.0.5 cloudpickle-1.6.0 cycler-0.10.0 hyperopt-0.2.5 kiwisolver-1.3.2 matplotlib-3.4.3 networkx-2.6.2 pandas-1.3.2 patsy-0.5.1 plotly-5.3.0 python-dateutil-2.8.2 statsmodels-0.12.2 tenacity-8.0.1 tqdm-4.62.2
Adding venv into cache: /home/elior/.clearml/venvs-builds/3.8
Running task id [24a54a473c234b00a126ec805d74046f]:
[.]$ /home/elior/.clearml/venvs-builds/3.8/bin/python -u tasks/hp_optimization.py
Summary - installed python packages:
pip:
- alabaster==0.7.12
- appdirs==1.4.4
- apturl==0.5.2
- attrs==21.2.0
- Babel==2.9.1
- bcrypt==3.1.7
- blinker==1.4
- Brlapi==0.7.0
- cachetools==4.0.0
- certifi==2019.11.28
- chardet==3.0.4
- chrome-gnome-shell==0.0.0
- clearml==1.0.5
- click==8.0.1
- cloud-sptheme==1.10.1.post20200504175005
- cloudpickle==1.6.0
- colorama==0.4.3
- command-not-found==0.3
- cryptography==2.8
- cupshelpers==1.0
- cycler==0.10.0
- Cython==0.29.24
- dbus-python==1.2.16
- decorator==4.4.2
- defer==1.0.6
- distlib==0.3.1
- distro==1.4.0
- distro-info===0.23ubuntu1
- docutils==0.17.1
- duplicity==0.8.12.0
- entrypoints==0.3
- fasteners==0.14.1
- filelock==3.0.12
- Flask==2.0.1
- funcsigs==1.0.2
- furl==2.1.2
- future==0.18.2
- google-api-python-client==1.7.11
- google-auth==1.5.1
- google-auth-httplib2==0.0.3
- httplib2==0.14.0
- humanfriendly==9.1
- hyperopt==0.2.5
- idna==2.8
- imagesize==1.2.0
- itsdangerous==2.0.1
- Jinja2==3.0.1
- joblib==1.0.1
- jsonschema==3.2.0
- kazam==1.4.5
- keyring==18.0.1
- kiwisolver==1.3.2
- language-selector==0.1
- launchpadlib==1.10.13
- lazr.restfulclient==0.14.2
- lazr.uri==1.0.3
- ldap3==2.4.1
- lockfile==0.12.2
- louis==3.12.0
- macaroonbakery==1.3.1
- Mako==1.1.0
- MarkupSafe==2.0.1
- matplotlib==3.4.3
- monotonic==1.5
- mysqlclient==1.4.4
- netifaces==0.10.4
- networkx==2.6.2
- numpy==1.21.2
- oauth2client==4.1.2
- oauthlib==3.1.0
- olefile==0.46
- orderedmultidict==1.0.1
- packaging==20.9
- pandas==1.3.2
- paramiko==2.6.0
- pathlib2==2.3.5
- patsy==0.5.1
- pbr==5.5.1
- pexpect==4.6.0
- Pillow==7.0.0
- plotly==5.3.0
- protobuf==3.6.1
- psutil==5.8.0
- pyasn1==0.4.2
- pyasn1-modules==0.2.1
- pycairo==1.16.2
- pycups==1.9.73
- PyDrive==1.3.1
- Pygments==2.9.0
- pygments-style-github==0.4
- PyGObject==3.36.0
- PyJWT==1.7.1
- pymacaroons==0.13.0
- PyNaCl==1.3.0
- pyOpenSSL==19.0.0
- pyparsing==2.4.7
- pyRFC3339==1.1
- pyrsistent==0.17.3
- python-apt==2.0.0+ubuntu0.20.4.5
- python-dateutil==2.8.2
- python-debian===0.1.36ubuntu1
- pytz==2019.3
- pyxdg==0.26
- PyYAML==5.3.1
- reportlab==3.5.34
- requests==2.22.0
- requests-file==1.5.1
- requests-unixsocket==0.2.0
- rsa==4.0
- scikit-learn==0.24.2
- scipy==1.6.3
- screen-resolution-extra==0.0.0
- SecretStorage==2.3.1
- simplejson==3.16.0
- six==1.14.0
- snowballstemmer==2.1.0
- Sphinx==4.0.2
- sphinx-rtd-theme==0.5.2
- sphinx-tabs==3.0.0
- sphinxcontrib-applehelp==1.0.2
- sphinxcontrib-devhelp==1.0.2
- sphinxcontrib-htmlhelp==2.0.0
- sphinxcontrib-httpdomain==1.7.0
- sphinxcontrib-jsmath==1.0.1
- sphinxcontrib-qthelp==1.0.3
- sphinxcontrib-serializinghtml==1.1.5
- SQLAlchemy==1.3.12
- sqlalchemy-migrate==0.13.0
- sqlparse==0.2.4
- statsmodels==0.12.2
- stevedore==3.3.0
- systemd-python==234
- Tempita==0.5.2
- tenacity==8.0.1
- threadpoolctl==2.1.0
- tqdm==4.62.2
- trains==0.16.4
- ubuntu-advantage-tools==27.2
- ubuntu-drivers-common==0.0.0
- ufw==0.36
- unattended-upgrades==0.1
- uritemplate==0.6
- urllib3==1.25.8
- usb-creator==0.3.7
- virtualenv==20.2.2
- virtualenv-clone==0.5.4
- virtualenvwrapper==4.8.4
- wadllib==1.3.3
- Werkzeug==2.0.1
- xkit==0.0.0
Environment setup completed successfully
Starting Task Execution:
Traceback (most recent call last):
File "tasks/hp_optimization.py", line 14, in <module>
from src.data.data_io import DataIO
File "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX", line 8, in <module>
from google.cloud import bigquery
ModuleNotFoundError: No module named 'google.cloud' `
TimelyPenguin76 I think our problem is that the agent is not using this environment, I'm not sure which one he does... Is there a way to hard-code the agent environment?
those are all? you can copy all the section from the UI, and hide the internal details
btw my site packages is false - should it be true? You pasted that but I'm not sure what it should be, in the paste is false but you are asking about true
Not in your current environment, in the one the clearml-agent creates. In the installed packages you have trains==0.16.4 , so I guess you are using it in your code (if not, you can check in your base task, under installed package, the imports analysis and get the information about where this import is coming from)
About the import issue, did you try adding the missing package and re run the task?
alabaster==0.7.12 appdirs==1.4.4 apturl==0.5.2 attrs==21.2.0 Babel==2.9.1 bcrypt==3.1.7 blinker==1.4 Brlapi==0.7.0 cachetools==4.0.0 certifi==2019.11.28 chardet==3.0.4 chrome-gnome-shell==0.0.0 clearml==1.0.5 click==8.0.1 cloud-sptheme==1.10.1.post20200504175005 cloudpickle==1.6.0 colorama==0.4.3 command-not-found==0.3 cryptography==2.8 cupshelpers==1.0 cycler==0.10.0 Cython==0.29.24 dbus-python==1.2.16 decorator==4.4.2 defer==1.0.6 distlib==0.3.1 distro==1.4.0 distro-info===0.23ubuntu1 docutils==0.17.1 duplicity==0.8.12.0 entrypoints==0.3 fasteners==0.14.1 filelock==3.0.12 Flask==2.0.1 funcsigs==1.0.2 furl==2.1.2 future==0.18.2 google-api-python-client==1.7.11 google-auth==1.5.1 google-auth-httplib2==0.0.3 httplib2==0.14.0 humanfriendly==9.1 hyperopt==0.2.5 idna==2.8 imagesize==1.2.0 itsdangerous==2.0.1 Jinja2==3.0.1 joblib==1.0.1 jsonschema==3.2.0 kazam==1.4.5 keyring==18.0.1 kiwisolver==1.3.2 language-selector==0.1 launchpadlib==1.10.13 lazr.restfulclient==0.14.2 lazr.uri==1.0.3 ldap3==2.4.1 lockfile==0.12.2 louis==3.12.0 macaroonbakery==1.3.1 Mako==1.1.0 MarkupSafe==2.0.1 matplotlib==3.4.3 monotonic==1.5 mysqlclient==1.4.4 netifaces==0.10.4 networkx==2.6.2 numpy==1.21.2 oauth2client==4.1.2 oauthlib==3.1.0 olefile==0.46 orderedmultidict==1.0.1 packaging==20.9 pandas==1.3.2 paramiko==2.6.0 pathlib2==2.3.5 patsy==0.5.1 pbr==5.5.1 pexpect==4.6.0 Pillow==7.0.0 plotly==5.3.0 protobuf==3.6.1 psutil==5.8.0 pyasn1==0.4.2 pyasn1-modules==0.2.1 pycairo==1.16.2 pycups==1.9.73 PyDrive==1.3.1 Pygments==2.9.0 pygments-style-github==0.4 PyGObject==3.36.0 PyJWT==1.7.1 pymacaroons==0.13.0 PyNaCl==1.3.0 pyOpenSSL==19.0.0 pyparsing==2.4.7 pyRFC3339==1.1 pyrsistent==0.17.3 python-apt==2.0.0+ubuntu0.20.4.5 python-dateutil==2.8.2 python-debian===0.1.36ubuntu1 pytz==2019.3 pyxdg==0.26 PyYAML==5.3.1 reportlab==3.5.34 requests==2.22.0 requests-file==1.5.1 requests-unixsocket==0.2.0 rsa==4.0 scikit-learn==0.24.2 scipy==1.6.3 screen-resolution-extra==0.0.0 SecretStorage==2.3.1 simplejson==3.16.0 six==1.14.0 snowballstemmer==2.1.0 Sphinx==4.0.2 sphinx-rtd-theme==0.5.2 sphinx-tabs==3.0.0 sphinxcontrib-applehelp==1.0.2 sphinxcontrib-devhelp==1.0.2 sphinxcontrib-htmlhelp==2.0.0 sphinxcontrib-httpdomain==1.7.0 sphinxcontrib-jsmath==1.0.1 sphinxcontrib-qthelp==1.0.3 sphinxcontrib-serializinghtml==1.1.5 SQLAlchemy==1.3.12 sqlalchemy-migrate==0.13.0 sqlparse==0.2.4 statsmodels==0.12.2 stevedore==3.3.0 systemd-python==234 Tempita==0.5.2 tenacity==8.0.1 threadpoolctl==2.1.0 tqdm==4.62.2 trains==0.16.4 ubuntu-advantage-tools==27.2 ubuntu-drivers-common==0.0.0 ufw==0.36 unattended-upgrades==0.1 uritemplate==0.6 urllib3==1.25.8 usb-creator==0.3.7 virtualenv==20.2.2 virtualenv-clone==0.5.4 virtualenvwrapper==4.8.4 wadllib==1.3.3 Werkzeug==2.0.1 xkit==0.0.0
I don't even know where trains is coming from... While using the same environment I can't even import trains, see
Anyway I checked the base task, and this is what it has in installed packages (seems like it doesn't list all the real packages in the environment)