Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
I Have To Say I'M Totally Confused By The Pipeline I Want To Execute The Pipeline On My Local Computer. I Followed

I have to say I'm totally confused by the pipeline

I want to execute the pipeline on my local computer. I followed https://github.com/allegroai/clearml/blob/master/examples/pipeline/pipeline_controller.py and I used the pipe = clearml.PipelineController(default_execution_queue='default', add_pipeline_tags=False) to create my pipe. When I execute it, all the jobs are pending and none is running...

  
  
Posted 2 years ago
Votes Newest

Answers 30


it seems that only the packages that are on the script are getting installed

  
  
Posted 2 years ago

I never installed trains on this environment

  
  
Posted 2 years ago

In the installed package you have trains==0.16.4 , do you import it somewhere in your script?

back to the main subject, can you try adding it and re run it?

  
  
Posted 2 years ago

pgrep -af trains shows that there is nothing running with that name

  
  
Posted 2 years ago

I'm quite confused... The package is not missing, it is in my environment and executing tasks normally ( python my_script.py.... ) works

  
  
Posted 2 years ago

those are all? you can copy all the section from the UI, and hide the internal details

  
  
Posted 2 years ago

sorry I think it trimmed it

  
  
Posted 2 years ago

` # Python 3.8.10 (default, Jun 2 2021, 10:49:15) [GCC 9.4.0]
clearml == 1.0.5
hyperopt == 0.2.5
matplotlib == 3.4.3
numpy == 1.21.2
pandas == 1.3.2
plotly == 5.3.0
python_dateutil == 2.8.2
scikit_learn == 0.24.2
statsmodels == 0.12.2
tqdm == 4.62.2

Detailed import analysis

**************************

IMPORT PACKAGE clearml

tasks/data_projection.py: 9

tasks/hp_optimization.py: 6

tasks/hpo_n_best_evaluation.py: 6

tasks/pipelines/monthly_predictions.py: 4

IMPORT PACKAGE hyperopt

src/models/hpo/init.py: 5

src/models/hpo/arima_hpo.py: 6,7

src/models/hpo/dummy.py: 6,7

tasks/hp_optimization.py: 10

IMPORT PACKAGE matplotlib

tasks/hpo_n_best_evaluation.py: 7

IMPORT PACKAGE numpy

src/metrics.py: 3

src/models/arima/arima_model.py: 3

src/models/feature_engineering.py: 4

src/models/hpo/dummy.py: 5

tasks/data_projection.py: 10

tasks/hp_optimization.py: 7

tasks/hpo_n_best_evaluation.py: 8

IMPORT PACKAGE pandas

src/data/data_io.py: 7

src/data/data_processing.py: 3

src/models/arima/arima_model.py: 4

src/models/feature_engineering.py: 5

tasks/data_projection.py: 3

tasks/hp_optimization.py: 8

tasks/hpo_n_best_evaluation.py: 9

IMPORT PACKAGE plotly

tasks/hp_optimization.py: 9

tasks/hpo_n_best_evaluation.py: 10

IMPORT PACKAGE python_dateutil

src/models/arima/arima_model.py: 5

src/models/hpo/arima_hpo.py: 5

tasks/data_projection.py: 12

tasks/hpo_n_best_evaluation.py: 11

tasks/pipelines/monthly_predictions.py: 5

IMPORT PACKAGE scikit_learn

src/debugging.py: [3]

src/models/arima/arima_model.py: [6]

src/models/feature_engineering.py: [6]

src/models/hpo/arima_hpo.py: [8]

src/models/hpo/dummy.py: [8]

tasks/data_projection.py: [11]

IMPORT PACKAGE statsmodels

src/models/arima/arima_model.py: 7

IMPORT PACKAGE tqdm

tasks/hpo_n_best_evaluation.py: 12

tasks/pipelines/monthly_predictions.py: 6 `

  
  
Posted 2 years ago

no this is from the task execution that failed

  
  
Posted 2 years ago

👍

I can not see google package, can you try clone and add it manually? You can always add any package you like to any task with Task.add_requirements('package name', 'package version') ,
Do you have a toy example so I can reproduce it my side (using google.cloud but package is not listed in task)?

  
  
Posted 2 years ago

I set it to true and restarted by agent

  
  
Posted 2 years ago

this

  
  
Posted 2 years ago

this is the one from the original (template) task? I can’t see the package that raise the error, can you try adding it and re-run? do you have the imports analysis?

BTW you have both trains and clearml , can you try with clearml only? it should support all the trains imports

  
  
Posted 2 years ago

what do you have under installed packages for this task?

  
  
Posted 2 years ago

alabaster==0.7.12 appdirs==1.4.4 apturl==0.5.2 attrs==21.2.0 Babel==2.9.1 bcrypt==3.1.7 blinker==1.4 Brlapi==0.7.0 cachetools==4.0.0 certifi==2019.11.28 chardet==3.0.4 chrome-gnome-shell==0.0.0 clearml==1.0.5 click==8.0.1 cloud-sptheme==1.10.1.post20200504175005 cloudpickle==1.6.0 colorama==0.4.3 command-not-found==0.3 cryptography==2.8 cupshelpers==1.0 cycler==0.10.0 Cython==0.29.24 dbus-python==1.2.16 decorator==4.4.2 defer==1.0.6 distlib==0.3.1 distro==1.4.0 distro-info===0.23ubuntu1 docutils==0.17.1 duplicity==0.8.12.0 entrypoints==0.3 fasteners==0.14.1 filelock==3.0.12 Flask==2.0.1 funcsigs==1.0.2 furl==2.1.2 future==0.18.2 google-api-python-client==1.7.11 google-auth==1.5.1 google-auth-httplib2==0.0.3 httplib2==0.14.0 humanfriendly==9.1 hyperopt==0.2.5 idna==2.8 imagesize==1.2.0 itsdangerous==2.0.1 Jinja2==3.0.1 joblib==1.0.1 jsonschema==3.2.0 kazam==1.4.5 keyring==18.0.1 kiwisolver==1.3.2 language-selector==0.1 launchpadlib==1.10.13 lazr.restfulclient==0.14.2 lazr.uri==1.0.3 ldap3==2.4.1 lockfile==0.12.2 louis==3.12.0 macaroonbakery==1.3.1 Mako==1.1.0 MarkupSafe==2.0.1 matplotlib==3.4.3 monotonic==1.5 mysqlclient==1.4.4 netifaces==0.10.4 networkx==2.6.2 numpy==1.21.2 oauth2client==4.1.2 oauthlib==3.1.0 olefile==0.46 orderedmultidict==1.0.1 packaging==20.9 pandas==1.3.2 paramiko==2.6.0 pathlib2==2.3.5 patsy==0.5.1 pbr==5.5.1 pexpect==4.6.0 Pillow==7.0.0 plotly==5.3.0 protobuf==3.6.1 psutil==5.8.0 pyasn1==0.4.2 pyasn1-modules==0.2.1 pycairo==1.16.2 pycups==1.9.73 PyDrive==1.3.1 Pygments==2.9.0 pygments-style-github==0.4 PyGObject==3.36.0 PyJWT==1.7.1 pymacaroons==0.13.0 PyNaCl==1.3.0 pyOpenSSL==19.0.0 pyparsing==2.4.7 pyRFC3339==1.1 pyrsistent==0.17.3 python-apt==2.0.0+ubuntu0.20.4.5 python-dateutil==2.8.2 python-debian===0.1.36ubuntu1 pytz==2019.3 pyxdg==0.26 PyYAML==5.3.1 reportlab==3.5.34 requests==2.22.0 requests-file==1.5.1 requests-unixsocket==0.2.0 rsa==4.0 scikit-learn==0.24.2 scipy==1.6.3 screen-resolution-extra==0.0.0 SecretStorage==2.3.1 simplejson==3.16.0 six==1.14.0 snowballstemmer==2.1.0 Sphinx==4.0.2 sphinx-rtd-theme==0.5.2 sphinx-tabs==3.0.0 sphinxcontrib-applehelp==1.0.2 sphinxcontrib-devhelp==1.0.2 sphinxcontrib-htmlhelp==2.0.0 sphinxcontrib-httpdomain==1.7.0 sphinxcontrib-jsmath==1.0.1 sphinxcontrib-qthelp==1.0.3 sphinxcontrib-serializinghtml==1.1.5 SQLAlchemy==1.3.12 sqlalchemy-migrate==0.13.0 sqlparse==0.2.4 statsmodels==0.12.2 stevedore==3.3.0 systemd-python==234 Tempita==0.5.2 tenacity==8.0.1 threadpoolctl==2.1.0 tqdm==4.62.2 trains==0.16.4 ubuntu-advantage-tools==27.2 ubuntu-drivers-common==0.0.0 ufw==0.36 unattended-upgrades==0.1 uritemplate==0.6 urllib3==1.25.8 usb-creator==0.3.7 virtualenv==20.2.2 virtualenv-clone==0.5.4 virtualenvwrapper==4.8.4 wadllib==1.3.3 Werkzeug==2.0.1 xkit==0.0.0

  
  
Posted 2 years ago

Not in your current environment, in the one the clearml-agent creates. In the installed packages you have trains==0.16.4 , so I guess you are using it in your code (if not, you can check in your base task, under installed package, the imports analysis and get the information about where this import is coming from)

About the import issue, did you try adding the missing package and re run the task?

  
  
Posted 2 years ago

This is the pip freeze of the environment I don't know why it differs from what the agent has... the agent only has a subset of these google libs

  
  
Posted 2 years ago

btw my site packages is false - should it be true? You pasted that but I'm not sure what it should be, in the paste is false but you are asking about true

  
  
Posted 2 years ago

I set it to true, I have more packages installed now, but it still fails.. here is the log TimelyPenguin76

` Successfully installed clearml-1.0.5 cloudpickle-1.6.0 cycler-0.10.0 hyperopt-0.2.5 kiwisolver-1.3.2 matplotlib-3.4.3 networkx-2.6.2 pandas-1.3.2 patsy-0.5.1 plotly-5.3.0 python-dateutil-2.8.2 statsmodels-0.12.2 tenacity-8.0.1 tqdm-4.62.2
Adding venv into cache: /home/elior/.clearml/venvs-builds/3.8
Running task id [24a54a473c234b00a126ec805d74046f]:
[.]$ /home/elior/.clearml/venvs-builds/3.8/bin/python -u tasks/hp_optimization.py
Summary - installed python packages:
pip:

  • alabaster==0.7.12
  • appdirs==1.4.4
  • apturl==0.5.2
  • attrs==21.2.0
  • Babel==2.9.1
  • bcrypt==3.1.7
  • blinker==1.4
  • Brlapi==0.7.0
  • cachetools==4.0.0
  • certifi==2019.11.28
  • chardet==3.0.4
  • chrome-gnome-shell==0.0.0
  • clearml==1.0.5
  • click==8.0.1
  • cloud-sptheme==1.10.1.post20200504175005
  • cloudpickle==1.6.0
  • colorama==0.4.3
  • command-not-found==0.3
  • cryptography==2.8
  • cupshelpers==1.0
  • cycler==0.10.0
  • Cython==0.29.24
  • dbus-python==1.2.16
  • decorator==4.4.2
  • defer==1.0.6
  • distlib==0.3.1
  • distro==1.4.0
  • distro-info===0.23ubuntu1
  • docutils==0.17.1
  • duplicity==0.8.12.0
  • entrypoints==0.3
  • fasteners==0.14.1
  • filelock==3.0.12
  • Flask==2.0.1
  • funcsigs==1.0.2
  • furl==2.1.2
  • future==0.18.2
  • google-api-python-client==1.7.11
  • google-auth==1.5.1
  • google-auth-httplib2==0.0.3
  • httplib2==0.14.0
  • humanfriendly==9.1
  • hyperopt==0.2.5
  • idna==2.8
  • imagesize==1.2.0
  • itsdangerous==2.0.1
  • Jinja2==3.0.1
  • joblib==1.0.1
  • jsonschema==3.2.0
  • kazam==1.4.5
  • keyring==18.0.1
  • kiwisolver==1.3.2
  • language-selector==0.1
  • launchpadlib==1.10.13
  • lazr.restfulclient==0.14.2
  • lazr.uri==1.0.3
  • ldap3==2.4.1
  • lockfile==0.12.2
  • louis==3.12.0
  • macaroonbakery==1.3.1
  • Mako==1.1.0
  • MarkupSafe==2.0.1
  • matplotlib==3.4.3
  • monotonic==1.5
  • mysqlclient==1.4.4
  • netifaces==0.10.4
  • networkx==2.6.2
  • numpy==1.21.2
  • oauth2client==4.1.2
  • oauthlib==3.1.0
  • olefile==0.46
  • orderedmultidict==1.0.1
  • packaging==20.9
  • pandas==1.3.2
  • paramiko==2.6.0
  • pathlib2==2.3.5
  • patsy==0.5.1
  • pbr==5.5.1
  • pexpect==4.6.0
  • Pillow==7.0.0
  • plotly==5.3.0
  • protobuf==3.6.1
  • psutil==5.8.0
  • pyasn1==0.4.2
  • pyasn1-modules==0.2.1
  • pycairo==1.16.2
  • pycups==1.9.73
  • PyDrive==1.3.1
  • Pygments==2.9.0
  • pygments-style-github==0.4
  • PyGObject==3.36.0
  • PyJWT==1.7.1
  • pymacaroons==0.13.0
  • PyNaCl==1.3.0
  • pyOpenSSL==19.0.0
  • pyparsing==2.4.7
  • pyRFC3339==1.1
  • pyrsistent==0.17.3
  • python-apt==2.0.0+ubuntu0.20.4.5
  • python-dateutil==2.8.2
  • python-debian===0.1.36ubuntu1
  • pytz==2019.3
  • pyxdg==0.26
  • PyYAML==5.3.1
  • reportlab==3.5.34
  • requests==2.22.0
  • requests-file==1.5.1
  • requests-unixsocket==0.2.0
  • rsa==4.0
  • scikit-learn==0.24.2
  • scipy==1.6.3
  • screen-resolution-extra==0.0.0
  • SecretStorage==2.3.1
  • simplejson==3.16.0
  • six==1.14.0
  • snowballstemmer==2.1.0
  • Sphinx==4.0.2
  • sphinx-rtd-theme==0.5.2
  • sphinx-tabs==3.0.0
  • sphinxcontrib-applehelp==1.0.2
  • sphinxcontrib-devhelp==1.0.2
  • sphinxcontrib-htmlhelp==2.0.0
  • sphinxcontrib-httpdomain==1.7.0
  • sphinxcontrib-jsmath==1.0.1
  • sphinxcontrib-qthelp==1.0.3
  • sphinxcontrib-serializinghtml==1.1.5
  • SQLAlchemy==1.3.12
  • sqlalchemy-migrate==0.13.0
  • sqlparse==0.2.4
  • statsmodels==0.12.2
  • stevedore==3.3.0
  • systemd-python==234
  • Tempita==0.5.2
  • tenacity==8.0.1
  • threadpoolctl==2.1.0
  • tqdm==4.62.2
  • trains==0.16.4
  • ubuntu-advantage-tools==27.2
  • ubuntu-drivers-common==0.0.0
  • ufw==0.36
  • unattended-upgrades==0.1
  • uritemplate==0.6
  • urllib3==1.25.8
  • usb-creator==0.3.7
  • virtualenv==20.2.2
  • virtualenv-clone==0.5.4
  • virtualenvwrapper==4.8.4
  • wadllib==1.3.3
  • Werkzeug==2.0.1
  • xkit==0.0.0
    Environment setup completed successfully
    Starting Task Execution:
    Traceback (most recent call last):
    File "tasks/hp_optimization.py", line 14, in <module>
    from src.data.data_io import DataIO
    File "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX", line 8, in <module>
    from google.cloud import bigquery
    ModuleNotFoundError: No module named 'google.cloud' `
  
  
Posted 2 years ago

alabaster==0.7.12 appdirs==1.4.4 apturl==0.5.2 attrs==21.2.0 Babel==2.9.1 bcrypt==3.1.7 blinker==1.4 Brlapi==0.7.0 cachetools==4.0.0 certifi==2019.11.28 chardet==3.0.4 chrome-gnome-shell==0.0.0 clearml==1.0.5 click==8.0.1 cloud-sptheme==1.10.1.post20200504175005 cloudpickle==1.6.0 colorama==0.4.3 command-not-found==0.3

  
  
Posted 2 years ago

No I don't have trains anywhere in my code

  
  
Posted 2 years ago

I'm really confused, I'm not sure what is wrong and what is the relationship between the templates the agent and all of those thing

For the meantime, I'm giving up on the pipeline thing and I'll write a bash script to orchestrate the execution, because I need to deliver and I'm not feeling this is going anywhere

On an end note I'd love for this to work as expected, I'm not sure what you need from me. A fully reproducible example will be hard because obviously this is proprietary code. What I can tell you about my environment is that I manage it with Poetry, and works well under all other cases. If you'll give me a list of the things you need to figure this out I'll try my best to help so let's have all those combined in one message and see how we can continue from there

For future reference, I'll tell the story from my point of view:
I have written two tasks, and successfully ran them both on my local machine. Looking at those tasks' requiremetns section after they are completed, in teh installed packages part there seem to be packages missing, but it works so I don't really care.
Then, I've written a pipeline script using pipeline controller, to execute those tasks one after the other. It just inserted them into a queue and they never ran. From this slack channel I learned that in order to use pipelines I must have a clearml-agent on my machine which I msut say is a necessity which should not be there for the case of local execution. After I had an agent running, it started consuming the tasks but they all fail because he can't find a specific package.

  
  
Posted 2 years ago

I don't even know where trains is coming from... While using the same environment I can't even import trains, see

  
  
Posted 2 years ago

Anyway I checked the base task, and this is what it has in installed packages (seems like it doesn't list all the real packages in the environment)

  
  
Posted 2 years ago

this is the full one TimelyPenguin76

  
  
Posted 2 years ago

TimelyPenguin76 I think our problem is that the agent is not using this environment, I'm not sure which one he does... Is there a way to hard-code the agent environment?

  
  
Posted 2 years ago

the output above is what the agent has as it seems... obviously on my machine I have it installed

  
  
Posted 2 years ago

btw my site packages is false - should it be true? You pasted that but I’m not sure what it should be, in the paste is false but you are asking about true

false by default, when you change it to true it should use the system packages, do you have this package install in the system? what do you have under installed packages for this task?

  
  
Posted 2 years ago

There are many ohter packages in my environment which are not listed

  
  
Posted 2 years ago

TimelyPenguin76

  
  
Posted 2 years ago