Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Unanswered
Good Morning, I'M Wondering If Someone Has Any Advice/Experience Configuring Clearml-Agent To Include Private Packages From Aws Codeartifact? So Far I Know I Have To Edit The


    1. I confirm, the venv is created with all packages, when I go to the cached location /home/ubuntu/.clearml/venvs-builds/3.8 and activate it, the packages are there. The full log:
      ` 1617282623837 ip-172-28-11-109:0 INFO task 4ce5aedc75404225b37eda2d9bd9ad8f pulled from 903741980edd416697ca8074f782bdb4 by worker ip-172-28-11-109:0

1617282629221 ip-172-28-11-109:0 DEBUG Current configuration (clearml_agent v0.17.2, location: /tmp/.clearml_agent.xo_7zbpi.cfg):

agent.worker_id = ip-172-28-11-109:0
agent.worker_name = ip-172-28-11-109
agent.force_git_ssh_protocol = true
agent.python_binary =
agent.package_manager.type = poetry
agent.package_manager.pip_version = <20.2
agent.package_manager.system_site_packages = false
agent.package_manager.force_upgrade = false
agent.package_manager.conda_channels.0 = defaults
agent.package_manager.conda_channels.1 = conda-forge
agent.package_manager.conda_channels.2 = pytorch
agent.package_manager.torch_nightly = false
agent.package_manager.extra_index_url.0 = https://***
agent.venvs_dir = /home/ubuntu/.clearml/venvs-builds
agent.venvs_cache.max_entries = 10
agent.venvs_cache.free_space_threshold_gb = 2.0
agent.vcs_cache.enabled = true
agent.vcs_cache.path = /home/ubuntu/.clearml/vcs-cache
agent.venv_update.enabled = false
agent.pip_download_cache.enabled = true
agent.pip_download_cache.path = /home/ubuntu/.clearml/pip-download-cache
agent.translate_ssh = true
agent.reload_config = false
agent.docker_pip_cache = /home/ubuntu/.clearml/pip-cache
agent.docker_apt_cache = /home/ubuntu/.clearml/apt-cache
agent.docker_force_pull = false
agent.default_docker.image = nvidia/cuda:10.1-runtime-ubuntu18.04
agent.enable_task_env = false
agent.default_python = 3.8
agent.cuda_version = 110
agent.cudnn_version = 0
sdk.storage.cache.default_base_dir = ~/.clearml/cache
sdk.storage.cache.size.min_free_bytes = 10GB
sdk.storage.direct_access.0.url = file://*
sdk.metrics.file_history_size = 100
sdk.metrics.matplotlib_untitled_history_size = 100
sdk.metrics.images.format = JPEG
sdk.metrics.images.quality = 87
sdk.metrics.images.subsampling = 0
sdk.metrics.tensorboard_single_series_per_graph = false
sdk.network.metrics.file_upload_threads = 4
sdk.network.metrics.file_upload_starvation_warning_sec = 120
sdk.network.iteration.max_retries_on_server_error = 5
sdk.network.iteration.retry_backoff_factor_sec = 10
sdk.aws.s3.key =
sdk.aws.s3.region =
sdk.aws.boto3.pool_connections = 512
sdk.aws.boto3.max_multipart_concurrency = 16
sdk.log.null_log_propagate = false
sdk.log.task_log_buffer_capacity = 66
sdk.log.disable_urllib3_info = true
sdk.development.task_reuse_time_window_in_hours = 72.0
sdk.development.vcs_repo_detect_async = true
sdk.development.store_uncommitted_code_diff = true
sdk.development.support_stopping = true
sdk.development.default_output_uri =
sdk.development.force_analyze_entire_repo = false
sdk.development.suppress_update_message = false
sdk.development.detect_with_pip_freeze = false
sdk.development.worker.report_period_sec = 2
sdk.development.worker.ping_period_sec = 30
sdk.development.worker.log_stdout = true
sdk.development.worker.report_global_mem_used = false
sdk.development.store_uncommitted_code_diff_on_train = true
api.version = 1.5
api.verify_certificate = true
api.default_version = 1.5
api.http.max_req_size = 15728640
api.http.retries.total = 240
api.http.retries.connect = 240
api.http.retries.read = 240
api.http.retries.redirect = 240
api.http.retries.status = 240
api.http.retries.backoff_factor = 1.0
api.http.retries.backoff_max = 120.0
api.http.wait_on_maintenance_forever = true
api.http.pool_maxsize = 512
api.http.pool_connections = 512
api.web_server = http://***
api.api_server = http://***
api.files_server = http://***
api.credentials.access_key = ***
api.host = ***

Executing task id [4ce5aedc75404225b37eda2d9bd9ad8f]:
repository =
branch = master
version_num = e267eca697341b8ccaf84be6f52480198b6e9682
tag =
docker_cmd = None
entry_point = create_dataset.py
working_dir = ***/dataset_creation

Using base prefix '/usr'
New python executable in /home/ubuntu/.clearml/venvs-builds/3.8/bin/python3.8
Also creating executable in /home/ubuntu/.clearml/venvs-builds/3.8/bin/python
Installing setuptools, pip, wheel...

1617282634455 ip-172-28-11-109:0 DEBUG done.

Using cached repository in "/home/ubuntu/.clearml/vcs-***"
Note: switching to 'e267eca697341b8ccaf84be6f52480198b6e9682'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

git switch -c <new-branch-name>

Or undo this operation with:

git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at e267eca Merge pull request #17 from ***
type: git
url:
branch: HEAD
commit: e267eca697341b8ccaf84be6f52480198b6e9682
root: /home/ubuntu/.clearml/venvs-builds/3.8/task_repository/***.git
Applying uncommitted changes

Poetry Enabled: Ignoring requested python packages, using repository poetry lock file!
Creating virtualenv *** in /home/ubuntu/.clearml/venvs-builds/3.8/task_repository/***.git/.venv
Installing dependencies from lock file

1617282639678 ip-172-28-11-109:0 DEBUG
Package operations: 125 installs, 0 updates, 0 removals

• Installing six (1.15.0)

1617282644955 ip-172-28-11-109:0 DEBUG • Installing certifi (2020.12.5)
• Installing chardet (4.0.0)
• Installing jmespath (0.10.0)
• Installing idna (2.10)
• Installing pyasn1 (0.4.8)
• Installing python-dateutil (2.8.1)
• Installing urllib3 (1.26.4)
• Installing botocore (1.20.36)
• Installing cachetools (4.2.1)
• Installing cycler (0.10.0)
• Installing kiwisolver (1.3.1)
• Installing numpy (1.19.5)
• Installing oauthlib (3.1.0)
• Installing pillow (6.2.2)
• Installing pyasn1-modules (0.2.8)
• Installing pytz (2021.1)
• Installing requests (2.25.1)
• Installing pyparsing (2.4.7)
• Installing rsa (4.7.2)
• Installing google-auth (1.28.0)
• Installing joblib (1.0.1)
• Installing matplotlib (3.3.4)
• Installing ply (3.11)
• Installing python-levenshtein (0.12.2)
• Installing requests-oauthlib (1.3.0)
• Installing scipy (1.6.1)
• Installing pandas (1.2.3)
• Installing requests-file (1.5.1)
• Installing soupsieve (2.2.1)
• Installing s3transfer (0.3.6)
• Installing texttable (1.6.3)

1617282650198 ip-172-28-11-109:0 DEBUG • Installing threadpoolctl (2.1.0)
• Installing tzlocal (2.1)
• Installing absl-py (0.12.0)
• Installing attrs (20.3.0)
• Installing beautifulsoup4 (4.9.3)
• Installing boto3 (1.17.36)
• Installing compress-pickle (1.2.0)
• Installing decorator (4.4.2)
• Installing future (0.18.2)
• Installing fuzzyset (0.0.19)
• Installing google-auth-oauthlib (0.4.3)
• Installing hjson (3.0.2)
• Installing html-parser (0.2)
• Installing grpcio (1.32.0)

1617282655480 ip-172-28-11-109:0 DEBUG • Installing lxml (4.6.3)
• Installing markdown (3.3.4)
• Installing orderedmultidict (1.0.1)
• Installing protobuf (3.15.6)
• Installing psycopg2-binary (2.8.6)
• Installing pyrsistent (0.17.3)
• Installing pyyaml (5.4.1)
• Installing rfc5424-logging-handler (1.4.3)
• Installing scikit-learn (0.23.2)
• Installing seaborn (0.10.1)
• Installing tensorboard-plugin-wit (1.8.0)
• Installing tldextract (2.2.3)
• Installing werkzeug (1.0.1)

1617282660768 ip-172-28-11-109:0 DEBUG • Installing astunparse (1.6.3)
• Installing click (7.1.2)
• Installing cython (0.29.14)
• Installing flatbuffers (1.12)
• Installing funcsigs (1.0.2)
• Installing gast (0.3.3)
• Installing google-pasta (0.2.0)
• Installing furl (2.1.0)
• Installing h5py (2.10.0)
• Installing humanfriendly (9.1)
• Installing isodate (0.6.0)
• Installing jsonschema (3.2.0)
• Installing keras-preprocessing (1.1.2)
• Installing llvmlite (0.34.0)
• Installing markupsafe (1.1.1)
• Installing more-itertools (8.7.0)
• Installing natebbcommon (3.1.0)
• Installing networkx (2.5)
• Installing opt-einsum (3.3.0)
• Installing packaging (20.9)
• Installing pathlib2 (2.3.5)
• Installing pluggy (0.13.1)
• Installing psutil (5.8.0)
• Installing py (1.10.0)
• Installing pyjwt (1.7.1)
• Installing regex (2021.3.17)
• Installing sentinels (1.0.0)
• Installing smart-open (4.2.0)
• Installing tensorboard (2.4.1)
• Installing tensorflow-estimator (2.4.0)
• Installing termcolor (1.1.0)
• Installing torch (1.6.0+cpu )
• Installing tqdm (4.59.0)
• Installing typing-extensions (3.7.4.3)
• Installing wcwidth (0.2.5)
• Installing wrapt (1.12.1)

1617282671036 ip-172-28-11-109:0 DEBUG • Installing ase (3.21.1)
• Installing gensim (3.8.3)
• Installing googledrivedownloader (0.4)
• Installing jinja2 (2.11.3)
• Installing mccabe (0.6.1)
• Installing mongomock (3.22.1)
• Installing natebbconnector (2.1.2)
• Installing nltk (3.5)
• Installing numba (0.51.2)
• Installing pycodestyle (2.7.0)
• Installing pyflakes (2.3.1)
• Installing pymongo (3.11.3)
• Installing pytest (5.4.3)
• Installing python-louvain (0.15)
• Installing pytorch-ranger (0.1.1)
• Installing rdflib (5.0.0)
• Installing tensorflow (2.4.1)
• Installing trains (0.16.4)

1617282686331 ip-172-28-11-109:0 DEBUG • Installing clearml (0.17.5)
• Installing flake8 (3.9.0)
• Installing natebblemae (4.1.0)
• Installing pytest-mock (3.5.1)
• Installing torch-cluster (1.5.8 )
• Installing pytest-mongodb (2.2.0)
• Installing torch-geometric (1.6.3)
• Installing torch-sparse (0.6.8 )
• Installing torch-spline-conv (1.2.0 )
• Installing torch-scatter (2.0.5 )
• Installing wget (3.2)
• Installing torch-optimizer (0.1.0)

Installing the current project: *** (2.2.0)
Adding venv into cache: /home/ubuntu/.clearml/venvs-builds/3.8
Running task id [4ce5aedc75404225b37eda2d9bd9ad8f]:
[***/dataset_creation]$ /home/ubuntu/.clearml/venvs-builds/3.8/bin/python -u create_dataset.py
Summary - installed python packages:
pip: []

Environment setup completed successfully

Starting Task Execution:

Traceback (most recent call last):
File "create_dataset.py", line 6, in <module>
from pymongo.collection import Collection
ModuleNotFoundError: No module named 'pymongo' `

  
  
Posted 3 years ago
175 Views
0 Answers
3 years ago
one year ago
Tags