(A Regular Experiment Did Execute In The Remote Agent, I Only Get This With The Pipe)

Unanswered

Full console log of the worker:

No tasks in Queues, sleeping for 5.0 seconds
No tasks in queue b5fe1e72614247f7a77e5f6cdac35580
No tasks in Queues, sleeping for 5.0 seconds
task 30ad27a7a1244b6e8aa722d81cb6015c pulled from b5fe1e72614247f7a77e5f6cdac35580 by worker NLEIN-315GNH2:0
Running task '30ad27a7a1244b6e8aa722d81cb6015c'
Storing stdout and stderr log to '/tmp/.clearml_agent_out.sppvun4p.txt', '/tmp/.clearml_agent_out.sppvun4p.txt'
Current configuration (clearml_agent v1.4.1, location: /tmp/.clearml_agent.gss2zozj.cfg):

sdk.storage.cache.default_base_dir = ~/.clearml/cache
sdk.storage.cache.size.min_free_bytes = 10GB
sdk.storage.direct_access.0.url = file://*
sdk.metrics.file_history_size = 100
sdk.metrics.matplotlib_untitled_history_size = 100
sdk.metrics.images.format = JPEG
sdk.metrics.images.quality = 87
sdk.metrics.images.subsampling = 0
sdk.metrics.tensorboard_single_series_per_graph = false
sdk.network.metrics.file_upload_threads = 4
sdk.network.metrics.file_upload_starvation_warning_sec = 120
sdk.network.iteration.max_retries_on_server_error = 5
sdk.network.iteration.retry_backoff_factor_sec = 10
sdk.aws.s3.key =
sdk.aws.s3.region =
sdk.aws.boto3.pool_connections = 512
sdk.aws.boto3.max_multipart_concurrency = 16
sdk.log.null_log_propagate = false
sdk.log.task_log_buffer_capacity = 66
sdk.log.disable_urllib3_info = true
sdk.development.task_reuse_time_window_in_hours = 72.0
sdk.development.vcs_repo_detect_async = true
sdk.development.store_uncommitted_code_diff = true
sdk.development.support_stopping = true
sdk.development.default_output_uri =
sdk.development.force_analyze_entire_repo = false
sdk.development.suppress_update_message = false
sdk.development.detect_with_pip_freeze = false
sdk.development.worker.report_period_sec = 2
sdk.development.worker.ping_period_sec = 30
sdk.development.worker.log_stdout = true
sdk.development.worker.report_global_mem_used = false
agent.worker_id = NLEIN-315GNH2:0
agent.worker_name = NLEIN-315GNH2
agent.force_git_ssh_protocol = false
agent.python_binary =
agent.package_manager.type = pip
agent.package_manager.pip_version = <20.2
agent.package_manager.system_site_packages = false
agent.package_manager.force_upgrade = false
agent.package_manager.conda_channels.0 = pytorch
agent.package_manager.conda_channels.1 = conda-forge
agent.package_manager.conda_channels.2 = defaults
agent.package_manager.priority_optional_packages.0 = pygobject
agent.package_manager.torch_nightly = false
agent.venvs_dir = /home/thermo/.clearml/venvs-builds
agent.venvs_cache.max_entries = 10
agent.venvs_cache.free_space_threshold_gb = 2.0
agent.vcs_cache.enabled = true
agent.vcs_cache.path = /home/thermo/.clearml/vcs-cache
agent.venv_update.enabled = false
agent.pip_download_cache.enabled = true
agent.pip_download_cache.path = /home/thermo/.clearml/pip-download-cache
agent.translate_ssh = true
agent.reload_config = false
agent.docker_pip_cache = /home/thermo/.clearml/pip-cache
agent.docker_apt_cache = /home/thermo/.clearml/apt-cache
agent.docker_force_pull = false
agent.default_docker.image = nvidia/cuda:10.2-cudnn7-runtime-ubuntu18.04
agent.enable_task_env = false
agent.hide_docker_command_env_vars.enabled = true
agent.hide_docker_command_env_vars.parse_embedded_urls = true
agent.abort_callback_max_timeout = 1800
agent.docker_internal_mounts.sdk_cache = /clearml_agent_cache
agent.docker_internal_mounts.apt_cache = /var/cache/apt/archives
agent.docker_internal_mounts.ssh_folder = ~/.ssh
agent.docker_internal_mounts.ssh_ro_folder = /.ssh
agent.docker_internal_mounts.pip_cache = /root/.cache/pip
agent.docker_internal_mounts.poetry_cache = /root/.cache/pypoetry
agent.docker_internal_mounts.vcs_cache = /root/.clearml/vcs-cache
agent.docker_internal_mounts.venv_build = ~/.clearml/venvs-builds
agent.docker_internal_mounts.pip_download = /root/.clearml/pip-download-cache
agent.apply_environment = true
agent.apply_files = true
agent.custom_build_script =
agent.git_user = MichaelThermo
agent.default_python = 3.8
agent.cuda_version = 0
agent.cudnn_version = 0
api.version = 1.5
api.verify_certificate = true
api.default_version = 1.5
api.http.max_req_size = 15728640
api.http.retries.total = 240
api.http.retries.connect = 240
api.http.retries.read = 240
api.http.retries.redirect = 240
api.http.retries.status = 240
api.http.retries.backoff_factor = 1.0
api.http.retries.backoff_max = 120.0
api.http.wait_on_maintenance_forever = true
api.http.pool_maxsize = 512
api.http.pool_connections = 512
api.api_server = https://api.clear.ml
api.web_server = https://app.clear.ml
api.files_server = https://files.clear.ml
api.credentials.access_key = HCH1PO3TF2EZY0226XUS
api.host = https://api.clear.ml

Executing task id [30ad27a7a1244b6e8aa722d81cb6015c]:
repository =
branch =
version_num =
tag =
docker_cmd =
entry_point = controller.py
working_dir = .

::: Python virtual environment cache is disabled. To accelerate spin-up time set agent.venvs_cache.path=~/.clearml/venvs-cache :::

created virtual environment CPython3.8.0.final.0-64 in 239ms
creator CPython3Posix(dest=/home/thermo/.clearml/venvs-builds/3.8, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/thermo/.local/share/virtualenv)
added seed packages: pip==22.3, setuptools==65.5.0, wheel==0.37.1
activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

Collecting pip<20.2
Using cached pip-20.1.1-py2.py3-none-any.whl (1.5 MB)
Installing collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 22.3
Uninstalling pip-22.3:
Successfully uninstalled pip-22.3
Successfully installed pip-20.1.1
Collecting Cython
Using cached Cython-0.29.32-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (1.9 MB)
Installing collected packages: Cython
Successfully installed Cython-0.29.32
Collecting attrs==21.4.0
Using cached attrs-21.4.0-py2.py3-none-any.whl (60 kB)
Collecting pathlib2==2.3.7.post1
Using cached pathlib2-2.3.7.post1-py2.py3-none-any.whl (18 kB)
Collecting six==1.16.0
Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting clearml==1.7.2
Using cached clearml-1.7.2-py2.py3-none-any.whl (950 kB)
Collecting pyjwt<2.5.0,>=2.4.0; python_version > "3.5"
Using cached PyJWT-2.4.0-py3-none-any.whl (18 kB)
Collecting jsonschema>=2.6.0
Using cached jsonschema-4.17.0-py3-none-any.whl (83 kB)
Collecting urllib3>=1.21.1
Using cached urllib3-1.26.12-py2.py3-none-any.whl (140 kB)
Collecting psutil>=3.4.2
Using cached psutil-5.9.3-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (295 kB)
Collecting python-dateutil>=2.6.1
Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting furl>=2.0.0
Using cached furl-2.1.3-py2.py3-none-any.whl (20 kB)
Collecting pyparsing>=2.0.3
Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB)
Collecting Pillow>=4.1.1
Using cached Pillow-9.3.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.2 MB)
Collecting numpy>=1.10
Using cached numpy-1.23.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.1 MB)
Collecting PyYAML>=3.12
Using cached PyYAML-6.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (701 kB)
Collecting requests>=2.20.0
Using cached requests-2.28.1-py3-none-any.whl (62 kB)
Collecting pkgutil-resolve-name>=1.3.10; python_version < "3.9"
Using cached pkgutil_resolve_name-1.3.10-py3-none-any.whl (4.7 kB)
Collecting pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0
Using cached pyrsistent-0.19.1-py3-none-any.whl (57 kB)
Collecting importlib-resources>=1.4.0; python_version < "3.9"
Using cached importlib_resources-5.10.0-py3-none-any.whl (34 kB)
Collecting orderedmultidict>=1.0.1
Using cached orderedmultidict-1.0.1-py2.py3-none-any.whl (11 kB)
Collecting charset-normalizer<3,>=2
Using cached charset_normalizer-2.1.1-py3-none-any.whl (39 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2022.9.24-py3-none-any.whl (161 kB)
Collecting idna<4,>=2.5
Using cached idna-3.4-py3-none-any.whl (61 kB)
Collecting zipp>=3.1.0; python_version < "3.10"
Using cached zipp-3.10.0-py3-none-any.whl (6.2 kB)
Installing collected packages: attrs, six, pathlib2, pyjwt, pkgutil-resolve-name, pyrsistent, zipp, importlib-resources, jsonschema, urllib3, psutil, python-dateutil, orderedmultidict, furl, pyparsing, Pillow, numpy, PyYAML, charset-normalizer, certifi, idna, requests, clearml
Successfully installed Pillow-9.3.0 PyYAML-6.0 attrs-21.4.0 certifi-2022.9.24 charset-normalizer-2.1.1 clearml-1.7.2 furl-2.1.3 idna-3.4 importlib-resources-5.10.0 jsonschema-4.17.0 numpy-1.23.4 orderedmultidict-1.0.1 pathlib2-2.3.7.post1 pkgutil-resolve-name-1.3.10 psutil-5.9.3 pyjwt-2.4.0 pyparsing-3.0.9 pyrsistent-0.19.1 python-dateutil-2.8.2 requests-2.28.1 six-1.16.0 urllib3-1.26.12 zipp-3.10.0
Adding venv into cache: /home/thermo/.clearml/venvs-builds/3.8
Running task id [30ad27a7a1244b6e8aa722d81cb6015c]:
[.]$ /home/thermo/.clearml/venvs-builds/3.8/bin/python -u /home/thermo/.clearml/venvs-builds/3.8/code/controller.py
Summary - installed python packages:
pip:

attrs==21.4.0
certifi==2022.9.24
charset-normalizer==2.1.1
clearml==1.7.2
Cython==0.29.32
furl==2.1.3
idna==3.4
importlib-resources==5.10.0
jsonschema==4.17.0
numpy==1.23.4
orderedmultidict==1.0.1
pathlib2==2.3.7.post1
Pillow==9.3.0
pkgutil-resolve-name==1.3.10
psutil==5.9.3
PyJWT==2.4.0
pyparsing==3.0.9
pyrsistent==0.19.1
python-dateutil==2.8.2
PyYAML==6.0
requests==2.28.1
six==1.16.0
urllib3==1.26.12
zipp==3.10.0

Environment setup completed successfully

Starting Task Execution:

Traceback (most recent call last):
File "/home/thermo/.clearml/venvs-builds/3.8/code/controller.py", line 20, in <module>
from .job import LocalClearmlJob, RunningJob, BaseJob
ImportError: attempted relative import with no known parent package

Leaving process id 1273
DONE: Running task '30ad27a7a1244b6e8aa722d81cb6015c', exit status 1
Process failed, exit code 1No tasks in queue b5fe1e72614247f7a77e5f6cdac35580
No tasks in Queues, sleeping for 5.0 seconds
No tasks in queue b5fe1e72614247f7a77e5f6cdac35580

  				
Posted 
	2 years ago

					More
				  		
  Report
		
					HollowDeer18
				
					0
					 × 1

167 Views

0 Answers

2 years ago

one year ago