AgitatedDove14 it was executed with Python 3 and I’m running in venv mode.
$ python --version Python 3.6.8 $ python repo/toy_workflow.py --logtostderr --logtoclearml --clearml_queue=ada_manual_jobs 2021-08-07 04:04:16,844 - clearml - WARNING - Switching to remote execution, output log page https://...
On the webpage logs I see this:
2021-08-07 04:04:12 ClearML Task: created new task id=f1092bcbe30249639122a49a9b3f9145 ClearML results page: `
2021-08-07 04:04:14
ClearML Monitor: GPU monitoring failed getting GPU reading, switching off GPU monitoring
2021-08-07 04:04:16
[INFO 2021-08-07 04:04:16,180] [git_manager.py:63] [MainThread] Pushed branch=clearml hash=57f382f51d124299788544b3e7afa11c4cba2d1f to upstream=origin.
2021-08-07 04:04:18
task f1092bcbe30249639122a49a9b3f9145 pulled from e78d2fdf2d5140b6b5c6678338c532bb by worker ada:gpu0
2021-08-07 04:04:23
Current configuration (clearml_agent v1.0.0, location: /tmp/.clearml_agent.ws8onrog.cfg):
sdk.storage.cache.default_base_dir = ~/.clearml/cache
sdk.storage.cache.size.min_free_bytes = 10GB
sdk.storage.direct_access.0.url = file://*
sdk.metrics.file_history_size = 100
sdk.metrics.matplotlib_untitled_history_size = 100
sdk.metrics.images.format = JPEG
sdk.metrics.images.quality = 87
sdk.metrics.images.subsampling = 0
sdk.metrics.tensorboard_single_series_per_graph = false
sdk.network.metrics.file_upload_threads = 4
sdk.network.metrics.file_upload_starvation_warning_sec = 120
sdk.network.iteration.max_retries_on_server_error = 5
sdk.network.iteration.retry_backoff_factor_sec = 10
sdk.aws.s3.key = AKIAI2C6JIPD657QPDRQ
sdk.aws.s3.region = us-west-2
sdk.aws.boto3.pool_connections = 512
sdk.aws.boto3.max_multipart_concurrency = 16
sdk.log.null_log_propagate = false
sdk.log.task_log_buffer_capacity = 66
sdk.log.disable_urllib3_info = true
sdk.development.task_reuse_time_window_in_hours = 72.0
sdk.development.vcs_repo_detect_async = true
sdk.development.store_uncommitted_code_diff = true
sdk.development.support_stopping = true
sdk.development.default_output_uri =
sdk.development.force_analyze_entire_repo = false
sdk.development.suppress_update_message = false
sdk.development.detect_with_pip_freeze = false
sdk.development.worker.report_period_sec = 2
sdk.development.worker.ping_period_sec = 30
sdk.development.worker.log_stdout = true
sdk.development.worker.report_global_mem_used = false
agent.worker_id = ada:gpu0
agent.worker_name = ada
agent.force_git_ssh_protocol = true
agent.python_binary =
agent.package_manager.type = pip3
agent.package_manager.pip_version = ==20.3.3
agent.package_manager.system_site_packages = true
agent.package_manager.force_upgrade = false
agent.package_manager.conda_channels.0 = pytorch
agent.package_manager.conda_channels.1 = conda-forge
agent.package_manager.conda_channels.2 = defaults
agent.package_manager.torch_nightly = false
agent.venvs_dir = /home/smjahad/.clearml/venvs-builds
agent.venvs_cache.max_entries = 10
agent.venvs_cache.free_space_threshold_gb = 2.0
agent.vcs_cache.enabled = true
agent.vcs_cache.path = /home/smjahad/.clearml/vcs-cache
agent.venv_update.enabled = false
agent.pip_download_cache.enabled = true
agent.pip_download_cache.path = /home/smjahad/.clearml/pip-download-cache
agent.translate_ssh = true
agent.reload_config = true
agent.docker_pip_cache = /home/smjahad/.clearml/pip-cache
agent.docker_apt_cache = /home/smjahad/.clearml/apt-cache
agent.docker_force_pull = false
agent.default_docker.image = nvidia/cuda:10.1-cudnn7-runtime-ubuntu18.04
agent.enable_task_env = false
agent.git_user =
agent.default_python = 3.6
agent.cuda_version = 102
agent.cudnn_version = 0
api.version = 1.5
api.verify_certificate = true
api.default_version = 1.5
api.http.max_req_size = 15728640
api.http.retries.total = 240
api.http.retries.connect = 240
api.http.retries.read = 240
api.http.retries.redirect = 240
api.http.retries.status = 240
api.http.retries.backoff_factor = 1.0
api.http.retries.backoff_max = 120.0
api.http.wait_on_maintenance_forever = true
api.http.pool_maxsize = 512
api.http.pool_connections = 512
api.api_server =
api.web_server =
api.files_server =
api.credentials.access_key = PI87JVO6OC12UIIS3J73
api.host =
Executing task id [f1092bcbe30249639122a49a9b3f9145]:
repository = git@github.com:mainuser/repo.git
branch = clearml
version_num = 57f382f51d124299788544b3e7afa11c4cba2d1f
tag =
docker_cmd = None
entry_point = repo/toy_workflow.py
working_dir = .
New python executable in /home/smjahad/.clearml/venvs-builds/3.6/bin/python2
Also creating executable in /home/smjahad/.clearml/venvs-builds/3.6/bin/python
Installing setuptools, pkg_resources, pip, wheel...done. `
If this doesn't help.
Go to your ~/clearml.conf file, at the bottom of the file you can add agent.python_binary and change it to to the location of python3.6 (you can run which python3.6 to get the full path):agent.python_binary: /full/path/to/python3.6
Are you running it in venv mode or docker mode?
that clearml-agent needs to be installed from system python mentioned anywhere in the docs, if not I suggest it gets added.
You are right, I will check and fix if not 🙂
Thank you so much for helping.
My pleasure
New python executable in /home/smjahad/.clearml/venvs-builds/3.6/bin/python2
This is the output of venv create this is odd.
Could it be that by accident you did:pip install cleamrl-agentand notpip3 install clearml-agentand now it is running on python2 (which would explain the error) ?
I would uninstall/reinstall on python3 to verify
BTW: do notice to install the agent on the system python packages and Not on any venv.
Just making sure, the original code was executed on python 3?
Can you send the full log? This is odd, it will by default use the python executable it (the agent) is running with.
Regardless you can specify the python executable to be used here:
https://github.com/allegroai/clearml-agent/blob/bd411a19843fbb1e063b131e830a4515233bdf04/docs/clearml.conf#L44
BTW:
This is very odd "~/.clearml/venvs-builds.3/3.6/bin/python" it thinks it is using "python 3.6" but it is linked with python 2.7 ...
No idea how that could happen
Issue seems fixed now, thanks! Is the fact that clearml-agent needs to be installed from system python mentioned anywhere in the docs, if not I suggest it gets added.
Thank you so much for helping.