clearml==0.17.4
` task dca2e3ded7fc4c28b342f912395ab9bc pulled from a238067927d04283842bc14cbdebdd86 by worker redacted-desktop:0
Running task 'dca2e3ded7fc4c28b342f912395ab9bc'
Storing stdout and stderr log to '/tmp/.clearml_agent_out.vjg4k7cj.txt', '/tmp/.clearml_agent_out.vjg4k7cj.txt'
Current configuration (clearml_agent v0.17.1, location: /tmp/.clearml_agent.us8pq3jj.cfg):
agent.worker_id = redacted-desktop:0
agent.worker_name = redacted-desktop
agent.force_git_ssh_protocol = false
agent.python_binary =
agent.package_manager.type = pip
agent.package_manager.pip_version = <20.2
agent.package_manager.system_site_packages = false
agent.package_manager.force_upgrade = false
agent.package_manager.conda_channels.0 = defaults
agent.package_manager.conda_channels.1 = conda-forge
agent.package_manager.conda_channels.2 = pytorch
agent.package_manager.torch_nightly = false
agent.venvs_dir = /home/redacted/.clearml/venvs-builds
agent.vcs_cache.enabled = true
agent.vcs_cache.path = /home/redacted/.clearml/vcs-cache
agent.venv_update.enabled = false
agent.pip_download_cache.enabled = true
agent.pip_download_cache.path = /home/redacted/.clearml/pip-download-cache
agent.translate_ssh = true
agent.reload_config = false
agent.docker_pip_cache = /home/redacted/.clearml/pip-cache
agent.docker_apt_cache = /home/redacted/.clearml/apt-cache
agent.docker_force_pull = false
agent.default_docker.image = ubuntu:20.04
agent.default_python = 3.7
agent.cuda_version = 112
agent.cudnn_version = 0
api.version = 1.5
api.verify_certificate = true
api.default_version = 1.5
api.http.max_req_size = 15728640
api.http.retries.total = 240
api.http.retries.connect = 240
api.http.retries.read = 240
api.http.retries.redirect = 240
api.http.retries.status = 240
api.http.retries.backoff_factor = 1.0
api.http.retries.backoff_max = 120.0
api.http.wait_on_maintenance_forever = true
api.http.pool_maxsize = 512
api.http.pool_connections = 512
api.api_server = redacted
api.web_server = redacted
api.files_server = redacted
api.credentials.access_key = redacted
api.host = redacted
sdk.storage.cache.default_base_dir = ~/.clearml/cache
sdk.storage.cache.size.min_free_bytes = 10GB
sdk.storage.direct_access.0.url = file://*
sdk.metrics.file_history_size = 100
sdk.metrics.matplotlib_untitled_history_size = 100
sdk.metrics.images.format = JPEG
sdk.metrics.images.quality = 87
sdk.metrics.images.subsampling = 0
sdk.metrics.tensorboard_single_series_per_graph = false
sdk.network.metrics.file_upload_threads = 4
sdk.network.metrics.file_upload_starvation_warning_sec = 120
sdk.network.iteration.max_retries_on_server_error = 5
sdk.network.iteration.retry_backoff_factor_sec = 10
sdk.aws.s3.key =
sdk.aws.s3.region =
sdk.aws.boto3.pool_connections = 512
sdk.aws.boto3.max_multipart_concurrency = 16
sdk.log.null_log_propagate = false
sdk.log.task_log_buffer_capacity = 66
sdk.log.disable_urllib3_info = true
sdk.development.task_reuse_time_window_in_hours = 72.0
sdk.development.vcs_repo_detect_async = true
sdk.development.store_uncommitted_code_diff = true
sdk.development.support_stopping = true
sdk.development.default_output_uri =
sdk.development.force_analyze_entire_repo = false
sdk.development.suppress_update_message = false
sdk.development.detect_with_pip_freeze = false
sdk.development.worker.report_period_sec = 2
sdk.development.worker.ping_period_sec = 30
sdk.development.worker.log_stdout = true
sdk.development.worker.report_global_mem_used = false
sdk.development.worker.console_cr_flush_period = 10
Executing task id [dca2e3ded7fc4c28b342f912395ab9bc]:
repository = git@github.com:redacted.git
branch = redacted
version_num = 8c5e5dc70630e83b35a90e04a0475ea495740a87
tag =
docker_cmd = None
entry_point = src/run_task.py
working_dir = .
Using base prefix '/home/redacted/miniconda3/envs/redacted'
New python executable in /home/redacted/.clearml/venvs-builds/3.7/bin/python3.7
Also creating executable in /home/redacted/.clearml/venvs-builds/3.7/bin/python
Installing setuptools, pip, wheel...
done.
Using cached repository in "/home/redacted/.clearml/vcs-cache/redacted.git.3f6461659473924a6515e793eacb9514/redacted.git"
From github.com:redacted
216a87b..8c5e5dc redacted -> origin/redacted
a91997f..8eb025a redacted -> origin/redacted
Note: switching to '8c5e5dc70630e83b35a90e04a0475ea495740a87'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:
git switch -c <new-branch-name>
Or undo this operation with:
git switch -
Turn off this advice by setting config variable advice.detachedHead to false
HEAD is now at 8c5e5dc Added clearml task
type: git
url: git@github.com:redacted/redacted.git
branch: HEAD
commit: 8c5e5dc70630e83b35a90e04a0475ea495740a87
root: /home/redacted/.clearml/venvs-builds/3.7/task_repository/redacted.git
Collecting pip<20.2
Using cached pip-20.1.1-py2.py3-none-any.whl (1.5 MB)
Installing collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 21.0.1
Uninstalling pip-21.0.1:
Successfully uninstalled pip-21.0.1
Successfully installed pip-20.1.1
Collecting Cython
Using cached Cython-0.29.21-cp37-cp37m-manylinux1_x86_64.whl (2.0 MB)
Installing collected packages: Cython
Successfully installed Cython-0.29.21
Collecting clearml==0.17.4
Using cached clearml-0.17.4-py2.py3-none-any.whl (873 kB)
Collecting attrs>=18.0
Using cached attrs-20.3.0-py2.py3-none-any.whl (49 kB)
Collecting pyjwt<3.0.0,>=1.6.4
Using cached PyJWT-2.0.1-py3-none-any.whl (15 kB)
Collecting pyparsing>=2.0.3
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting Pillow>=4.1.1
Using cached Pillow-8.1.0-cp37-cp37m-manylinux1_x86_64.whl (2.2 MB)
Collecting furl>=2.0.0
Using cached furl-2.1.0-py2.py3-none-any.whl (20 kB)
Collecting numpy>=1.10
Using cached numpy-1.20.0-cp37-cp37m-manylinux2010_x86_64.whl (15.3 MB)
Collecting urllib3>=1.21.1
Using cached urllib3-1.26.3-py2.py3-none-any.whl (137 kB)
Collecting six>=1.11.0
Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Processing /home/redacted/.cache/pip/wheels/56/b0/fe/4410d17b32f1f0c3cf54cdfb2bc04d7b4b8f4ae377e2229ba0/future-0.18.2-py3-none-any.whl
Collecting PyYAML>=3.12
Using cached PyYAML-5.4.1-cp37-cp37m-manylinux1_x86_64.whl (636 kB)
Collecting requests>=2.20.0
Using cached requests-2.25.1-py2.py3-none-any.whl (61 kB)
Collecting python-dateutil>=2.6.1
Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting psutil>=3.4.2
Using cached psutil-5.8.0-cp37-cp37m-manylinux2010_x86_64.whl (296 kB)
Collecting jsonschema>=2.6.0
Using cached jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)
Collecting pathlib2>=2.3.0
Using cached pathlib2-2.3.5-py2.py3-none-any.whl (18 kB)
Collecting requests-file>=1.4.2
Using cached requests_file-1.5.1-py2.py3-none-any.whl (3.7 kB)
Collecting humanfriendly>=2.1
Using cached humanfriendly-9.1-py2.py3-none-any.whl (86 kB)
Collecting orderedmultidict>=1.0.1
Using cached orderedmultidict-1.0.1-py2.py3-none-any.whl (11 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.12.5-py2.py3-none-any.whl (147 kB)
Collecting chardet<5,>=3.0.2
Using cached chardet-4.0.0-py2.py3-none-any.whl (178 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Requirement already satisfied: setuptools in /home/redacted/.clearml/venvs-builds/3.7/lib/python3.7/site-packages (from jsonschema>=2.6.0->clearml==0.17.4->-r /tmp/cached-reqsiubxqhl6.txt (line 10)) (53.0.0)
Processing /home/redacted/.cache/pip/wheels/a5/52/bf/71258a1d7b3c8cbe1ee53f9314c6f65f20385481eaee573cc5/pyrsistent-0.17.3-cp37-cp37m-linux_x86_64.whl
Collecting importlib-metadata; python_version < "3.8"
Using cached importlib_metadata-3.4.0-py3-none-any.whl (10 kB)
Collecting typing-extensions>=3.6.4; python_version < "3.8"
Using cached typing_extensions-3.7.4.3-py3-none-any.whl (22 kB)
Collecting zipp>=0.5
Using cached zipp-3.4.0-py3-none-any.whl (5.2 kB)
Installing collected packages: attrs, pyjwt, pyparsing, Pillow, six, orderedmultidict, furl, numpy, urllib3, future, PyYAML, certifi, chardet, idna, requests, python-dateutil, psutil, pyrsistent, typing-extensions, zipp, importlib-metadata, jsonschema, pathlib2, requests-file, humanfriendly, clearml
Successfully installed Pillow-8.1.0 PyYAML-5.4.1 attrs-20.3.0 certifi-2020.12.5 chardet-4.0.0 clearml-0.17.4 furl-2.1.0 future-0.18.2 humanfriendly-9.1 idna-2.10 importlib-metadata-3.4.0 jsonschema-3.2.0 numpy-1.20.0 orderedmultidict-1.0.1 pathlib2-2.3.5 psutil-5.8.0 pyjwt-2.0.1 pyparsing-2.4.7 pyrsistent-0.17.3 python-dateutil-2.8.1 requests-2.25.1 requests-file-1.5.1 six-1.15.0 typing-extensions-3.7.4.3 urllib3-1.26.3 zipp-3.4.0
Running task id [dca2e3ded7fc4c28b342f912395ab9bc]:
[.]$ /home/redacted/.clearml/venvs-builds/3.7/bin/python -u /home/redacted/.clearml/venvs-builds/3.7/task_repository/redacted.git/src/run_task.py
Summary - installed python packages:
pip:
- attrs==20.3.0
- certifi==2020.12.5
- chardet==4.0.0
- clearml==0.17.4
- Cython==0.29.21
- furl==2.1.0
- future==0.18.2
- humanfriendly==9.1
- idna==2.10
- importlib-metadata==3.4.0
- jsonschema==3.2.0
- numpy==1.20.0
- orderedmultidict==1.0.1
- pathlib2==2.3.5
- Pillow==8.1.0
- psutil==5.8.0
- PyJWT==2.0.1
- pyparsing==2.4.7
- pyrsistent==0.17.3
- python-dateutil==2.8.1
- PyYAML==5.4.1
- requests==2.25.1
- requests-file==1.5.1
- six==1.15.0
- typing-extensions==3.7.4.3
- urllib3==1.26.3
- zipp==3.4.0
Environment setup completed successfully
Starting Task Execution:
DONE: Running task 'dca2e3ded7fc4c28b342f912395ab9bc', exit status 0
No tasks in queue a238067927d04283842bc14cbdebdd86
No tasks in Queues, sleeping for 5.0 seconds
No tasks in queue a238067927d04283842bc14cbdebdd86
No tasks in Queues, sleeping for 5.0 seconds `
ReassuredTiger98 when you look for task "dca2e3ded7fc4c28b342f912395ab9bc" there are no artifacts ?
Could you add some prints? this should have worked...
Hi ReassuredTiger98
Could you add some print ? before / after the artifact upload?
Also what's the clearml version you are using ?
When I go into the GUI there are no artifacts displayed.
ReassuredTiger98 I'm trying to debug what's going on, because it should have worked.
Regrading Prints ...
` from clearml import Task
from time import sleep
def main():
task = Task.init(project_name="test", task_name="test")
d = {"a": "1"}
print('uploading artifact')
task.upload_artifact("myArtifact", d)
print('done uploading artifact')
# not sure if this helps but it won'r hurt to debug
sleep(3.0)
if name == "main":
main() `
Local execution output:ClearML Task: created new task id=855948f5d73c47e2ae37bb821385e15b ======> WARNING! Git diff to large to store (2190kb), skipping uncommitted changes <====== ClearML results page: uploading artifact done uploading artifact 2021-02-05 16:24:56,112 - clearml.Task - INFO - Waiting to finish uploads 2021-02-05 16:24:58,499 - clearml.Task - INFO - Finished uploading
Could you elaborate on that:
"So the agent failed to actually restore it from the git (files that are not added are not considered part of the git diff, this is usually git behavior)."
Okay that look s good, now in the UI start here and then get to the artifacts Tab,
Is it there ?
My agent shows the same as before:
` ...
Environment setup completed successfully
Starting Task Execution:
DONE: Running task 'aff7c6605b7243d38968f95b4351b127', exit status 0 `
However, I cloned the experiment again via the web UI. Then I enqueued it.
Yes, that looks alright. Similar to before. Local execution works.
ReassuredTiger98
Okay, but you should have had the prints ...uploading artifactanddone uploading artifactSo I suspect something is going on with the agent.
Did you manage to run any experiment on this agent ?
EDIT: Can you try with artifacts example we have on the repo:
https://github.com/allegroai/clearml/blob/master/examples/reporting/artifacts.py
Could you run your code not from the git repository.
I have a theory, you never actually added the entry point file to the git repo, so the agent never actually installed it, and it just did nothing (it should have reported an error, I'll look into it)
WDYT?
Seems possible because I didn't know I had to specify an entrypoint somewhere. I will do some additional tests.
Okay there should not be any difference ... 😞
And in the web UI artifacts is still empty.
ReassuredTiger98
Can you explain what you meant byÂ
entropy point file?
There is no need to specify entry point file.
It is automatically detected when you run the Code manually on your machine.
My assumption was that the file "src/run_task.py" (based on your log) is just a test file, and hence was not added top the repository. So the agent failed to actually restore it from the git (files that are not added are not considered part of the git diff, this is usually git behavior). meaning the agent was not able to run your code.
Make sense ?
Great!
I'll make sure the agent outputs the proper error 🙂
Can you explain what you meant by entropy point file? In a new git repository my code works fine.
Nvm. I think I understood. When the file has never been added to repository it is not tracked.
When I add the file the to repo it works fine just like you said.