Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
I Have A Self-Hosted Clearm-Server And And Clearml-Agent Started With

I have a self-hosted clearm-server and and clearml-agent started with clearml-agent daemon --queue default --foreground
I try to execute the following simple program:
` from clearml import Task

def main():
task = Task.init(project_name="test", task_name="test")

d = {"a": "1"}
task.upload_artifact("myArtifact", d)

if name == "main":
main() Locally it works well, I can see the artifact in the web gui. However, when I clone the experiment in the web gui and then submit it to my default queue, no artifact is generated. My agent only outputs all the install stuff and then Environment setup completed successfully

Starting Task Execution:

DONE: Running task 'ca6d1832b9e247178c35ad909c73bd50', exit status 0 `Any idea, why it is not creating the artifact?

  
  
Posted 3 years ago
Votes Newest

Answers 31


Hi ReassuredTiger98
Could you add some print ? before / after the artifact upload?
Also what's the clearml version you are using ?

  
  
Posted 3 years ago

When I add the file the to repo it works fine just like you said.

  
  
Posted 3 years ago

Thank you for all the help!

  
  
Posted 3 years ago

Makes sense.

  
  
Posted 3 years ago

image
image

  
  
Posted 3 years ago

Great!
I'll make sure the agent outputs the proper error 🙂

  
  
Posted 3 years ago

Nvm. I think I understood. When the file has never been added to repository it is not tracked.

  
  
Posted 3 years ago

Okay that look s good, now in the UI start here and then get to the artifacts Tab,
Is it there ?

  
  
Posted 3 years ago

clearml==0.17.4
` task dca2e3ded7fc4c28b342f912395ab9bc pulled from a238067927d04283842bc14cbdebdd86 by worker redacted-desktop:0
Running task 'dca2e3ded7fc4c28b342f912395ab9bc'
Storing stdout and stderr log to '/tmp/.clearml_agent_out.vjg4k7cj.txt', '/tmp/.clearml_agent_out.vjg4k7cj.txt'
Current configuration (clearml_agent v0.17.1, location: /tmp/.clearml_agent.us8pq3jj.cfg):

agent.worker_id = redacted-desktop:0
agent.worker_name = redacted-desktop
agent.force_git_ssh_protocol = false
agent.python_binary =
agent.package_manager.type = pip
agent.package_manager.pip_version = <20.2
agent.package_manager.system_site_packages = false
agent.package_manager.force_upgrade = false
agent.package_manager.conda_channels.0 = defaults
agent.package_manager.conda_channels.1 = conda-forge
agent.package_manager.conda_channels.2 = pytorch
agent.package_manager.torch_nightly = false
agent.venvs_dir = /home/redacted/.clearml/venvs-builds
agent.vcs_cache.enabled = true
agent.vcs_cache.path = /home/redacted/.clearml/vcs-cache
agent.venv_update.enabled = false
agent.pip_download_cache.enabled = true
agent.pip_download_cache.path = /home/redacted/.clearml/pip-download-cache
agent.translate_ssh = true
agent.reload_config = false
agent.docker_pip_cache = /home/redacted/.clearml/pip-cache
agent.docker_apt_cache = /home/redacted/.clearml/apt-cache
agent.docker_force_pull = false
agent.default_docker.image = ubuntu:20.04
agent.default_python = 3.7
agent.cuda_version = 112
agent.cudnn_version = 0
api.version = 1.5
api.verify_certificate = true
api.default_version = 1.5
api.http.max_req_size = 15728640
api.http.retries.total = 240
api.http.retries.connect = 240
api.http.retries.read = 240
api.http.retries.redirect = 240
api.http.retries.status = 240
api.http.retries.backoff_factor = 1.0
api.http.retries.backoff_max = 120.0
api.http.wait_on_maintenance_forever = true
api.http.pool_maxsize = 512
api.http.pool_connections = 512
api.api_server = redacted
api.web_server = redacted
api.files_server = redacted
api.credentials.access_key = redacted
api.host = redacted
sdk.storage.cache.default_base_dir = ~/.clearml/cache
sdk.storage.cache.size.min_free_bytes = 10GB
sdk.storage.direct_access.0.url = file://*
sdk.metrics.file_history_size = 100
sdk.metrics.matplotlib_untitled_history_size = 100
sdk.metrics.images.format = JPEG
sdk.metrics.images.quality = 87
sdk.metrics.images.subsampling = 0
sdk.metrics.tensorboard_single_series_per_graph = false
sdk.network.metrics.file_upload_threads = 4
sdk.network.metrics.file_upload_starvation_warning_sec = 120
sdk.network.iteration.max_retries_on_server_error = 5
sdk.network.iteration.retry_backoff_factor_sec = 10
sdk.aws.s3.key =
sdk.aws.s3.region =
sdk.aws.boto3.pool_connections = 512
sdk.aws.boto3.max_multipart_concurrency = 16
sdk.log.null_log_propagate = false
sdk.log.task_log_buffer_capacity = 66
sdk.log.disable_urllib3_info = true
sdk.development.task_reuse_time_window_in_hours = 72.0
sdk.development.vcs_repo_detect_async = true
sdk.development.store_uncommitted_code_diff = true
sdk.development.support_stopping = true
sdk.development.default_output_uri =
sdk.development.force_analyze_entire_repo = false
sdk.development.suppress_update_message = false
sdk.development.detect_with_pip_freeze = false
sdk.development.worker.report_period_sec = 2
sdk.development.worker.ping_period_sec = 30
sdk.development.worker.log_stdout = true
sdk.development.worker.report_global_mem_used = false
sdk.development.worker.console_cr_flush_period = 10

Executing task id [dca2e3ded7fc4c28b342f912395ab9bc]:
repository = git@github.com:redacted.git
branch = redacted
version_num = 8c5e5dc70630e83b35a90e04a0475ea495740a87
tag =
docker_cmd = None
entry_point = src/run_task.py
working_dir = .

Using base prefix '/home/redacted/miniconda3/envs/redacted'
New python executable in /home/redacted/.clearml/venvs-builds/3.7/bin/python3.7
Also creating executable in /home/redacted/.clearml/venvs-builds/3.7/bin/python
Installing setuptools, pip, wheel...
done.

Using cached repository in "/home/redacted/.clearml/vcs-cache/redacted.git.3f6461659473924a6515e793eacb9514/redacted.git"
From github.com:redacted
216a87b..8c5e5dc redacted -> origin/redacted
a91997f..8eb025a redacted -> origin/redacted
Note: switching to '8c5e5dc70630e83b35a90e04a0475ea495740a87'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

git switch -c <new-branch-name>

Or undo this operation with:

git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 8c5e5dc Added clearml task
type: git
url: git@github.com:redacted/redacted.git
branch: HEAD
commit: 8c5e5dc70630e83b35a90e04a0475ea495740a87
root: /home/redacted/.clearml/venvs-builds/3.7/task_repository/redacted.git

Collecting pip<20.2
Using cached pip-20.1.1-py2.py3-none-any.whl (1.5 MB)
Installing collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 21.0.1
Uninstalling pip-21.0.1:
Successfully uninstalled pip-21.0.1
Successfully installed pip-20.1.1
Collecting Cython
Using cached Cython-0.29.21-cp37-cp37m-manylinux1_x86_64.whl (2.0 MB)
Installing collected packages: Cython
Successfully installed Cython-0.29.21
Collecting clearml==0.17.4
Using cached clearml-0.17.4-py2.py3-none-any.whl (873 kB)
Collecting attrs>=18.0
Using cached attrs-20.3.0-py2.py3-none-any.whl (49 kB)
Collecting pyjwt<3.0.0,>=1.6.4
Using cached PyJWT-2.0.1-py3-none-any.whl (15 kB)
Collecting pyparsing>=2.0.3
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting Pillow>=4.1.1
Using cached Pillow-8.1.0-cp37-cp37m-manylinux1_x86_64.whl (2.2 MB)
Collecting furl>=2.0.0
Using cached furl-2.1.0-py2.py3-none-any.whl (20 kB)
Collecting numpy>=1.10
Using cached numpy-1.20.0-cp37-cp37m-manylinux2010_x86_64.whl (15.3 MB)
Collecting urllib3>=1.21.1
Using cached urllib3-1.26.3-py2.py3-none-any.whl (137 kB)
Collecting six>=1.11.0
Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Processing /home/redacted/.cache/pip/wheels/56/b0/fe/4410d17b32f1f0c3cf54cdfb2bc04d7b4b8f4ae377e2229ba0/future-0.18.2-py3-none-any.whl
Collecting PyYAML>=3.12
Using cached PyYAML-5.4.1-cp37-cp37m-manylinux1_x86_64.whl (636 kB)
Collecting requests>=2.20.0
Using cached requests-2.25.1-py2.py3-none-any.whl (61 kB)
Collecting python-dateutil>=2.6.1
Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting psutil>=3.4.2
Using cached psutil-5.8.0-cp37-cp37m-manylinux2010_x86_64.whl (296 kB)
Collecting jsonschema>=2.6.0
Using cached jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)
Collecting pathlib2>=2.3.0
Using cached pathlib2-2.3.5-py2.py3-none-any.whl (18 kB)
Collecting requests-file>=1.4.2
Using cached requests_file-1.5.1-py2.py3-none-any.whl (3.7 kB)
Collecting humanfriendly>=2.1
Using cached humanfriendly-9.1-py2.py3-none-any.whl (86 kB)
Collecting orderedmultidict>=1.0.1
Using cached orderedmultidict-1.0.1-py2.py3-none-any.whl (11 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.12.5-py2.py3-none-any.whl (147 kB)
Collecting chardet<5,>=3.0.2
Using cached chardet-4.0.0-py2.py3-none-any.whl (178 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Requirement already satisfied: setuptools in /home/redacted/.clearml/venvs-builds/3.7/lib/python3.7/site-packages (from jsonschema>=2.6.0->clearml==0.17.4->-r /tmp/cached-reqsiubxqhl6.txt (line 10)) (53.0.0)
Processing /home/redacted/.cache/pip/wheels/a5/52/bf/71258a1d7b3c8cbe1ee53f9314c6f65f20385481eaee573cc5/pyrsistent-0.17.3-cp37-cp37m-linux_x86_64.whl
Collecting importlib-metadata; python_version < "3.8"
Using cached importlib_metadata-3.4.0-py3-none-any.whl (10 kB)
Collecting typing-extensions>=3.6.4; python_version < "3.8"
Using cached typing_extensions-3.7.4.3-py3-none-any.whl (22 kB)
Collecting zipp>=0.5
Using cached zipp-3.4.0-py3-none-any.whl (5.2 kB)
Installing collected packages: attrs, pyjwt, pyparsing, Pillow, six, orderedmultidict, furl, numpy, urllib3, future, PyYAML, certifi, chardet, idna, requests, python-dateutil, psutil, pyrsistent, typing-extensions, zipp, importlib-metadata, jsonschema, pathlib2, requests-file, humanfriendly, clearml
Successfully installed Pillow-8.1.0 PyYAML-5.4.1 attrs-20.3.0 certifi-2020.12.5 chardet-4.0.0 clearml-0.17.4 furl-2.1.0 future-0.18.2 humanfriendly-9.1 idna-2.10 importlib-metadata-3.4.0 jsonschema-3.2.0 numpy-1.20.0 orderedmultidict-1.0.1 pathlib2-2.3.5 psutil-5.8.0 pyjwt-2.0.1 pyparsing-2.4.7 pyrsistent-0.17.3 python-dateutil-2.8.1 requests-2.25.1 requests-file-1.5.1 six-1.15.0 typing-extensions-3.7.4.3 urllib3-1.26.3 zipp-3.4.0
Running task id [dca2e3ded7fc4c28b342f912395ab9bc]:
[.]$ /home/redacted/.clearml/venvs-builds/3.7/bin/python -u /home/redacted/.clearml/venvs-builds/3.7/task_repository/redacted.git/src/run_task.py
Summary - installed python packages:
pip:

  • attrs==20.3.0
  • certifi==2020.12.5
  • chardet==4.0.0
  • clearml==0.17.4
  • Cython==0.29.21
  • furl==2.1.0
  • future==0.18.2
  • humanfriendly==9.1
  • idna==2.10
  • importlib-metadata==3.4.0
  • jsonschema==3.2.0
  • numpy==1.20.0
  • orderedmultidict==1.0.1
  • pathlib2==2.3.5
  • Pillow==8.1.0
  • psutil==5.8.0
  • PyJWT==2.0.1
  • pyparsing==2.4.7
  • pyrsistent==0.17.3
  • python-dateutil==2.8.1
  • PyYAML==5.4.1
  • requests==2.25.1
  • requests-file==1.5.1
  • six==1.15.0
  • typing-extensions==3.7.4.3
  • urllib3==1.26.3
  • zipp==3.4.0

Environment setup completed successfully

Starting Task Execution:

DONE: Running task 'dca2e3ded7fc4c28b342f912395ab9bc', exit status 0
No tasks in queue a238067927d04283842bc14cbdebdd86
No tasks in Queues, sleeping for 5.0 seconds
No tasks in queue a238067927d04283842bc14cbdebdd86
No tasks in Queues, sleeping for 5.0 seconds `

  
  
Posted 3 years ago

ReassuredTiger98 when you look for task "dca2e3ded7fc4c28b342f912395ab9bc" there are no artifacts ?
Could you add some prints? this should have worked...

  
  
Posted 3 years ago

ReassuredTiger98 I'm trying to debug what's going on, because it should have worked.
Regrading Prints ...
` from clearml import Task
from time import sleep

def main():
task = Task.init(project_name="test", task_name="test")
d = {"a": "1"}
print('uploading artifact')
task.upload_artifact("myArtifact", d)
print('done uploading artifact')
# not sure if this helps but it won'r hurt to debug
sleep(3.0)
if name == "main":
main() `

  
  
Posted 3 years ago

When I go into the GUI there are no artifacts displayed.

  
  
Posted 3 years ago

What do you mean by prints?

  
  
Posted 3 years ago

Local execution output:
ClearML Task: created new task id=855948f5d73c47e2ae37bb821385e15b ======> WARNING! Git diff to large to store (2190kb), skipping uncommitted changes <====== ClearML results page: uploading artifact done uploading artifact 2021-02-05 16:24:56,112 - clearml.Task - INFO - Waiting to finish uploads 2021-02-05 16:24:58,499 - clearml.Task - INFO - Finished uploading

  
  
Posted 3 years ago

Thanks.

  
  
Posted 3 years ago

My agent shows the same as before:
` ...
Environment setup completed successfully

Starting Task Execution:

DONE: Running task 'aff7c6605b7243d38968f95b4351b127', exit status 0 `

  
  
Posted 3 years ago

However, I cloned the experiment again via the web UI. Then I enqueued it.

  
  
Posted 3 years ago

Yes, that looks alright. Similar to before. Local execution works.

  
  
Posted 3 years ago

And in the web UI artifacts is still empty.

  
  
Posted 3 years ago

So no change in behaviour.

  
  
Posted 3 years ago

Okay, this seems to work fine.

  
  
Posted 3 years ago

Sure, let me try.

  
  
Posted 3 years ago

Could you run your code not from the git repository.
I have a theory, you never actually added the entry point file to the git repo, so the agent never actually installed it, and it just did nothing (it should have reported an error, I'll look into it)
WDYT?

  
  
Posted 3 years ago

Seems possible because I didn't know I had to specify an entrypoint somewhere. I will do some additional tests.

  
  
Posted 3 years ago

Okay there should not be any difference ... 😞

  
  
Posted 3 years ago

ReassuredTiger98

Can you explain what you meant by 

entropy point file?

There is no need to specify entry point file.
It is automatically detected when you run the Code manually on your machine.
My assumption was that the file "src/run_task.py" (based on your log) is just a test file, and hence was not added top the repository. So the agent failed to actually restore it from the git (files that are not added are not considered part of the git diff, this is usually git behavior). meaning the agent was not able to run your code.
Make sense ?

  
  
Posted 3 years ago

ReassuredTiger98
Okay, but you should have had the prints ...
uploading artifactand
done uploading artifactSo I suspect something is going on with the agent.
Did you manage to run any experiment on this agent ?

EDIT: Can you try with artifacts example we have on the repo:
https://github.com/allegroai/clearml/blob/master/examples/reporting/artifacts.py

  
  
Posted 3 years ago

Could you elaborate on that:
"So the agent failed to actually restore it from the git (files that are not added are not considered part of the git diff, this is usually git behavior)."

  
  
Posted 3 years ago

Thanks!

  
  
Posted 3 years ago

Can you explain what you meant by entropy point file? In a new git repository my code works fine.

  
  
Posted 3 years ago
17K Views
31 Answers
3 years ago
7 months ago
Tags