Unanswered
When Trying To Run My Agent With Poetry As Package Manager (Could Also Be With Pip, But Definitely With Poetry), Strange Things Seem To Happen When Trying To Execute A Pipeline:
1721735525702 myHostName info ClearML Task: created new task id=a08289ce6f2e47f2afc4e9f4e8540575
ClearML results page:
1721735525998 myHostName info ClearML pipeline page:
Starting the pipeline in queue pipeline
1721735542512 dnsName:cpu:0 INFO task a08289ce6f2e47f2afc4e9f4e8540575 pulled from cbf8757ca3f44980a450f9ea8a1c300a by worker dnsName:cpu:0
1721735547620 dnsName:cpu:0 DEBUG DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): ipAddrOfClearMLServer:8008
DEBUG:urllib3.connectionpool:
"GET /auth.login HTTP/1.1" 200 611
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): ipAddrOfClearMLServer:8008
DEBUG:urllib3.connectionpool:
"GET /v2.5/tasks.get_by_id HTTP/1.1" 200 3225
DEBUG:urllib3.connectionpool:Resetting dropped connection: ipAddrOfClearMLServer
DEBUG:urllib3.connectionpool:
"GET /tasks.dequeue HTTP/1.1" 200 313
DEBUG:urllib3.connectionpool:Resetting dropped connection: ipAddrOfClearMLServer
DEBUG:urllib3.connectionpool:
"GET /v2.5/tasks.started HTTP/1.1" 200 330
DEBUG:urllib3.connectionpool:Resetting dropped connection: ipAddrOfClearMLServer
DEBUG:urllib3.connectionpool:
"GET /workers.status_report HTTP/1.1" 200 283
Running task 'a08289ce6f2e47f2afc4e9f4e8540575'
1721735551432 dnsName:cpu:0:service:a08289ce6f2e47f2afc4e9f4e8540575 DEBUG DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): ipAddrOfClearMLServer:8008
DEBUG:urllib3.connectionpool:
"GET /auth.login HTTP/1.1" 200 611
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): ipAddrOfClearMLServer:8008
DEBUG:urllib3.connectionpool:
"GET /v2.5/tasks.get_by_id HTTP/1.1" 200 3232
DEBUG:urllib3.connectionpool:Resetting dropped connection: ipAddrOfClearMLServer
DEBUG:urllib3.connectionpool:
"GET /tasks.dequeue HTTP/1.1" 200 313
DEBUG:urllib3.connectionpool:Resetting dropped connection: ipAddrOfClearMLServer
DEBUG:urllib3.connectionpool:
"GET /v2.5/tasks.started HTTP/1.1" 200 331
DEBUG:clearml_agent.session:Run by interpreter: /opt/clearml/.venv/bin/python3.11
Current configuration (clearml_agent v1.8.1, location: /tmp/.clearml_agent.c06iczcd.cfg):
----------------------
api.version = 1.5
api.verify_certificate = true
api.default_version = 1.5
api.http.max_req_size = 15728640
api.http.retries.total = 240
api.http.retries.connect = 240
api.http.retries.read = 240
api.http.retries.redirect = 240
api.http.retries.status = 240
api.http.retries.backoff_factor = 1.0
api.http.retries.backoff_max = 120.0
api.http.wait_on_maintenance_forever = true
api.http.pool_maxsize = 512
api.http.pool_connections = 512
api.auth.token_expiration_threshold_sec = ****
api.api_server =
api.web_server =
api.files_server =
api.credentials.access_key = XTM42TSDNPGYYO3GJDLS
api.credentials.secret_key = ****
api.host =
agent.worker_id = dnsName:cpu:0:service:a08289ce6f2e47f2afc4e9f4e8540575
agent.worker_name = dnsName
agent.force_git_ssh_protocol = false
agent.python_binary =
agent.package_manager.type = poetry
agent.package_manager.pip_version.0 = <20.2 ; python_version < '3.10'
agent.package_manager.pip_version.1 = <22.3 ; python_version >\= '3.10'
agent.package_manager.system_site_packages = false
agent.package_manager.force_upgrade = false
agent.package_manager.conda_channels.0 = pytorch
agent.package_manager.conda_channels.1 = conda-forge
agent.package_manager.conda_channels.2 = nvidia
agent.package_manager.conda_channels.3 = defaults
agent.package_manager.priority_optional_packages.0 = pygobject
agent.package_manager.torch_nightly = false
agent.package_manager.poetry_files_from_repo_working_dir = false
agent.venvs_dir = /home/clearml_user/.clearml/venvs-builds.2
agent.venvs_cache.max_entries = 10
agent.venvs_cache.free_space_threshold_gb = 2.0
agent.venvs_cache.path = ~/.clearml/venvs-cache
agent.vcs_cache.enabled = true
agent.vcs_cache.path = /home/clearml_user/.clearml/vcs-cache
agent.venv_update.enabled = false
agent.pip_download_cache.enabled = true
agent.pip_download_cache.path = /home/clearml_user/.clearml/pip-download-cache
agent.translate_ssh = true
agent.reload_config = false
agent.docker_pip_cache = /home/clearml_user/.clearml/pip-cache
agent.docker_apt_cache = /home/clearml_user/.clearml/apt-cache.2
agent.docker_force_pull = false
agent.default_docker.image = python:3.11.9-slim-bookworm
agent.enable_task_env = false
agent.sanitize_config_printout = ****
agent.hide_docker_command_env_vars.enabled = true
agent.hide_docker_command_env_vars.parse_embedded_urls = true
agent.abort_callback_max_timeout = 1800
agent.docker_internal_mounts.sdk_cache = /clearml_agent_cache
agent.docker_internal_mounts.apt_cache = /var/cache/apt/archives
agent.docker_internal_mounts.ssh_folder = ~/.ssh
agent.docker_internal_mounts.ssh_ro_folder = /.ssh
agent.docker_internal_mounts.pip_cache = /root/.cache/pip
agent.docker_internal_mounts.poetry_cache = /root/.cache/pypoetry
agent.docker_internal_mounts.vcs_cache = /root/.clearml/vcs-cache
agent.docker_internal_mounts.venv_build = ~/.clearml/venvs-builds
agent.docker_internal_mounts.pip_download = /root/.clearml/pip-download-cache
agent.apply_environment = true
agent.apply_files = true
agent.custom_build_script =
agent.disable_task_docker_override = false
agent.git_user =
agent.git_pass = ****
agent.debug = true
agent.default_python = 3.11
agent.cuda_version = 0
agent.cudnn_version = 0
sdk.storage.cache.default_base_dir = /home/clearml_user/.clearml/cache.2
sdk.storage.cache.size.min_free_bytes = 10GB
sdk.storage.direct_access.0.url = file://*
sdk.metrics.file_history_size = 100
sdk.metrics.matplotlib_untitled_history_size = 100
sdk.metrics.images.format = JPEG
sdk.metrics.images.quality = 87
sdk.metrics.images.subsampling = 0
sdk.metrics.tensorboard_single_series_per_graph = false
sdk.network.metrics.file_upload_threads = 4
sdk.network.metrics.file_upload_starvation_warning_sec = 120
sdk.network.iteration.max_retries_on_server_error = 5
sdk.network.iteration.retry_backoff_factor_sec = 10
sdk.aws.s3.key =
sdk.aws.s3.secret = ****
sdk.aws.s3.region =
sdk.aws.boto3.pool_connections = 512
sdk.aws.boto3.max_multipart_concurrency = 16
sdk.log.null_log_propagate = false
sdk.log.task_log_buffer_capacity = 66
sdk.log.disable_urllib3_info = true
sdk.development.task_reuse_time_window_in_hours = 72.0
sdk.development.vcs_repo_detect_async = true
sdk.development.store_uncommitted_code_diff = true
sdk.development.support_stopping = true
sdk.development.default_output_uri =
sdk.development.force_analyze_entire_repo = false
sdk.development.suppress_update_message = false
sdk.development.detect_with_pip_freeze = false
sdk.development.worker.report_period_sec = 2
sdk.development.worker.ping_period_sec = 30
sdk.development.worker.log_stdout = true
sdk.development.worker.report_global_mem_used = false
DEBUG:urllib3.connectionpool:Resetting dropped connection: ipAddrOfClearMLServer
DEBUG:urllib3.connectionpool:
"GET /v2.5/tasks.started HTTP/1.1" 200 389
Executing task id [a08289ce6f2e47f2afc4e9f4e8540575]:
repository = myGitRepo
branch = stable
version_num = 1d99307ce75a60fc7c53ba9be75f16d07258d11c
tag =
docker_cmd =
entry_point = main_pipeline.py
working_dir = .
DEBUG:clearml_agent.commands.worker:Searching for python3.11
DEBUG:clearml_agent.commands.worker:Found: python3.11
created virtual environment CPython3.11.2.final.0-64 in 442ms
creator CPython3Posix(dest=/home/clearml_user/.clearml/venvs-builds.2/3.11, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/clearml_user/.local/share/virtualenv)
added seed packages: pip==24.1.2, setuptools==70.1.1, wheel==0.43.0
activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
DEBUG:clearml_agent.helper.repo:Running: ['git', 'config', '--global', '--replace-all', 'safe.directory', '*']
Using cached repository in "/home/clearml_user/.clearml/vcs-cache/clearml_showcase.git.01582d138e03d7305278c7f2c2f25dd2/clearml_showcase.git"
pulling git
DEBUG:clearml_agent.helper.repo:Running: ['git', 'fetch', '--all', '--recurse-submodules']
1721735556359 dnsName:cpu:0:service:a08289ce6f2e47f2afc4e9f4e8540575 DEBUG pulling git completed
DEBUG:clearml_agent.helper.repo:Running: ['git', 'checkout', '1d99307ce75a60fc7c53ba9be75f16d07258d11c', '--force']
Hinweis: Wechsle zu '1d99307ce75a60fc7c53ba9be75f16d07258d11c'.
Sie befinden sich im Zustand eines 'losgelösten HEAD'. Sie können sich
umschauen, experimentelle Änderungen vornehmen und diese committen, und
Sie können alle möglichen Commits, die Sie in diesem Zustand machen,
ohne Auswirkungen auf irgendeinen Branch verwerfen, indem Sie zu einem
anderen Branch wechseln.
Wenn Sie einen neuen Branch erstellen möchten, um Ihre erstellten Commits
zu behalten, können Sie das (jetzt oder später) durch Nutzung von
'switch' mit der Option -c tun. Beispiel:
git switch -c <neuer-Branchname>
Oder um diese Operation rückgängig zu machen:
git switch -
Sie können diesen Hinweis ausschalten, indem Sie die Konfigurationsvariable
'advice.detachedHead' auf 'false' setzen.
HEAD ist jetzt bei 1d99307 Update config
DEBUG:clearml_agent.helper.repo:Running: ['git', 'submodule', 'update', '--recursive']
type: git
url: myGitRepo
branch: HEAD
commit: 1d99307ce75a60fc7c53ba9be75f16d07258d11c
root: /home/clearml_user/.clearml/venvs-builds.2/3.11/task_repository/clearml_showcase.git
Applying uncommitted changes
INFO:clearml_agent.helper.repo:applying diff to /home/clearml_user/.clearml/venvs-builds.2/3.11/task_repository/clearml_showcase.git
DEBUG:clearml_agent.helper.repo:Running: ['git', 'apply', '--unidiff-zero']
INFO:clearml_agent.helper.repo:successfully applied uncommitted changes
DEBUG:clearml_agent.helper.package.poetry_api:PoetryConfig: calling initialize, enabled = True
Executing: (PosixPath('python3.11'), '-m', 'pip', '--disable-pip-version-check', 'install', "pip<20.2 ; python_version < '3.10'", "pip<22.3 ; python_version >= '3.10'", '--upgrade')
Looking in indexes:
,
Ignoring pip: markers 'python_version < "3.10"' don't match your environment
Requirement already satisfied: pip<22.3 in /opt/clearml/.venv/lib/python3.11/site-packages (22.2.2)
Executing: (PosixPath('python3.11'), '-m', 'pip', '--disable-pip-version-check', 'list')
1721735561359 dnsName:cpu:0:service:a08289ce6f2e47f2afc4e9f4e8540575 DEBUG Notice: Poetry was found, no specific version required, skipping poetry installation
DEBUG:clearml_agent.helper.package.poetry_api:running: Executing: ('poetry', 'config', '--local', 'virtualenvs.in-project', 'true')
Poetry Enabled: Ignoring requested python packages, using repository poetry lock file!
DEBUG:clearml_agent.helper.package.poetry_api:running: Executing: ('poetry', 'install', '-n')
Creating virtualenv clearml-showcase in /home/clearml_user/.clearml/venvs-builds.2/3.11/task_repository/clearml_showcase.git/.venv
Installing dependencies from lock file
45 Views
0
Answers
4 months ago
4 months ago