it now shows git_pass and git_user as well as enable_git_ask_pass
update: clearml-agent was stuck on version 1.4.0 it has now been upgraded to the latest version 1.9.2
I am noticing that I also don’t see the enable_git_ask_pass in the logs
okay I still can not seem to get the git_pass to show up on the logs, when I make other changes to the clearml.conf they do show up on the logs but not the git_pass any ideas?
Please rerun again and provide a log that does show the git_pass
I do have the virtualenv installed on the same python environment
just to be clear, the second log I sent was awhile ago, anything showing there is largely irrelevant
let me know if it changes anything. Of course rerun the agent afterwards
Try installing virtualenv on the ubuntu machine running the agent
yes the ubuntu machine is running the agent, and yes all the changes I make to clearml.conf are on that same ubuntu machine
and an ubuntu machine is running the agent on the same network i guess and all clearml.conf changes you also make on that machine. correct?
the local machine does not have clearml-agent
the macbook is the local machine, it has the script that uses execute_remotely
Also, make sure to install virtualenv, I see there was a failure in the log on that as well
and each individual machine in the process
you said you're on ubuntu... please describe exactly where/how the server + agent are set up
yeah that’s a great question, I’m running the command in a regular terminal on my macbook, nothing to do with vscode
Also please try deleting ~/.clearml folder as well
This is very strange and unrelated to ClearML:
Error: connect ECONNREFUSED /run/user/1003/vscode-git-7ea59f7d55.sock
at PipeConnectWrap.afterConnect [as oncomplete] (node:net:1607:16) {
errno: -111,
code: 'ECONNREFUSED',
syscall: 'connect',
address: '/run/user/1003/vscode-git-7ea59f7d55.sock'
}
Missing or invalid credentials.
Error: connect ECONNREFUSED /run/user/1003/vscode-git-7ea59f7d55.sock
at PipeConnectWrap.afterConnect [as oncomplete] (node:net:1607:16) {
errno: -111,
code: 'ECONNREFUSED',
syscall: 'connect',
address: '/run/user/1003/vscode-git-7ea59f7d55.sock'
}
How are you running the commands? Regular terminal? Inside VSCode or pycharm?
1732640020078 Ajeels-MacBook-Pro-392.local info ClearML Task: created new task id=85d62ab2b6a74438b587c7b249b33c93
ClearML results page:
2024-11-26 16:53:40,842 - clearml.Task - INFO - Waiting for repository detection and full package requirement analysis
1732640022115 Ajeels-MacBook-Pro-392.local info ClearML Monitor: GPU monitoring failed getting GPU reading, switching off GPU monitoring
1732640022437 Ajeels-MacBook-Pro-392.local info 2024-11-26 16:53:42,437 - clearml.Task - INFO - Finished repository detection and package analysis
1732640026335 adamastor:gpu0,1,2,3 INFO task 85d62ab2b6a74438b587c7b249b33c93 pulled from 565fe48e848c4cf39fc5bb749569682b by worker adamastor:gpu0,1,2,3
1732640031529 adamastor:gpu0,1,2,3 DEBUG Process failed, exit code 1
1732640031669 adamastor:gpu0,1,2,3 DEBUG Current configuration (clearml_agent v1.9.1, location: /tmp/.clearml_agent.1xfib73d.cfg):
----------------------
api.version = 1.5
api.verify_certificate = true
api.default_version = 1.5
api.http.max_req_size = 15728640
api.http.retries.total = 240
api.http.retries.connect = 240
api.http.retries.read = 240
api.http.retries.redirect = 240
api.http.retries.status = 240
api.http.retries.backoff_factor = 1.0
api.http.retries.backoff_max = 120.0
api.http.wait_on_maintenance_forever = true
api.http.pool_maxsize = 512
api.http.pool_connections = 512
api.auth.token_expiration_threshold_sec = ****
api.web_server =
api.api_server =
api.files_server =
api.credentials.access_key = GY9DE1JBN3O6HXILXCXC
api.credentials.secret_key = ****
api.host =
agent.worker_id = adamastor:gpu0,1,2,3
agent.worker_name = adamastor
agent.force_git_ssh_protocol = false
agent.python_binary =
agent.package_manager.type = pip
agent.package_manager.pip_version =
agent.package_manager.system_site_packages = false
agent.package_manager.force_upgrade = false
agent.package_manager.conda_channels.0 = pytorch
agent.package_manager.conda_channels.1 = conda-forge
agent.package_manager.conda_channels.2 = defaults
agent.package_manager.priority_optional_packages.0 = pygobject
agent.package_manager.torch_nightly = false
agent.package_manager.poetry_files_from_repo_working_dir = false
agent.package_manager.force_repo_requirements_txt = true
agent.venvs_dir = /media/adamastor/hdd/.clearml/venvs-builds.1
agent.venvs_cache.max_entries = 3
agent.venvs_cache.free_space_threshold_gb = 50.0
agent.venvs_cache.path = /media/adamastor/hdd/.clearml/venvs-cache
agent.vcs_cache.enabled = true
agent.vcs_cache.path = /media/adamastor/hdd/.clearml/vcs-cache
agent.venv_update.enabled = false
agent.pip_download_cache.enabled = true
agent.pip_download_cache.path = /media/adamastor/hdd/.clearml/pip-download-cache
agent.translate_ssh = true
agent.reload_config = false
agent.docker_pip_cache = /media/adamastor/hdd/.clearml/pip-cache
agent.docker_apt_cache = /media/adamastor/hdd/.clearml/apt-cache.1
agent.docker_force_pull = false
agent.default_docker.image = nvidia/cuda:10.2-cudnn7-runtime-ubuntu18.04
agent.default_docker.match_rules.0.image = python:3.6-bullseye
agent.default_docker.match_rules.0.arguments = --ipc\=host
agent.default_docker.match_rules.0.match.script.binary = python3.6$
agent.default_docker.match_rules.1.image = python:3.7-bullseye
agent.default_docker.match_rules.1.arguments = --ipc\=host
agent.default_docker.match_rules.1.match.script.binary = python3.7$
agent.default_docker.match_rules.2.image = python:3.8-bullseye
agent.default_docker.match_rules.2.arguments = --ipc\=host
agent.default_docker.match_rules.2.match.script.binary = python3.8$
agent.default_docker.match_rules.3.image = python:3.9-bullseye
agent.default_docker.match_rules.3.arguments = --ipc\=host
agent.default_docker.match_rules.3.match.script.binary = python3.9$
agent.default_docker.match_rules.4.image = python:3.10-bullseye
agent.default_docker.match_rules.4.arguments = --ipc\=host
agent.default_docker.match_rules.4.match.script.binary = python3.10$
agent.default_docker.match_rules.5.image = python:3.11-bullseye
agent.default_docker.match_rules.5.arguments = --ipc\=host
agent.default_docker.match_rules.5.match.script.binary = python3.11$
agent.default_docker.match_rules.6.image = python:3.12-bullseye
agent.default_docker.match_rules.6.arguments = --ipc\=host
agent.default_docker.match_rules.6.match.script.binary = python3.12$
agent.enable_task_env = false
agent.sanitize_config_printout = ****
agent.hide_docker_command_env_vars.enabled = true
agent.hide_docker_command_env_vars.parse_embedded_urls = true
agent.abort_callback_max_timeout = 1800
agent.docker_internal_mounts.sdk_cache = /clearml_agent_cache
agent.docker_internal_mounts.apt_cache = /var/cache/apt/archives
agent.docker_internal_mounts.ssh_folder = ~/.ssh
agent.docker_internal_mounts.ssh_ro_folder = /.ssh
agent.docker_internal_mounts.pip_cache = /root/.cache/pip
agent.docker_internal_mounts.poetry_cache = /root/.cache/pypoetry
agent.docker_internal_mounts.vcs_cache = /media/adamastor/hdd/.clearml/vcs-cache
agent.docker_internal_mounts.venvs_cache = /root/.clearml/venvs-cache
agent.docker_internal_mounts.venv_build = /media/adamastor/hdd/.clearml/venvs-builds
agent.docker_internal_mounts.pip_download = /media/adamastor/hdd/.clearml/pip-download-cache
agent.apply_environment = true
agent.apply_files = true
agent.custom_build_script =
agent.disable_task_docker_override = false
agent.git_user = eduardo.guerra1
agent.git_pass = ****
agent.docker_use_activated_venv = true
agent.default_python = 3.8
agent.cuda_version = 122
agent.cudnn_version = 0
sdk.storage.cache.default_base_dir = /media/adamastor/hdd/.clearml/cache
sdk.storage.cache.size.min_free_bytes = 50GB
sdk.storage.cache.size.cleanup_margin_percent = 5%
sdk.storage.direct_access.0.url = file://*
sdk.metrics.file_history_size = 100
sdk.metrics.matplotlib_untitled_history_size = 100
sdk.metrics.images.format = JPEG
sdk.metrics.images.quality = 87
sdk.metrics.images.subsampling = 0
sdk.metrics.tensorboard_single_series_per_graph = false
sdk.network.metrics.file_upload_threads = 4
sdk.network.metrics.file_upload_starvation_warning_sec = 120
sdk.network.iteration.max_retries_on_server_error = 5
sdk.network.iteration.retry_backoff_factor_sec = 10
sdk.aws.s3.key =
sdk.aws.s3.secret = ****
sdk.aws.s3.region =
sdk.aws.boto3.pool_connections = 512
sdk.aws.boto3.max_multipart_concurrency = 16
sdk.log.null_log_propagate = false
sdk.log.task_log_buffer_capacity = 66
sdk.log.disable_urllib3_info = true
sdk.development.task_reuse_time_window_in_hours = 72.0
sdk.development.vcs_repo_detect_async = true
sdk.development.store_uncommitted_code_diff = true
sdk.development.support_stopping = true
sdk.development.default_output_uri =
sdk.development.force_analyze_entire_repo = false
sdk.development.suppress_update_message = false
sdk.development.detect_with_pip_freeze = false
sdk.development.worker.report_period_sec = 2
sdk.development.worker.ping_period_sec = 30
sdk.development.worker.log_stdout = true
sdk.development.worker.report_global_mem_used = false
Executing task id [85d62ab2b6a74438b587c7b249b33c93]:
repository = git@gitlab.com:planet-farms/periplo-innovation/project-seeds/seed-detection.git
branch = main
version_num = 75974f22658a292d65880b8e141b5244e81a5a1a
tag =
docker_cmd =
entry_point = testing.py
working_dir = .
[package_manager.force_repo_requirements_txt=true] Skipping requirements, using repository "requirements.txt"
/home/eguerra/miniconda3/envs/herbs/bin/python3.10: No module named virtualenv
WARNING: virtualenv call failed: Command '['python3.10', '-m', 'virtualenv', '/media/adamastor/hdd/.clearml/venvs-builds.1/3.10']' returned non-zero exit status 1.
INFO: Creating virtual environment with venv
Using cached repository in "/media/adamastor/hdd/.clearml/vcs-cache/seed-detection.git.582db738cdaa8af9b2f1316f556a91bd/seed-detection.git"
pulling git
Using user/pass credentials - replacing ssh url 'git@gitlab.com:planet-farms/periplo-innovation/project-seeds/seed-detection.git' with https url '
'
Missing or invalid credentials.
Error: connect ECONNREFUSED /run/user/1003/vscode-git-7ea59f7d55.sock
at PipeConnectWrap.afterConnect [as oncomplete] (node:net:1607:16) {
errno: -111,
code: 'ECONNREFUSED',
syscall: 'connect',
address: '/run/user/1003/vscode-git-7ea59f7d55.sock'
}
Missing or invalid credentials.
Error: connect ECONNREFUSED /run/user/1003/vscode-git-7ea59f7d55.sock
at PipeConnectWrap.afterConnect [as oncomplete] (node:net:1607:16) {
errno: -111,
code: 'ECONNREFUSED',
syscall: 'connect',
address: '/run/user/1003/vscode-git-7ea59f7d55.sock'
}
remote: HTTP Basic: Access denied. If a password was provided for Git authentication, the password was incorrect or you're required to use a token instead of a password. If a token was provided, it was either incorrect, expired, or improperly scoped. See
fatal: Authentication failed for '
'
error: Could not fetch origin
git pull failed: Command '['git', 'fetch', '--all', '--tags', '--recurse-submodules']' returned non-zero exit status 1.
Repository cloning failed: Command '['git', 'fetch', '--all', '--tags', '--recurse-submodules']' returned non-zero exit status 1.
clearml_agent: ERROR: Failed cloning repository.
1) Make sure you pushed the requested commit:
(repository='git@gitlab.com:planet-farms/periplo-innovation/project-seeds/seed-detection.git', branch='main', commit_id='75974f22658a292d65880b8e141b5244e81a5a1a', tag='', docker_cmd=None, entry_point='testing.py', working_dir='.')
2) Check if remote-worker has valid credentials [see worker configuration file]
Can you also add a full log of the run that was showing the git pass in the startup print?
and force_git_ssh_protocol to false, I have also tried turning it to true and enable_git_ask_pass to false but no luck
as you can see enable_git_ask_pass is set to true
self-hosted so clearml keys are fine