I generated the access token in gitlab and passed it to git_pass
You're sure this is a generated token and not a password?
I do have the virtualenv installed on the same python environment
just to be clear, the second log I sent was awhile ago, anything showing there is largely irrelevant
1732722421215 Ajeels-MacBook-Pro-392.local info ClearML Task: created new task id=b045fc0540134b3b860150260fc0d4ac
ClearML results page:
2024-11-27 15:47:01,439 - clearml.Task - INFO - Waiting for repository detection and full package requirement analysis
1732722423002 Ajeels-MacBook-Pro-392.local info ClearML Monitor: GPU monitoring failed getting GPU reading, switching off GPU monitoring
1732722423507 Ajeels-MacBook-Pro-392.local info 2024-11-27 15:47:03,507 - clearml.Task - INFO - Finished repository detection and package analysis
1732722428918 adamastor:gpu0,1,2,3 INFO task b045fc0540134b3b860150260fc0d4ac pulled from 565fe48e848c4cf39fc5bb749569682b by worker adamastor:gpu0,1,2,3
1732722434061 adamastor:gpu0,1,2,3 DEBUG Process failed, exit code 1
1732722434143 adamastor:gpu0,1,2,3 DEBUG Current configuration (clearml_agent v1.4.0, location: /tmp/.clearml_agent.xye48kjf.cfg):
----------------------
api.version = 1.5
api.verify_certificate = true
api.default_version = 1.5
api.http.max_req_size = 15728640
api.http.retries.total = 240
api.http.retries.connect = 240
api.http.retries.read = 240
api.http.retries.redirect = 240
api.http.retries.status = 240
api.http.retries.backoff_factor = 1.0
api.http.retries.backoff_max = 120.0
api.http.wait_on_maintenance_forever = true
api.http.pool_maxsize = 512
api.http.pool_connections = 512
api.api_server =
api.web_server =
api.files_server =
api.credentials.access_key = DVKF8JECVAEY862A8HUJRG2WTCIIJJ
api.host =
agent.worker_id = adamastor:gpu0,1,2,3
agent.worker_name = adamastor
agent.force_git_ssh_protocol = true
agent.python_binary =
agent.package_manager.type = pip
agent.package_manager.pip_version = <20.2
agent.package_manager.system_site_packages = false
agent.package_manager.force_upgrade = false
agent.package_manager.conda_channels.0 = pytorch
agent.package_manager.conda_channels.1 = conda-forge
agent.package_manager.conda_channels.2 = defaults
agent.package_manager.priority_optional_packages.0 = pygobject
agent.package_manager.torch_nightly = false
agent.package_manager.priority_packages.0 = pip
agent.package_manager.priority_packages.1 = cython
agent.package_manager.priority_packages.2 = numpy
agent.package_manager.priority_packages.3 = setuptools
agent.venvs_dir = /home/adamastor/.clearml/venvs-builds
agent.venvs_cache.max_entries = 10
agent.venvs_cache.free_space_threshold_gb = 2.0
agent.vcs_cache.enabled = true
agent.vcs_cache.path = /home/adamastor/.clearml/vcs-cache
agent.venv_update.enabled = false
agent.pip_download_cache.enabled = true
agent.pip_download_cache.path = /home/adamastor/.clearml/pip-download-cache
agent.translate_ssh = true
agent.reload_config = false
agent.docker_pip_cache = /home/adamastor/.clearml/pip-cache
agent.docker_apt_cache = /home/adamastor/.clearml/apt-cache
agent.docker_force_pull = false
agent.default_docker.image = nvidia/cuda:10.2-cudnn7-runtime-ubuntu18.04
agent.enable_task_env = false
agent.hide_docker_command_env_vars.enabled = true
agent.hide_docker_command_env_vars.parse_embedded_urls = true
agent.abort_callback_max_timeout = 1800
agent.docker_internal_mounts.sdk_cache = /clearml_agent_cache
agent.docker_internal_mounts.apt_cache = /var/cache/apt/archives
agent.docker_internal_mounts.ssh_folder = /root/.ssh
agent.docker_internal_mounts.ssh_ro_folder = /.ssh
agent.docker_internal_mounts.pip_cache = /root/.cache/pip
agent.docker_internal_mounts.poetry_cache = /root/.cache/pypoetry
agent.docker_internal_mounts.vcs_cache = /root/.clearml/vcs-cache
agent.docker_internal_mounts.venv_build = /root/.clearml/venvs-builds
agent.docker_internal_mounts.pip_download = /root/.clearml/pip-download-cache
agent.apply_environment = true
agent.apply_files = true
agent.custom_build_script =
agent.git_user = ajeelahmed
agent.ignore_requested_python_version = true
agent.default_python = 3.7
agent.cuda_version = 122
agent.cudnn_version = 0
sdk.storage.cache.default_base_dir = ~/.clearml/cache
sdk.storage.cache.size.min_free_bytes = 10GB
sdk.storage.direct_access.0.url = file://*
sdk.metrics.file_history_size = 100
sdk.metrics.matplotlib_untitled_history_size = 100
sdk.metrics.images.format = JPEG
sdk.metrics.images.quality = 87
sdk.metrics.images.subsampling = 0
sdk.metrics.tensorboard_single_series_per_graph = false
sdk.network.metrics.file_upload_threads = 4
sdk.network.metrics.file_upload_starvation_warning_sec = 120
sdk.network.iteration.max_retries_on_server_error = 5
sdk.network.iteration.retry_backoff_factor_sec = 10
sdk.aws.s3.key =
sdk.aws.s3.region =
sdk.aws.boto3.pool_connections = 512
sdk.aws.boto3.max_multipart_concurrency = 16
sdk.log.null_log_propagate = false
sdk.log.task_log_buffer_capacity = 66
sdk.log.disable_urllib3_info = true
sdk.development.task_reuse_time_window_in_hours = 72.0
sdk.development.vcs_repo_detect_async = true
sdk.development.store_uncommitted_code_diff = true
sdk.development.support_stopping = true
sdk.development.default_output_uri =
sdk.development.force_analyze_entire_repo = false
sdk.development.suppress_update_message = false
sdk.development.detect_with_pip_freeze = false
sdk.development.worker.report_period_sec = 2
sdk.development.worker.ping_period_sec = 30
sdk.development.worker.log_stdout = true
sdk.development.worker.report_global_mem_used = false
Executing task id [b045fc0540134b3b860150260fc0d4ac]:
repository = git@gitlab.com:planet-farms/periplo-innovation/project-seeds/seed-detection.git
branch = main
version_num = 75974f22658a292d65880b8e141b5244e81a5a1a
tag =
docker_cmd =
entry_point = simple.py
working_dir = .
created virtual environment CPython3.7.6.final.0-64 in 195ms
creator CPython3Posix(dest=/home/adamastor/.clearml/venvs-builds/3.7, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/adamastor/.local/share/virtualenv)
added seed packages: pip==24.0, setuptools==68.0.0, wheel==0.42.0
activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
Using cached repository in "/home/adamastor/.clearml/vcs-cache/seed-detection.git.582db738cdaa8af9b2f1316f556a91bd/seed-detection.git"
fatal: could not read Username for '
': terminal prompts disabled
error: Could not fetch origin
Repository cloning failed: Command '['git', 'fetch', '--all', '--recurse-submodules']' returned non-zero exit status 1.
clearml_agent: ERROR: Failed cloning repository.
1) Make sure you pushed the requested commit:
(repository='git@gitlab.com:planet-farms/periplo-innovation/project-seeds/seed-detection.git', branch='main', commit_id='75974f22658a292d65880b8e141b5244e81a5a1a', tag='', docker_cmd=None, entry_point='simple.py', working_dir='.')
2) Check if remote-worker has valid credentials [see worker configuration file]
Also, make sure to install virtualenv, I see there was a failure in the log on that as well
as you can see enable_git_ask_pass is set to true
the local machine does not have clearml-agent
it now shows git_pass and git_user as well as enable_git_ask_pass
Did you set your password or an access token? Also, please try sticking to a single thread per topic and not multiple messages, it spams the channel
oops sorry! And yes I have set up the access token from gitlab and that’s what I’m passing in git_pass
Try installing virtualenv on the ubuntu machine running the agent
they all seem to have the ‘glpat’ prefix before them
you are right, I can’t see the git_pass but in the previous logs I see it
Please rerun again and provide a log that does show the git_pass
okay I still can not seem to get the git_pass to show up on the logs, when I make other changes to the clearml.conf they do show up on the logs but not the git_pass any ideas?
let me know if it changes anything. Of course rerun the agent afterwards
and an ubuntu machine is running the agent on the same network i guess and all clearml.conf changes you also make on that machine. correct?
yes, the remote machine is running the clearml-agent, this file resides in that same machine, same account
update: clearml-agent was stuck on version 1.4.0 it has now been upgraded to the latest version 1.9.2
yes the ubuntu machine is running the agent, and yes all the changes I make to clearml.conf are on that same ubuntu machine
I’m positive, this is where I got it from
And can you manually clone the same exact commit on the machine running the agent?
This is very strange and unrelated to ClearML:
Error: connect ECONNREFUSED /run/user/1003/vscode-git-7ea59f7d55.sock
at PipeConnectWrap.afterConnect [as oncomplete] (node:net:1607:16) {
errno: -111,
code: 'ECONNREFUSED',
syscall: 'connect',
address: '/run/user/1003/vscode-git-7ea59f7d55.sock'
}
Missing or invalid credentials.
Error: connect ECONNREFUSED /run/user/1003/vscode-git-7ea59f7d55.sock
at PipeConnectWrap.afterConnect [as oncomplete] (node:net:1607:16) {
errno: -111,
code: 'ECONNREFUSED',
syscall: 'connect',
address: '/run/user/1003/vscode-git-7ea59f7d55.sock'
}
How are you running the commands? Regular terminal? Inside VSCode or pycharm?