I do have the virtualenv installed on the same python environment
1732722421215 Ajeels-MacBook-Pro-392.local info ClearML Task: created new task id=b045fc0540134b3b860150260fc0d4ac
ClearML results page:
2024-11-27 15:47:01,439 - clearml.Task - INFO - Waiting for repository detection and full package requirement analysis
1732722423002 Ajeels-MacBook-Pro-392.local info ClearML Monitor: GPU monitoring failed getting GPU reading, switching off GPU monitoring
1732722423507 Ajeels-MacBook-Pro-392.local info 2024-11-27 15:47:03,507 - clearml.Task - INFO - Finished repository detection and package analysis
1732722428918 adamastor:gpu0,1,2,3 INFO task b045fc0540134b3b860150260fc0d4ac pulled from 565fe48e848c4cf39fc5bb749569682b by worker adamastor:gpu0,1,2,3
1732722434061 adamastor:gpu0,1,2,3 DEBUG Process failed, exit code 1
1732722434143 adamastor:gpu0,1,2,3 DEBUG Current configuration (clearml_agent v1.4.0, location: /tmp/.clearml_agent.xye48kjf.cfg):
----------------------
api.version = 1.5
api.verify_certificate = true
api.default_version = 1.5
api.http.max_req_size = 15728640
api.http.retries.total = 240
api.http.retries.connect = 240
api.http.retries.read = 240
api.http.retries.redirect = 240
api.http.retries.status = 240
api.http.retries.backoff_factor = 1.0
api.http.retries.backoff_max = 120.0
api.http.wait_on_maintenance_forever = true
api.http.pool_maxsize = 512
api.http.pool_connections = 512
api.api_server =
api.web_server =
api.files_server =
api.credentials.access_key = DVKF8JECVAEY862A8HUJRG2WTCIIJJ
api.host =
agent.worker_id = adamastor:gpu0,1,2,3
agent.worker_name = adamastor
agent.force_git_ssh_protocol = true
agent.python_binary =
agent.package_manager.type = pip
agent.package_manager.pip_version = <20.2
agent.package_manager.system_site_packages = false
agent.package_manager.force_upgrade = false
agent.package_manager.conda_channels.0 = pytorch
agent.package_manager.conda_channels.1 = conda-forge
agent.package_manager.conda_channels.2 = defaults
agent.package_manager.priority_optional_packages.0 = pygobject
agent.package_manager.torch_nightly = false
agent.package_manager.priority_packages.0 = pip
agent.package_manager.priority_packages.1 = cython
agent.package_manager.priority_packages.2 = numpy
agent.package_manager.priority_packages.3 = setuptools
agent.venvs_dir = /home/adamastor/.clearml/venvs-builds
agent.venvs_cache.max_entries = 10
agent.venvs_cache.free_space_threshold_gb = 2.0
agent.vcs_cache.enabled = true
agent.vcs_cache.path = /home/adamastor/.clearml/vcs-cache
agent.venv_update.enabled = false
agent.pip_download_cache.enabled = true
agent.pip_download_cache.path = /home/adamastor/.clearml/pip-download-cache
agent.translate_ssh = true
agent.reload_config = false
agent.docker_pip_cache = /home/adamastor/.clearml/pip-cache
agent.docker_apt_cache = /home/adamastor/.clearml/apt-cache
agent.docker_force_pull = false
agent.default_docker.image = nvidia/cuda:10.2-cudnn7-runtime-ubuntu18.04
agent.enable_task_env = false
agent.hide_docker_command_env_vars.enabled = true
agent.hide_docker_command_env_vars.parse_embedded_urls = true
agent.abort_callback_max_timeout = 1800
agent.docker_internal_mounts.sdk_cache = /clearml_agent_cache
agent.docker_internal_mounts.apt_cache = /var/cache/apt/archives
agent.docker_internal_mounts.ssh_folder = /root/.ssh
agent.docker_internal_mounts.ssh_ro_folder = /.ssh
agent.docker_internal_mounts.pip_cache = /root/.cache/pip
agent.docker_internal_mounts.poetry_cache = /root/.cache/pypoetry
agent.docker_internal_mounts.vcs_cache = /root/.clearml/vcs-cache
agent.docker_internal_mounts.venv_build = /root/.clearml/venvs-builds
agent.docker_internal_mounts.pip_download = /root/.clearml/pip-download-cache
agent.apply_environment = true
agent.apply_files = true
agent.custom_build_script =
agent.git_user = ajeelahmed
agent.ignore_requested_python_version = true
agent.default_python = 3.7
agent.cuda_version = 122
agent.cudnn_version = 0
sdk.storage.cache.default_base_dir = ~/.clearml/cache
sdk.storage.cache.size.min_free_bytes = 10GB
sdk.storage.direct_access.0.url = file://*
sdk.metrics.file_history_size = 100
sdk.metrics.matplotlib_untitled_history_size = 100
sdk.metrics.images.format = JPEG
sdk.metrics.images.quality = 87
sdk.metrics.images.subsampling = 0
sdk.metrics.tensorboard_single_series_per_graph = false
sdk.network.metrics.file_upload_threads = 4
sdk.network.metrics.file_upload_starvation_warning_sec = 120
sdk.network.iteration.max_retries_on_server_error = 5
sdk.network.iteration.retry_backoff_factor_sec = 10
sdk.aws.s3.key =
sdk.aws.s3.region =
sdk.aws.boto3.pool_connections = 512
sdk.aws.boto3.max_multipart_concurrency = 16
sdk.log.null_log_propagate = false
sdk.log.task_log_buffer_capacity = 66
sdk.log.disable_urllib3_info = true
sdk.development.task_reuse_time_window_in_hours = 72.0
sdk.development.vcs_repo_detect_async = true
sdk.development.store_uncommitted_code_diff = true
sdk.development.support_stopping = true
sdk.development.default_output_uri =
sdk.development.force_analyze_entire_repo = false
sdk.development.suppress_update_message = false
sdk.development.detect_with_pip_freeze = false
sdk.development.worker.report_period_sec = 2
sdk.development.worker.ping_period_sec = 30
sdk.development.worker.log_stdout = true
sdk.development.worker.report_global_mem_used = false
Executing task id [b045fc0540134b3b860150260fc0d4ac]:
repository = git@gitlab.com:planet-farms/periplo-innovation/project-seeds/seed-detection.git
branch = main
version_num = 75974f22658a292d65880b8e141b5244e81a5a1a
tag =
docker_cmd =
entry_point = simple.py
working_dir = .
created virtual environment CPython3.7.6.final.0-64 in 195ms
creator CPython3Posix(dest=/home/adamastor/.clearml/venvs-builds/3.7, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/adamastor/.local/share/virtualenv)
added seed packages: pip==24.0, setuptools==68.0.0, wheel==0.42.0
activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
Using cached repository in "/home/adamastor/.clearml/vcs-cache/seed-detection.git.582db738cdaa8af9b2f1316f556a91bd/seed-detection.git"
fatal: could not read Username for '
': terminal prompts disabled
error: Could not fetch origin
Repository cloning failed: Command '['git', 'fetch', '--all', '--recurse-submodules']' returned non-zero exit status 1.
clearml_agent: ERROR: Failed cloning repository.
1) Make sure you pushed the requested commit:
(repository='git@gitlab.com:planet-farms/periplo-innovation/project-seeds/seed-detection.git', branch='main', commit_id='75974f22658a292d65880b8e141b5244e81a5a1a', tag='', docker_cmd=None, entry_point='simple.py', working_dir='.')
2) Check if remote-worker has valid credentials [see worker configuration file]
the local machine does not have clearml-agent
Did you set your password or an access token? Also, please try sticking to a single thread per topic and not multiple messages, it spams the channel
oops sorry! And yes I have set up the access token from gitlab and that’s what I’m passing in git_pass
Try installing virtualenv on the ubuntu machine running the agent
they all seem to have the ‘glpat’ prefix before them
you are right, I can’t see the git_pass but in the previous logs I see it
Please rerun again and provide a log that does show the git_pass
okay I still can not seem to get the git_pass to show up on the logs, when I make other changes to the clearml.conf they do show up on the logs but not the git_pass any ideas?
let me know if it changes anything. Of course rerun the agent afterwards
and an ubuntu machine is running the agent on the same network i guess and all clearml.conf changes you also make on that machine. correct?
yes, the remote machine is running the clearml-agent, this file resides in that same machine, same account
update: clearml-agent was stuck on version 1.4.0 it has now been upgraded to the latest version 1.9.2
yes the ubuntu machine is running the agent, and yes all the changes I make to clearml.conf are on that same ubuntu machine
This is very strange and unrelated to ClearML:
Error: connect ECONNREFUSED /run/user/1003/vscode-git-7ea59f7d55.sock
at PipeConnectWrap.afterConnect [as oncomplete] (node:net:1607:16) {
errno: -111,
code: 'ECONNREFUSED',
syscall: 'connect',
address: '/run/user/1003/vscode-git-7ea59f7d55.sock'
}
Missing or invalid credentials.
Error: connect ECONNREFUSED /run/user/1003/vscode-git-7ea59f7d55.sock
at PipeConnectWrap.afterConnect [as oncomplete] (node:net:1607:16) {
errno: -111,
code: 'ECONNREFUSED',
syscall: 'connect',
address: '/run/user/1003/vscode-git-7ea59f7d55.sock'
}
How are you running the commands? Regular terminal? Inside VSCode or pycharm?
Also, you need to restart the agent between changes in the config
and you can see it includes the commit that the log mentions
Strange. Can you add your clearml.conf from the agent machine? Please make sure to obscure all secrets 🙂
well, I’m not sure how I’d go about cloning just one specific commit but I have cloned the repository
Also please try deleting ~/.clearml folder as well
and force_git_ssh_protocol to false, I have also tried turning it to true and enable_git_ask_pass to false but no luck
the macbook is the local machine, it has the script that uses execute_remotely