Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
I Have The User/Pass Set Up On The

I have the user/pass set up on the clearml.conf and also have the enable_git_ask_pass set to True

  
  
Posted 10 months ago
Votes Newest

Answers 52


What OS are you on?

  
  
Posted 10 months ago

ubuntu 18.04

  
  
Posted 10 months ago

wait.

  
  
Posted 10 months ago

Did you set your password or an access token? Also, please try sticking to a single thread per topic and not multiple messages, it spams the channel

  
  
Posted 10 months ago

yes the ubuntu machine is running the agent, and yes all the changes I make to clearml.conf are on that same ubuntu machine

  
  
Posted 10 months ago

Also please try deleting ~/.clearml folder as well

  
  
Posted 10 months ago

You're sure this is a generated token and not a password?
image

  
  
Posted 10 months ago

noted and updated

  
  
Posted 10 months ago

I generated another one just to show

  
  
Posted 10 months ago

okay I still can not seem to get the git_pass to show up on the logs, when I make other changes to the clearml.conf they do show up on the logs but not the git_pass any ideas?

  
  
Posted 10 months ago

you said you're on ubuntu... please describe exactly where/how the server + agent are set up

  
  
Posted 10 months ago

self-hosted so clearml keys are fine

  
  
Posted 10 months ago

let me know if it changes anything. Of course rerun the agent afterwards

  
  
Posted 10 months ago

it now shows git_pass and git_user as well as enable_git_ask_pass

  
  
Posted 10 months ago

And this is from the same machine running the agent correct?

  
  
Posted 10 months ago

and force_git_ssh_protocol to false, I have also tried turning it to true and enable_git_ask_pass to false but no luck

  
  
Posted 10 months ago

Strange. Can you add your clearml.conf from the agent machine? Please make sure to obscure all secrets šŸ™‚

  
  
Posted 10 months ago

as you can see enable_git_ask_pass is set to true

  
  
Posted 10 months ago

the macbook is the local machine, it has the script that uses execute_remotely

  
  
Posted 10 months ago

Can you add the full log of the run?

  
  
Posted 10 months ago

I’m positive, this is where I got it from
image

  
  
Posted 10 months ago

image

  
  
Posted 10 months ago

And can you manually clone the same exact commit on the machine running the agent?

  
  
Posted 10 months ago

I generated the access token in gitlab and passed it to git_pass

  
  
Posted 10 months ago

and each individual machine in the process

  
  
Posted 10 months ago

This is very strange and unrelated to ClearML:

Error: connect ECONNREFUSED /run/user/1003/vscode-git-7ea59f7d55.sock
    at PipeConnectWrap.afterConnect [as oncomplete] (node:net:1607:16) {
  errno: -111,
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '/run/user/1003/vscode-git-7ea59f7d55.sock'
}
Missing or invalid credentials.
Error: connect ECONNREFUSED /run/user/1003/vscode-git-7ea59f7d55.sock
    at PipeConnectWrap.afterConnect [as oncomplete] (node:net:1607:16) {
  errno: -111,
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '/run/user/1003/vscode-git-7ea59f7d55.sock'
}

How are you running the commands? Regular terminal? Inside VSCode or pycharm?

  
  
Posted 10 months ago

The agent prints its configuration before the execution step, I don't see agent.git_pass set anywhere in the log. Are you sure you set it up on the correct machine? This needs to be set up on the machine running the agent.

  
  
Posted 10 months ago

back then the error was this
image

  
  
Posted 10 months ago

1732722421215 Ajeels-MacBook-Pro-392.local info ClearML Task: created new task id=b045fc0540134b3b860150260fc0d4ac
ClearML results page: 

2024-11-27 15:47:01,439 - clearml.Task - INFO - Waiting for repository detection and full package requirement analysis
1732722423002 Ajeels-MacBook-Pro-392.local info ClearML Monitor: GPU monitoring failed getting GPU reading, switching off GPU monitoring
1732722423507 Ajeels-MacBook-Pro-392.local info 2024-11-27 15:47:03,507 - clearml.Task - INFO - Finished repository detection and package analysis
1732722428918 adamastor:gpu0,1,2,3 INFO task b045fc0540134b3b860150260fc0d4ac pulled from 565fe48e848c4cf39fc5bb749569682b by worker adamastor:gpu0,1,2,3

1732722434061 adamastor:gpu0,1,2,3 DEBUG Process failed, exit code 1
1732722434143 adamastor:gpu0,1,2,3 DEBUG Current configuration (clearml_agent v1.4.0, location: /tmp/.clearml_agent.xye48kjf.cfg):
----------------------
api.version = 1.5
api.verify_certificate = true
api.default_version = 1.5
api.http.max_req_size = 15728640
api.http.retries.total = 240
api.http.retries.connect = 240
api.http.retries.read = 240
api.http.retries.redirect = 240
api.http.retries.status = 240
api.http.retries.backoff_factor = 1.0
api.http.retries.backoff_max = 120.0
api.http.wait_on_maintenance_forever = true
api.http.pool_maxsize = 512
api.http.pool_connections = 512
api.api_server = 

api.web_server = 

api.files_server = 

api.credentials.access_key = DVKF8JECVAEY862A8HUJRG2WTCIIJJ
api.host = 

agent.worker_id = adamastor:gpu0,1,2,3
agent.worker_name = adamastor
agent.force_git_ssh_protocol = true
agent.python_binary = 
agent.package_manager.type = pip
agent.package_manager.pip_version = <20.2
agent.package_manager.system_site_packages = false
agent.package_manager.force_upgrade = false
agent.package_manager.conda_channels.0 = pytorch
agent.package_manager.conda_channels.1 = conda-forge
agent.package_manager.conda_channels.2 = defaults
agent.package_manager.priority_optional_packages.0 = pygobject
agent.package_manager.torch_nightly = false
agent.package_manager.priority_packages.0 = pip
agent.package_manager.priority_packages.1 = cython
agent.package_manager.priority_packages.2 = numpy
agent.package_manager.priority_packages.3 = setuptools
agent.venvs_dir = /home/adamastor/.clearml/venvs-builds
agent.venvs_cache.max_entries = 10
agent.venvs_cache.free_space_threshold_gb = 2.0
agent.vcs_cache.enabled = true
agent.vcs_cache.path = /home/adamastor/.clearml/vcs-cache
agent.venv_update.enabled = false
agent.pip_download_cache.enabled = true
agent.pip_download_cache.path = /home/adamastor/.clearml/pip-download-cache
agent.translate_ssh = true
agent.reload_config = false
agent.docker_pip_cache = /home/adamastor/.clearml/pip-cache
agent.docker_apt_cache = /home/adamastor/.clearml/apt-cache
agent.docker_force_pull = false
agent.default_docker.image = nvidia/cuda:10.2-cudnn7-runtime-ubuntu18.04
agent.enable_task_env = false
agent.hide_docker_command_env_vars.enabled = true
agent.hide_docker_command_env_vars.parse_embedded_urls = true
agent.abort_callback_max_timeout = 1800
agent.docker_internal_mounts.sdk_cache = /clearml_agent_cache
agent.docker_internal_mounts.apt_cache = /var/cache/apt/archives
agent.docker_internal_mounts.ssh_folder = /root/.ssh
agent.docker_internal_mounts.ssh_ro_folder = /.ssh
agent.docker_internal_mounts.pip_cache = /root/.cache/pip
agent.docker_internal_mounts.poetry_cache = /root/.cache/pypoetry
agent.docker_internal_mounts.vcs_cache = /root/.clearml/vcs-cache
agent.docker_internal_mounts.venv_build = /root/.clearml/venvs-builds
agent.docker_internal_mounts.pip_download = /root/.clearml/pip-download-cache
agent.apply_environment = true
agent.apply_files = true
agent.custom_build_script = 
agent.git_user = ajeelahmed
agent.ignore_requested_python_version = true
agent.default_python = 3.7
agent.cuda_version = 122
agent.cudnn_version = 0
sdk.storage.cache.default_base_dir = ~/.clearml/cache
sdk.storage.cache.size.min_free_bytes = 10GB
sdk.storage.direct_access.0.url = file://*
sdk.metrics.file_history_size = 100
sdk.metrics.matplotlib_untitled_history_size = 100
sdk.metrics.images.format = JPEG
sdk.metrics.images.quality = 87
sdk.metrics.images.subsampling = 0
sdk.metrics.tensorboard_single_series_per_graph = false
sdk.network.metrics.file_upload_threads = 4
sdk.network.metrics.file_upload_starvation_warning_sec = 120
sdk.network.iteration.max_retries_on_server_error = 5
sdk.network.iteration.retry_backoff_factor_sec = 10
sdk.aws.s3.key = 
sdk.aws.s3.region = 
sdk.aws.boto3.pool_connections = 512
sdk.aws.boto3.max_multipart_concurrency = 16
sdk.log.null_log_propagate = false
sdk.log.task_log_buffer_capacity = 66
sdk.log.disable_urllib3_info = true
sdk.development.task_reuse_time_window_in_hours = 72.0
sdk.development.vcs_repo_detect_async = true
sdk.development.store_uncommitted_code_diff = true
sdk.development.support_stopping = true
sdk.development.default_output_uri = 
sdk.development.force_analyze_entire_repo = false
sdk.development.suppress_update_message = false
sdk.development.detect_with_pip_freeze = false
sdk.development.worker.report_period_sec = 2
sdk.development.worker.ping_period_sec = 30
sdk.development.worker.log_stdout = true
sdk.development.worker.report_global_mem_used = false

Executing task id [b045fc0540134b3b860150260fc0d4ac]:
repository = git@gitlab.com:planet-farms/periplo-innovation/project-seeds/seed-detection.git
branch = main
version_num = 75974f22658a292d65880b8e141b5244e81a5a1a
tag = 
docker_cmd = 
entry_point = simple.py
working_dir = .

created virtual environment CPython3.7.6.final.0-64 in 195ms
  creator CPython3Posix(dest=/home/adamastor/.clearml/venvs-builds/3.7, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/adamastor/.local/share/virtualenv)
    added seed packages: pip==24.0, setuptools==68.0.0, wheel==0.42.0
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator


Using cached repository in "/home/adamastor/.clearml/vcs-cache/seed-detection.git.582db738cdaa8af9b2f1316f556a91bd/seed-detection.git"
fatal: could not read Username for '
': terminal prompts disabled
error: Could not fetch origin
Repository cloning failed: Command '['git', 'fetch', '--all', '--recurse-submodules']' returned non-zero exit status 1.

clearml_agent: ERROR: Failed cloning repository. 
1) Make sure you pushed the requested commit:
(repository='git@gitlab.com:planet-farms/periplo-innovation/project-seeds/seed-detection.git', branch='main', commit_id='75974f22658a292d65880b8e141b5244e81a5a1a', tag='', docker_cmd=None, entry_point='simple.py', working_dir='.')
2) Check if remote-worker has valid credentials [see worker configuration file]
  
  
Posted 10 months ago

Can you also add a full log of the run that was showing the git pass in the startup print?

  
  
Posted 10 months ago
103K Views
52 Answers
10 months ago
10 months ago
Tags
Similar posts