Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi All, I Am Faced With The Situation That My Company'S Gitlab Is Temporarily Out Of Service At A Certain Time In The Early Morning (Due To Regular Maintenance Service, Something I Cannot Control). Normally, My System'S Inference Pipelines Are Scheduled T

Hi all, I am faced with the situation that my company's GitLab is temporarily out of service at a certain time in the early morning (due to regular maintenance service, something I cannot control). Normally, my system's inference pipelines are scheduled to launch at that moment, and there may also be some training tasks running at that time interval. When this happens, the ClearML agent can't read the Git repository and raises the following error:
GitLab: Failed to authorize your Git request: internal API unreachable fatal: Could not read from remote repository.Is there any contingency plan for an agent to continue running a task without reading the repository on the GitLab server? At runtime, can I ask the agent to use some cached repository? Have you ever faced a similar situation?

  
  
Posted 2 years ago
Votes Newest

Answers 6


Is there any contingency plan for an agent to continue running a task without reading the repository on the GitLab server?

Not sure what can be done ... any suggestions ?

At runtime, can I ask the agent to use some cached repository?

sometimes you will have it (as the agent stores a cached copy, but I would hardly count on it (and it might be at different states on different machines...)

... (due to regular maintenance service, something I cannot control).

Maybe let "them" know that CI/CD processes actually count on Git being available ? 🤔

  
  
Posted 2 years ago

Mmm well, I can think of a pipeline that could save its state in the instant before the error occurred.

This is already the case, if you clone the pipeline Task change the Args/_continue_pipeline_ to True and enqueue

  
  
Posted 2 years ago

GiganticTurtle0 is there any git redundancy on your network ? maybe you could configure a fallback server ?

  
  
Posted 2 years ago

Thanks! I will try it

  
  
Posted 2 years ago

Mmm well, I can think of a pipeline that could save its state in the instant before the error occurred. So that using some crontab/scheduler the pipeline could be resumed at the point where it was stopped in the case of not having been completed. Is there any functionality like this? Something like PipelineDecorator/PipelineController.resume_from(state_filepath) ?

  
  
Posted 2 years ago

is there any git redundancy on your network ? maybe you could configure a fallback server ?

I will ask this to the IT team

  
  
Posted 2 years ago
617 Views
6 Answers
2 years ago
one year ago
Tags