Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hello, I Had A Setup For Training Ml Model Of Multiple Sub-Modules, Used Pipelines From Task. Didn'T Run It For 4 Months, Was Working Fine Back Then, Now When I Run Pipeline, Steps Are Being Run And Completed Emty, No Console Log, No Stats, Nothing, What

Hello, i had a setup for training ML model of multiple sub-modules, used pipelines from task. Didn't run it for 4 months, was working fine back then, now when i run pipeline, steps are being run and completed emty, no console log, no stats, nothing, what could have happened ?

  
  
Posted 8 months ago
Votes Newest

Answers 14


Expected:
image

  
  
Posted 8 months ago

I installed ClearML Server with tutorial, yes i did it through docker, as i understood i need to look for Elastic logs in /var/log/ecs/ecs-agent.log but i have no such file or even ecs folder in this location on my ClearML Server machine

  
  
Posted 8 months ago

You need to inspect the ES container directly

  
  
Posted 8 months ago

the container logs that is

  
  
Posted 8 months ago

At this moment i somehow made it work, at least with debug sample

  
  
Posted 8 months ago

Can you check if there are any errors in Elastic container when you're running a task?

  
  
Posted 8 months ago

i think its actually a bug anyways, since there are no even task creation log in console, and still it's marked as succesfully completed

  
  
Posted 8 months ago

since i do not use docker and run with pipe.start_locally(run_pipeline_steps_locally=True)
at both local and server, do i still have a some kind of containers? how can i view logs of it?

  
  
Posted 8 months ago

Result:
image

  
  
Posted 8 months ago

image

  
  
Posted 8 months ago

Hi @<1623491856241266688:profile|TenseCrab59> , are you self deployed? Can you provide some logs/screenshots? If you go directly into the task information of each step there console is empty?

  
  
Posted 8 months ago

It worked ok on my local machine but failed on remote, only difference i can think of is huge git diff on which clearml if complaining, can this be the reason?

  
  
Posted 8 months ago

i run all seslf-hosted and without docker, with pipe.start_locally(run_pipeline_steps_locally=True)

  
  
Posted 8 months ago

You mentioned you are self deployed. When you deploy the server, one of containers deployed is the ES container. Did you not deploy the server via docker compose?

  
  
Posted 8 months ago
618 Views
14 Answers
8 months ago
8 months ago
Tags