Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hello, I Had A Setup For Training Ml Model Of Multiple Sub-Modules, Used Pipelines From Task. Didn'T Run It For 4 Months, Was Working Fine Back Then, Now When I Run Pipeline, Steps Are Being Run And Completed Emty, No Console Log, No Stats, Nothing, What

Hello, i had a setup for training ML model of multiple sub-modules, used pipelines from task. Didn't run it for 4 months, was working fine back then, now when i run pipeline, steps are being run and completed emty, no console log, no stats, nothing, what could have happened ?

  
  
Posted 21 days ago
Votes Newest

Answers 14


since i do not use docker and run with pipe.start_locally(run_pipeline_steps_locally=True)
at both local and server, do i still have a some kind of containers? how can i view logs of it?

  
  
Posted 21 days ago

You mentioned you are self deployed. When you deploy the server, one of containers deployed is the ES container. Did you not deploy the server via docker compose?

  
  
Posted 21 days ago

It worked ok on my local machine but failed on remote, only difference i can think of is huge git diff on which clearml if complaining, can this be the reason?

  
  
Posted 21 days ago

You need to inspect the ES container directly

  
  
Posted 21 days ago

Expected:
image

  
  
Posted 21 days ago

Can you check if there are any errors in Elastic container when you're running a task?

  
  
Posted 21 days ago

i think its actually a bug anyways, since there are no even task creation log in console, and still it's marked as succesfully completed

  
  
Posted 21 days ago

Hi @<1623491856241266688:profile|TenseCrab59> , are you self deployed? Can you provide some logs/screenshots? If you go directly into the task information of each step there console is empty?

  
  
Posted 21 days ago

the container logs that is

  
  
Posted 21 days ago

Result:
image

  
  
Posted 21 days ago

I installed ClearML Server with tutorial, yes i did it through docker, as i understood i need to look for Elastic logs in /var/log/ecs/ecs-agent.log but i have no such file or even ecs folder in this location on my ClearML Server machine

  
  
Posted 21 days ago

image

  
  
Posted 21 days ago

At this moment i somehow made it work, at least with debug sample

  
  
Posted 21 days ago

i run all seslf-hosted and without docker, with pipe.start_locally(run_pipeline_steps_locally=True)

  
  
Posted 21 days ago
100 Views
14 Answers
21 days ago
21 days ago
Tags