Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hello, I Had A Setup For Training Ml Model Of Multiple Sub-Modules, Used Pipelines From Task. Didn'T Run It For 4 Months, Was Working Fine Back Then, Now When I Run Pipeline, Steps Are Being Run And Completed Emty, No Console Log, No Stats, Nothing, What

Hello, i had a setup for training ML model of multiple sub-modules, used pipelines from task. Didn't run it for 4 months, was working fine back then, now when i run pipeline, steps are being run and completed emty, no console log, no stats, nothing, what could have happened ?

  
  
Posted one month ago
Votes Newest

Answers 14


Hi @<1623491856241266688:profile|TenseCrab59> , are you self deployed? Can you provide some logs/screenshots? If you go directly into the task information of each step there console is empty?

  
  
Posted one month ago

i run all seslf-hosted and without docker, with pipe.start_locally(run_pipeline_steps_locally=True)

  
  
Posted one month ago

It worked ok on my local machine but failed on remote, only difference i can think of is huge git diff on which clearml if complaining, can this be the reason?

  
  
Posted one month ago

Expected:
image

  
  
Posted one month ago

i think its actually a bug anyways, since there are no even task creation log in console, and still it's marked as succesfully completed

  
  
Posted one month ago

Result:
image

  
  
Posted one month ago

since i do not use docker and run with pipe.start_locally(run_pipeline_steps_locally=True)
at both local and server, do i still have a some kind of containers? how can i view logs of it?

  
  
Posted one month ago

You mentioned you are self deployed. When you deploy the server, one of containers deployed is the ES container. Did you not deploy the server via docker compose?

  
  
Posted one month ago

Can you check if there are any errors in Elastic container when you're running a task?

  
  
Posted one month ago

I installed ClearML Server with tutorial, yes i did it through docker, as i understood i need to look for Elastic logs in /var/log/ecs/ecs-agent.log but i have no such file or even ecs folder in this location on my ClearML Server machine

  
  
Posted one month ago

the container logs that is

  
  
Posted one month ago

image

  
  
Posted one month ago

At this moment i somehow made it work, at least with debug sample

  
  
Posted one month ago

You need to inspect the ES container directly

  
  
Posted one month ago
142 Views
14 Answers
one month ago
one month ago
Tags