Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
I Was Giving A Look To The Triggerschedule And I Noticed It

I was giving a look to the TriggerSchedule and I noticed it doesn’t work (at least for me) when I use start_remotely(). The Schedule tasks gets correctly created and it runs , but it never executes any of my triggers (they do work when executed locally with start() instead). I noticed that the logs of the Schedule tasks never sho anything after the set up of the dependencies (when it runs locally it says it sleeps until the next pool instead). Looking at the code I noticed that start_remotely() executes execute_remotely() with exit_process=True, but then it still expects to do more after that, but the process has already exited at that point

  
  
Posted 2 months ago
Votes Newest

Answers 12


This is the log of the remote run (the local one is just a bunch of “Sleeping until the next pool instead 1.5 minutes” plus the logs of my schedule function as expected). The version of the sdk is 1.17.1

  
  
Posted 2 months ago

Hi HollowKangaroo53 , can you please share the log of this run, as well as the clearml SDK version you're using?

  
  
Posted 2 months ago

HollowKangaroo53 , is it a self hosted server?

  
  
Posted 2 months ago

Do I maybe need to do something more that I wasn’t aware to start the trigger schedule on the service queue? Or is it better at this point to just manually run a script on the machine and do “start()” instead of “start_remotely()”?

  
  
Posted 2 months ago

And the services agent? Can you get its log?

  
  
Posted 2 months ago

Assuming you're running it in docker-compose, you should be able to get he docker logs for the apps-agent container

  
  
Posted 2 months ago

HollowKangaroo53 what is the ClearML Server you're using? I don't see the task being run at all, just installing dependencies and exiting.

  
  
Posted 2 months ago

Yes

  
  
Posted 2 months ago

The container is still running and doesn’t show any log entry when I start the trigger scheduler remotely

  
  
Posted 2 months ago

The log of the “clearml-agent-services” only contained this:

  
  
Posted 2 months ago

Exactly, I guess that is exactly the problem probably caused by start_remotely() terminating the process accidentally. The server is 2.0.0-613

  
  
Posted 2 months ago

Where can I find it?

  
  
Posted 2 months ago
186 Views
12 Answers
2 months ago
2 months ago
Tags
Similar posts