Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi. I'M Using Clearml Agent 1.16.1 My Code Is Running A Multi-Process Pool With "Spawn" (See

Hi.
I'm using clearml agent 1.16.1

My code is running a multi-process pool with "spawn" (see here for why)

from multiprocessing import get_context
...
    with get_context("spawn").Pool(processes=num_processes) as pool:
        results = pool.starmap(
            my_function,
            [(arguments)...(arguments)]
        )
...

This is all happening in a vertexai node (possibly irrelevant to the issue but thought I'd mention this)

I noticed that the clearml console log shows only the stdout printouts before and after the multi-processed part (where all the interesting stuff actually happens).

If it's any consolation, the vertexai logs do show the printouts from within the processes in the pool (so I know they did their work).
It'd be nice if clearml were able to capture these logs. Apparently using "spawn" in multiprocessing will become the default in future python versions so this issue will be commonplace where multiprocessing is used.

  
  
Posted 4 months ago
Votes Newest

Answers 4


Hi @<1523701713440083968:profile|PanickyMoth78> ! Make sure you are calling Task.init in my_function (this is because the bindings made by clearml will be lost in a spawned process as opposed to a forked one). Also make sure that, in the spawned process, you have CLEARML_PROC_MASTER_ID env var set to the pid of the master process and CLEARML_TASK_ID to the ID task initialized in the master process (this should happen automatically)

  
  
Posted 4 months ago

Oh, cool. So would this then report the activities of the spawned processes to the same task as that of the spawning process?

  
  
Posted 4 months ago

yes

  
  
Posted 4 months ago

We're using a self-hosted clearml server version 1.14.0

  
  
Posted 4 months ago