Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hey! I Would Like To Connect To Same Task From Multiple Consumer And Upload Debug Image. Is It Possibile? It Seems Like I Can Connect To The Task. Get The Logger But Nothing Is Uploaded.

Hey!
I would like to connect to same task from multiple consumer and upload debug image.
Is it possibile?
It seems like i can connect to the task. get the logger but nothing is uploaded.
AgitatedDove14 please help 🙂

  
  
Posted 3 years ago
Votes Newest

Answers 30


args = parser.parse_args() os.environ['TRAINS_PROC_MASTER_ID'] = f"1:{args.trains_id}" os.environ['OMPI_COMM_WORLD_NODE_RANK'] = str(randint(0, sys.maxsize)) trains_task = trains.Task.init(project_name=f'{ALG_NAME}-inference', task_name='inference')failes on init

  
  
Posted 3 years ago

Notice that you have to Have the task already started by the Master process

  
  
Posted 3 years ago

ValueError: Task object can only be updated if created or in_progress

  
  
Posted 3 years ago

It has to be alive so all the "child nodes" could report to it....

  
  
Posted 3 years ago

it started and then ended

  
  
Posted 3 years ago

failed on init

  
  
Posted 3 years ago

should OMPI_COMM_WORLD_NODE_RANK be number or can be some guid?

  
  
Posted 3 years ago

each consumer in different container

  
  
Posted 3 years ago

integer

  
  
Posted 3 years ago

yes...

  
  
Posted 3 years ago

Ok, so it is master node and not consumer producer pattern..

  
  
Posted 3 years ago

so if i plot image with matplot lib..it would not upload? i need use the logger.

  
  
Posted 3 years ago

Correct, and that also means the code the runs is not auto-magically logged.

  
  
Posted 3 years ago

so if i load task and not init it..so it is not the main one?

  
  
Posted 3 years ago

so how can i make it run with the "auto magic"

  
  
Posted 3 years ago

i do not have another task

  
  
Posted 3 years ago

so if i plot image with matplot lib..it would not upload? i need use the logger.

Correct, if you have no "main" task , no automagic 😞

so how can i make it run with the "auto magic"

Automagic logs a single instance... unless those are subprocesses, in which case, the main task takes care of "copying" itself to the subprocess.

Again what is the use case for multiple machines?

  
  
Posted 3 years ago

os.environ['TRAINS_PROC_MASTER_ID'] = args.trains_id os.environ['OMPI_COMM_WORLD_NODE_RANK'] = str(randint(0, sys.maxsize)) trains_task = trains.Task.init(project_name=f'{ALG_NAME}-inference', task_name='inference') print(type(trains_task))<class 'trains.task.Task.init.<locals>._TaskStub'>

  
  
Posted 3 years ago

os.environ['TRAINS_PROC_MASTER_ID'] = '1:da0606f2e6fb40f692f5c885f807902a' os.environ['OMPI_COMM_WORLD_NODE_RANK'] = '1' task = Task.init(project_name="examples", task_name="Manual reporting") print(type(task))Should be: <class 'trains.task.Task'>

  
  
Posted 3 years ago

os.environ['TRAINS_PROC_MASTER_ID'] = args.trains_idit should be '1:'+args.trains_id

os.environ['TRAINS_PROC_MASTER_ID'] = '1:{}'.format(args.trains_id)Also str(randint(1, sys.maxsize))

  
  
Posted 3 years ago

yes you are correct, OS environment:
TRAINS_PROC_MASTER_ID=1:task_id_here

  
  
Posted 3 years ago

how can I pass to task_init the task_id? should it also be in some env?

  
  
Posted 3 years ago

hhhhmm..then i can not get master task params

  
  
Posted 3 years ago

i get subtask

  
  
Posted 3 years ago

FranticCormorant35 As far as I understand what you have going is a multi-node setup, that you manage yourself. Something like Horovod Torch distributed or any MPI setup. Since Trains support all of the above standard multi-node. The easiest way is to do the following:

On the master Node set OS environment:
OMPI_COMM_WORLD_NODE_RANK=0Then on any client node:
OMPI_COMM_WORLD_NODE_RANK=unique_client_node_numberIn all processes you can Call Task.init - with all the automagic kicking in. The Master node will be the only one registering the execution section of the experiment (i.e. git arg parser etc.) while all the rest will be logged as usual (console output, tensorboard matplotlib etc.)

How does that sound?

  
  
Posted 3 years ago

Should work out of the box, as long as the task was started. You can forcefully start the task with:
task.mark_started()

  
  
Posted 3 years ago

trains_task.started()

  
  
Posted 3 years ago

is it the same?

  
  
Posted 3 years ago

Logger.current_logger()Will return the logger for the "main" Task.
The "Main" task is the task of this process, a singleton for the process.
All other instances create Task object. you can have multiple Task objects and log different things to them, but you can only have a single "main" Task (the one created with Task.init).
All the auto-magic stuff is logged automatically to the "main" task.
Make sense ?

  
  
Posted 3 years ago

is it the same?

  
  
Posted 3 years ago
117 Views
30 Answers
3 years ago
4 months ago
Tags