Unanswered
I Have A Logical Task That I Want To Split To Multiple Workers. The Task Involves Processing Media Files (Not Training).
The Optimal Design For Me Would Be:
Hi RoughTiger69
Interesting question, maybe something like:
` @PipelineDecorator.component(...)
def process_sub_list(things_to_do=[0,1,2]):
r = []
for i in things_to_do:
print("doing", i)
r.append("done{}".format(i))
return r
@PipelineDecorator.pipeline(...)
def pipeline():
create some stuff to do:
results = []
for step in range(10):
r = process_sub_list(list(range(step*10, (step+1)*10)))
results.append(r)
push into one list with all result, this will actually wait for them to be completed
merged = []
for r in results:
if bool(r):
merged.extend(list(r))
print(max(merged)) `
wdyt?
164 Views
0
Answers
2 years ago
one year ago