Unanswered
Hi,
I’M Trying Out Clearml Pipelines From Decorators, And I’M Encountering A Few Problems I Don’T Know How To Solve.
btw, when running pipelines, a common pattern is that two consequential components end up on the same node. The current implementation will upload the result of the first component, and then the first thing the next component will do is download it. I assume get_local_copy
method is used and the output is stored to local cache. Wouldn’t it be more performant for the first component to store its result to the local cache along uploading it to file server? In that way, the next component if run on the same node wouldn’t need to download it from the file server.
169 Views
0
Answers
one year ago
one year ago