Yes

This is from the console by the way

Hmm okay, I'm doing a hyper parameter search by launching multiple processes of my train function. I've got a main task runing the search to log the final results, and a bunch of training tasks running in parallel. It would've been nice to be able to come back to each one individual training task, but I guess I'll do without

From what I could see, generating SHA2:
i7-10700K: ~ 10 - 15 minutes Xeon E3-1240: 4 - 5 hours!Then in both cases I still have about an 1h30 to upload the images to the fileserver. Which I also find quite a bit slow, but the ClearML fileserver is on my old Xeon. I plan to upgrade my server and to test it again

Yeah I had the same issue: https://clearml.slack.com/archives/CTK20V944/p1664887550256279

Yeah, I understand that it's a bit confusing what I'm asking. Here's a sample code:
` from clearml.automation.controller import PipelineDecorator

@PipelineDecorator.component(cache=True)
def step_one():
import numpy as np

image = np.ones((100, 100, 3))
crop = image[0:50, 0:50]
print("here's my crop of shape:", crop.shape)

@PipelineDecorator.pipeline(name='custom pipeline logic', project='examples', version='0.0.5')
def executing_pipeline():
step_one()

if name ==...

Thank you very much!

Thanks for trying to help me out! Here's some code that should reproduce the error (at least, it did for me): https://github.com/allegroai/clearml-agent/issues/111

With default settings, to upload 2 datasets of 120 GB and 70 Gb it took more than 6 hours! And this is to upload the dataset on the server itself, the upload pipeline is done on the same computer as clearml

Sure, I will do that

I can make an issue in GitHub if that helps keep track of it

So this seems like it could work as a work-around:
` Python 3.10.6 (main, Aug 10 2022, 11:40:04) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

import numpy as np
a = np.ones((100, 100, 3))
a.take(range(40), 0).take(range(40), 1).shape
(40, 40, 3)

replaces a[0:40, 0:40] `

This crashes with:
File "/tmp/tmpa5l_cvuv.py", line 8 crop = image[(0:50, 0:50)] ^ SyntaxError: invalid syntax

WebApp: 1.7.0-232 • Server: 1.7.0-232 • API: 2.21

Yes sure, I will do that

Yes sure CostlyOstrich36 , I'm just trying to pass some arguments from my __main__ to my pipeline_entry() to my component get_best_model() . But for some reason, I'm getting None into get_best_model instead of what I've given it in pipeline_entry

1.7.1

Weeell it seems to work with version 1.7.0 and not with 1.7.1

From what I've seen, another fix would be to pass a default value to your parameters, for example: def executing_pipeline(start_date=0, end_date=0):

I'm using a self hosted server

Thanks! Seems to be working just fine

Here are the versions: WebApp: 1.7.0-232 • Server: 1.7.0-232 • API: 2.21

I had the same issue, they fixed it in 1.7.2rc1 I think. You can try pip install clearml==1.7.2rc1

I wasn't able to reproduce it with a simple piece of code, I'll try to see later if I can. But what I've seen is that I was logging too many images and it was somehow missing my last reports. With fewer image logs it seems like it's working normally

But this works strangely:
` @PipelineDecorator.component(cache=False, execution_queue="default")
def get_param():
return 'hello'

@PipelineDecorator.component(cache=False, execution_queue="default")
def get_best_model(task_ids):
import ...

print('task_ids:', task_ids, type(task_ids)) # task_ids: None <class 'NoneType'>

...

@PipelineDecorator.pipeline(
name='...',
project='...',
version='0.1'
)
def pipeline_entry(task_ids: List[str], ...):
print(task_ids, typ...

Yep I'm dumb, it worked. However I've launch a couple of tasks with name ='custom pipeline logic', project ='examples' and I have to delete them manually. When I try through the UI it just waits forever

CostlyOstrich36 This looks like a bug? Here's a simpler version of it and what I'm getting:
` from clearml.automation.controller import PipelineDecorator

@PipelineDecorator.component(cache=False)
def step_one(my_arg):
print('step_one/my_arg:', my_arg) # step_one/my_arg: None
# I should not get None here! At least that's what I'm expecting

@PipelineDecorator.pipeline(name='custom pipeline logic', project='examples', version='0.0.5')
def executing_pipeline(my_arg):
print('my_ar...

This however works fine:
` Python 3.10.6 (main, Aug 10 2022, 11:40:04) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

import numpy as np
a = np.ones((100, 100, 3))
a[0:50, 0:50].shape
(50, 50, 3)
`

Apparently, the issue is that it will always pass None if there's no default value

Sooo for the SHA2 generation, I've tested 2 very different CPUs, and it makes a HUGE difference 😅 I probably have to upgrade my server