No that's the right one

It doesnt seem so if you look at the REST api documentation, might be available as an ENterprise plan feature

Yup we too had to implement a lots of little things for ClearML in our tooling library due to it being pretty bare bone in some area

Note that you might need to log out login or reboot the machine for the change to take effect

Component's prototype seems fine:
@PipelineDecorator.component( return_values=['dataset_id'], cache=False, task_type=TaskTypes.data_processing, execution_queue='Quad_VCPU_16GB', ) def generate_dataset(start_date: str, end_date: str, input_aws_credentials_profile: str = 'default'):

Sure but the same pattern can be achieved using explicitly the PipelineController class and defining steps using .add_step() pointing to CLearML's Task objects right ?

The decorators simply abstract away the controller but both methods (decorators or controller/tasks) allows to decouple your pipelines in steps each having an independent compute target, right ?

So basically choosing one method or the other only a question of best-practice or style ?

upsy sorry didnt read the entire backtrace

It's funny cause the line in the backtrace is the correct one so I don't think it has something to do with strange cachine behavior

Most of the time is took by building wheels for nympy and pandas which are apparently deps of clearml-agent if I read the log correctly

No I was was pointing out the lack of one, but turns out on some model the iteration is so slow even on GPUs when training on a lots of time serie that you have to set the pytorch lightning trainer argument log_every_n_steps to 1 (default 50 ) to prevent the ClearML iteration logger from timing-out

Ohwow, okay Ill test it with another type

And running with a Python 3.10 interpreter

CostlyOstrich36 Having the same issue running on a remote worker, even tho the line works correctly on python interpreter and the component run correctly in local debug mode (but not standard local mode):
` File "/root/.clearml/venvs-builds/3.10/code/generate_dataset.py", line 18, in generate_dataset
time_range = pd.date_range(start=start_date, end=end_date, freq='D').to_pydatetime().tolist()
File "/root/.clearml/venvs-builds/3.10/lib/python3.10/site-packages/pandas/core/indexes/date...

@<1523701205467926528:profile|AgitatedDove14> Yup I tested to no avail, a bit sad that there is no working integration with one of the leading time series framework...

And after your modifications are made you can use . https://clear.ml/docs/latest/docs/references/sdk/dataset/#datasetsquash to squash your modified subset with the main dataset if you want to re-integrate it in your flow. But I don't remember if squash requires the both datasets to be present locally or not...

Normally it's an environment variable set to the current user, you can check by typing echo $USER in the terminal

As env variables

This is funny cause the auto-scaler on GPU instances is working fine, but as the backtrace suggests it seems to be linked to this instance family

BoredHedgehog47 here, enlight yourself https://github.com/allegroai/clearml/releases

I can test it empirically but I want to be sure what is the expected behavior so my pipeline don't get auto-magically broken after a patch

Hey Mathias,

The project SDK is pretty barebone and according to the doc you should use the REST API for further actions , the simplest approach would be to simply use the project id on the POST /projects.get_by_id endpoint .

Best regards,

Trying it now

print(f"start_date: {start_date} end_date: {end_date}") time_range = pd.date_range(start=start_date, end=end_date, freq='D').to_pydatetime().tolist()

Thanks @<1523701435869433856:profile|SmugDolphin23> , tho are you sure I don't need to override the deserialization function even if I pass multiple distinct objects as a tuple ?

Nope, I tried several upload AFTER putting the policy and all my uploads attempts were met with a progress stuck at 0% when trying

I checked the 'CPU-only' option in the auto-scaler config but that's seemed logic at the time

Yup, if you want to access it through https you're required to have a domain pointing to that IP with a certificate in place (using letsencrypt as instance) or else you'll get some SSL error

Well I simply duplicated code across my components instead of centraliwing the operations that needed that env variable in the controller

And there was still 2 instances running from the last pipeline run

So it seems to be an issue with the component parameter called in:
` @PipelineDecorator.pipeline(
name="VINZ Auto-Retrain",
project="VINZ",
version="0.0.1",
pipeline_execution_queue="Quad_VCPU_16GB"
)
def executing_pipeline(start_date, end_date):
print("Starting VINZ Auto-Retrain pipeline...")
print(f"Start date: {start_date}")
print(f"End date: {end_date}")

window_dataset_id = generate_dataset(start_date, end_date)

if name == 'main':
PipelineDec...