Hi @<1668427963986612224:profile|GracefulCoral77> ! The error is a bit misleading. What it actually means is that you shouldn't attempt to modify a finalized clearml dataset (I suppose that is what you are trying to achieve). Instead, you should create a new dataset that inherits from the finalized one and sync that dataset, or leave the dataset in an unfinalized state

@<1668427963986612224:profile|GracefulCoral77> You can both create a child or keep the same dataset as long as it is not finalized.
You can skip the finalization using the --skip-close argument. Anyhow, I can see why the current workflow is confusing. I will discuss it with the team, maybe we should allow syncing unfinalized datasets as well.

Yes, so even if you use a docker image with 3.8, the agent doesn't really know that you have 3.8 installed. If it is ran with 3.9, it will assume that is the desired version you want to use. So you need to change it in the config.
Not really sure why default_python is ignored (we will need to look into this), but python_binary should work...

JitteryCoyote63 very odd, it seems to work just fine on my machine

can you share your config? (make sure to remove any credentials)

SuccessfulKoala55 1.5.1

Hi @<1523701949617147904:profile|PricklyRaven28> ! We released ClearmlSDK 1.9.1 yesterday. Can you please try it?

@<1526734383564722176:profile|BoredBat47> How would you connect with boto3 ? ClearML uses boto3 as well, what it basically does is getting the key/secret/region from the conf file. After that it opens a Session with the credentials. Have you tried deleting the region altogether from the conf file?

Hi again, @<1526734383564722176:profile|BoredBat47> ! I actually took a closer look at this. The config file should look like this:

        s3 {
            key: "KEY"
            secret: "SECRET"
            use_credentials_chain: false

            credentials: [
                {
                    host: "myendpoint:443"  # no http(s):// and no s3:// prefix, also no bucket name
                    key: "KEY"
                    secret: "SECRET"
                    secure: true  # ...

I meant the code where you upload an artifact, sorry

And I believe that by default we send artifacts to the clearml server if not specified

it's the same file you added your s3 creds to

The S3 endpoint

Hi @<1533620191232004096:profile|NuttyLobster9> We likely print the warning by mistake. We will look into it soon and handle it properly

Hi @<1558986839216361472:profile|FuzzyCentipede59> ! Can you share some snippets of your code, and tell us what you expect to see vs what you actually see is happening?

Hi @<1545216070686609408:profile|EnthusiasticCow4> ! This is a known bug, we will likely fix it in the next version

Hi @<1523701504827985920:profile|SubstantialElk6> !
Regarding 1: pth files get pickled.
The flow is like this:

• The step is created by the controller by writing some code to a file and running that file in python
• The following line is ran in the step when returning values: None
• This is eventually ran: [None](https://github.com/allegroai/clearml/blob/cbd...

I think that will work, but I'm not sure actually. I know for sure that something like us-east-2 is supported

Hi @<1590514584836378624:profile|AmiableSeaturtle81> ! Having tqdm installed in your environment might help

Hi @<1633638724258500608:profile|BitingDeer35> ! Looks like the SDK doesn't currently allow to create steps/controllers with a designated cwd. You will need to call the set_script function on your step's tasks and on the controller for now.
For the controller: If you are using the PipelineDecorator, you can do something like: PipelineDecorator._singleton._task.set_script(working_dir="something") , before you are running the pipeline function. In the case of regular `PipelineControll...

Hi @<1545216070686609408:profile|EnthusiasticCow4> ! That's correct. The job function will run in a separate thread on the machine you are running the scheduler from. That's it. You can create tasks from functions tho using backend_interface.task.populate.CreateFromFunction.create_task_from_function

With that said, can I run another thing by you related to this. What do you think about a PR that adds the functionality I originally assumed schedule_function was for? By this I mean: adding a new parameter (this wouldn't change anything about schedule_function or how .add_task() currently behaves) that also takes a function but the function expects to get a task_id when called. This function is run at runtime (when the task scheduler would normally execute the scheduled task) and use ...

Hi @<1523721697604145152:profile|YummyWhale40> ! Are you able to upload artifacts of any kind other than models to the CLEARML_DEFAULT_OUTPUT_URI?

Could you please try with an older sdk version just to make sure there were no regressions?

What clearml sdk version are you using?

1.10.2 should be old enough

@<1523721697604145152:profile|YummyWhale40> are you able to manually save models from SageMaker using OutputModel ? None

I think we should just have a new parameter

No need, I think I will review it on Monday

Sure, thank you!