After you do s['Function']['random_number'] = random.random() you still need to call set_parameters_as_dict(s)

ClearML does not officially support a remotely executed task to spawn more tasks we do through pipelines, it that helps you somehow. Note that doing things the way you do them right now might break some other functionality.
Anyway, I will talk with the team and maybe change this behaviour because it should be easy 👍

Hi OutrageousSheep60 ! The fix for Dataset.list_datasets() will be out in the next release of ClearML SDK. Sorry for the inconvenience!

JitteryCoyote63 very odd, it seems to work just fine on my machine

Hi @<1545216070686609408:profile|EnthusiasticCow4> ! Can't you just get the values of the hyperparameters and the losses, then plot them with something like mathplotlib then just report the plot to ClearML?

how does your model look like?

UnevenDolphin73 looks like we clear all loggers when a task is closed, not just clearml ones. this is the problem

Hi @<1523701949617147904:profile|PricklyRaven28> ! We released ClearmlSDK 1.9.1 yesterday. Can you please try it?

PanickyMoth78 Something is definitely wrong here. The fix doesn't seem to be trivial as well... we will prioritize this for the next version

Do you have the full exception trace?

Thank you

Can you please update it to the latest version? pip install -U jsonschema

basically, I think that the pipeline run starts from __ main_ _ and not the pipeline function, which causes the file to be read

@<1554638160548335616:profile|AverageSealion33> looks like hydra pulls the config relative to the scripts directory, and not the current working directory. The pipeline controller actually creates a temp file in /tmp when it pulls the step, so the script's directory will be /tmp and when searching for ../data , hydra will search in / . The .git likely caused your repository to be pulled, so your repo structure was created in /tmp , which caused the step to run correctly...

it's the same file you added your s3 creds to

I meant the code where you upload an artifact, sorry

The only expection is the models if I'm not mistaken, which are stored locally by default.

can you share your config? (make sure to remove any credentials)

SuccessfulKoala55 1.5.1

Hi PricklyRaven28 ! What dict do you connect? Do you have a small script we could use to reproduce?

MotionlessCoral18 If you provide the model as a hyperparam, then I believe you should query its value by calling https://clear.ml/docs/latest/docs/references/sdk/task/#get_parameters or https://clear.ml/docs/latest/docs/references/sdk/task/#get_parameter

@<1526734383564722176:profile|BoredBat47> How would you connect with boto3 ? ClearML uses boto3 as well, what it basically does is getting the key/secret/region from the conf file. After that it opens a Session with the credentials. Have you tried deleting the region altogether from the conf file?

Hi @<1523701304709353472:profile|OddShrimp85> ! Can you please share the logs (make sure to remove any sensitive data, if it exists)

Hi DangerousDragonfly8 ! The file is there to test the upload to the bucket, as the name suggests. I don't think deleting it is a problem, and we will likely do that automatically in a future version

Yes, so even if you use a docker image with 3.8, the agent doesn't really know that you have 3.8 installed. If it is ran with 3.9, it will assume that is the desired version you want to use. So you need to change it in the config.
Not really sure why default_python is ignored (we will need to look into this), but python_binary should work...

@<1643060801088524288:profile|HarebrainedOstrich43> you are right. we actually attempt to copy the default arguments as well. What happens is that we aggregate these arguments in the kwargs dict, then we dump str(kwargs) in the script of the pipeline step. Problem is, str(dict) actually calls __ repr_ _ on each key/value of the dict, so you end up with repr(MyEnum.FALSE) in your code, which is <MyEnum.FALSE: 'FALSE'> . One way to work around this is to add somet...

DeliciousKoala34 can you upgrade to clearml==1.8.0 ? the issue should be fixed now

Hi @<1523702652678967296:profile|DeliciousKoala34> ! Looks like this is a bug in set_metadata . The model ID is not set, and set_metadata doesn't set it automatically. I would first upload the model file, then set the meta-data to avoid this bug. You can call update_weights to do that. None

Can you see your task if you run this minimal example UnevenDolphin73 ?
` from clearml import Task, Dataset

task = Task.init(task_name="name_unique", project_name="project")
d = Dataset.create(dataset_name=task.name, dataset_project=task.get_project_name(), use_current_task=True)
d.upload()
d.finalize() `

Hi PanickyMoth78 ! I ran the script and yes, it does take a lot more memory than it should. There is likely a memory leak somewhere in our code. We will keep you updated