Hi @<1578918167965601792:profile|DistinctBeetle43> ! This is currently not possible. A different task will be created for each instance

What if you add images to the dataset? Can you see them being previewed? @<1523701168822292480:profile|ExuberantBat52>

Haha, should not be too complicated to add one. We will consider it. Thanks for reporting the issue

Yeah, that's always the case with complex systems 😕

Hi HomelyShells16 How about doing things this way? does it work for you?
` class ClearmlLightningCLI(LightningCLI):
def init(self, *args, **kwargs):
Task.add_requirements("requirements.txt")
self.task = Task.init(
project_name="example",
task_name="pytorch_lightning_jsonargparse",
)
super().init(*args, **kwargs)

def instantiate_classes(self, *args, **kwargs):
    super().instantiate_classes(*args, **kwargs)
  ...

UnevenDolphin73 The task shouldn't disappear when using use_current_task=False . There might be something else that makes it disappear.

Can you actually add the bucket to the credentials just to try it out?
Also, can you check that this snippet works for you (with your creds):

import boto3
import json
import six

key = ""
secret = ""
host = "our_host.com"
bucket_name = "bucket"
profile = None
filename = "test"
data = {"test": "data"}

boto_session = boto3.Session(aws_access_key_id=key, aws_secret_access_key=secret, profile_name=profile)
endpoint = "https://" + host
boto_resource = boto_session.resource("s3", region_name...

How about if Task.running_locally(): ?

I left another comment today. It’s about something raising an exception when creating a set from the file entries

FiercePenguin76 Are you changing the model by pressing the circled button in the first photo? Are you promted with a menu like in the second photo?

After you do s['Function']['random_number'] = random.random() you still need to call set_parameters_as_dict(s)

you might want to prefix both the host in the configuration file and the uri in Task.init / StorageHelper.get with s3. if the script above works if you do that

ClearML does not officially support a remotely executed task to spawn more tasks we do through pipelines, it that helps you somehow. Note that doing things the way you do them right now might break some other functionality.
Anyway, I will talk with the team and maybe change this behaviour because it should be easy 👍

Hi OutrageousSheep60 ! The fix for Dataset.list_datasets() will be out in the next release of ClearML SDK. Sorry for the inconvenience!

JitteryCoyote63 very odd, it seems to work just fine on my machine

Hi @<1545216070686609408:profile|EnthusiasticCow4> ! Can't you just get the values of the hyperparameters and the losses, then plot them with something like mathplotlib then just report the plot to ClearML?

how does your model look like?

UnevenDolphin73 looks like we clear all loggers when a task is closed, not just clearml ones. this is the problem

Hi @<1523701949617147904:profile|PricklyRaven28> ! We released ClearmlSDK 1.9.1 yesterday. Can you please try it?

PanickyMoth78 Something is definitely wrong here. The fix doesn't seem to be trivial as well... we will prioritize this for the next version

Do you have the full exception trace?

Thank you

Can you please update it to the latest version? pip install -U jsonschema

basically, I think that the pipeline run starts from __ main_ _ and not the pipeline function, which causes the file to be read

@<1554638160548335616:profile|AverageSealion33> looks like hydra pulls the config relative to the scripts directory, and not the current working directory. The pipeline controller actually creates a temp file in /tmp when it pulls the step, so the script's directory will be /tmp and when searching for ../data , hydra will search in / . The .git likely caused your repository to be pulled, so your repo structure was created in /tmp , which caused the step to run correctly...

it's the same file you added your s3 creds to

I meant the code where you upload an artifact, sorry

Could you try adding region under credentials as well?

The only expection is the models if I'm not mistaken, which are stored locally by default.

can you share your config? (make sure to remove any credentials)