Hi DangerousDragonfly8 ! The file is there to test the upload to the bucket, as the name suggests. I don't think deleting it is a problem, and we will likely do that automatically in a future version

Yes, so even if you use a docker image with 3.8, the agent doesn't really know that you have 3.8 installed. If it is ran with 3.9, it will assume that is the desired version you want to use. So you need to change it in the config.
Not really sure why default_python is ignored (we will need to look into this), but python_binary should work...

@<1643060801088524288:profile|HarebrainedOstrich43> you are right. we actually attempt to copy the default arguments as well. What happens is that we aggregate these arguments in the kwargs dict, then we dump str(kwargs) in the script of the pipeline step. Problem is, str(dict) actually calls __ repr_ _ on each key/value of the dict, so you end up with repr(MyEnum.FALSE) in your code, which is <MyEnum.FALSE: 'FALSE'> . One way to work around this is to add somet...

Please let me know if this works!

DeliciousKoala34 can you upgrade to clearml==1.8.0 ? the issue should be fixed now

Hi @<1523702652678967296:profile|DeliciousKoala34> ! Looks like this is a bug in set_metadata . The model ID is not set, and set_metadata doesn't set it automatically. I would first upload the model file, then set the meta-data to avoid this bug. You can call update_weights to do that. None

Can you see your task if you run this minimal example UnevenDolphin73 ?
` from clearml import Task, Dataset

task = Task.init(task_name="name_unique", project_name="project")
d = Dataset.create(dataset_name=task.name, dataset_project=task.get_project_name(), use_current_task=True)
d.upload()
d.finalize() `

Hi PanickyMoth78 ! I ran the script and yes, it does take a lot more memory than it should. There is likely a memory leak somewhere in our code. We will keep you updated

otherwise, you could run this as a hack:

        dataset._dataset_file_entries = {
            k: v
            for k, v in self._dataset_file_entries.items()
            if k not in files_to_remove  # you need to define this
        }

then call dataset.remove_files with a path that doesn't exist in the dataset.

Hi ObedientDolphin41 ! Python allows you to decorate functions dynamically. See this example:
` from clearml.automation.controller import PipelineDecorator

@PipelineDecorator.component(repo=" ", repo_branch="master")
def step_one():
print('step_one')
return 1

def step_two_dynamic_decorator(repo=" ", repo_branch="master"):
@PipelineDecorator.component(repo=repo, repo_branch=repo_branch)
def step_two(arg):
print("step_two")
return arg
return step...

You could consider downgrading to something like 1.7.1 in the meantime, it should work with that version

Hi @<1523711002288328704:profile|YummyLion54> ! By default, we don't upload the models to our file server, so in the remote run we will try to pull the file from you local machine which will fail most of the time. Specify the upload_uri to the api.files_server entry in your clearml.conf if you want to upload it to the clearml server, or any s3/gs/azure links if you prefer a cloud provider

Hi PetiteRabbit11 . This snippet works for me:
` from clearml import Task
from pathlib2 import Path

t = Task.init()
config = t.connect_configuration(Path("config.yml"))
print(open(config).read()) Note the you need to use the return value of connect_configuration ` when you open the configuration file

UnevenDolphin73 looking at the code again, I think it is actually correct. it's a bit hackish, but we do use deferred_init as an int internally. Why do you need to close the task exactly? Do you have a script that would highlight the behaviour change between <1.8.1 and >=1.8.1 ?

I see. We need to fix both anyway, so we will just do that

So the flow is like:
MASTER PROCESS -> (optional) calls task.init -> spawns some children CHILD PROCESS -> calls Task.init. The init is deferred even tho it should not be?
If so, we need to fix this for sure

FlutteringWorm14 we do batch the reported scalars. The flow is like this: the task object will create a Reporter object which will spawn a daemon in another child process that batches multiple report events. The batching is done after a certain time in the child process, or the parent process can force the batching after a certain number of report events are queued.
You could try this hack to achieve what you want:
` from clearml import Task
from clearml.backend_interface.metrics.repor...

[package_manager.force_repo_requirements_txt=true] Skipping requirements, using repository "requirements.txt"
Try adding clearml to the requirements

@<1668427963986612224:profile|GracefulCoral77> You can both create a child or keep the same dataset as long as it is not finalized.
You can skip the finalization using the --skip-close argument. Anyhow, I can see why the current workflow is confusing. I will discuss it with the team, maybe we should allow syncing unfinalized datasets as well.

can you try setting the repo when calling add_function_step ?

Hi ApprehensiveSeahorse83 ! Looks like this is a bug. We will fix it ASAP

Hi FreshParrot56 ! This is currently not supported 🙁

HomelyShells16 looks like some changes have been made to jsonargparse and pytorch_lightning since we released this binding feature. could you try with jsonargparse==3.19.4 and pytorch_lightning==1.5.0 ? (no namespace parsing hack should be needed with these versions I believe)

ShinyPuppy47 do you have a small example we could take a look at?

Hi @<1643060801088524288:profile|HarebrainedOstrich43> ! Could you please share some code that could help us reproduced the issue? I tried cloning, changing parameters and running a decorated pipeline but the whole process worked as expected for me.

Hi EnergeticGoose10 . This is a bug we are aware of. We have already prepared a fix and we will release it ASAP.

Hi BoredBat47 ! What jsonschema version are you using?

ShinyPuppy47 Try this: use task = Task.init(...) (no create ) then call task.set_base_docker

Hi @<1576381444509405184:profile|ManiacalLizard2> ! Can you please share a code snippet that I could run to investigate the issue?

ShinyPuppy47 does add_task_init_call help your case? https://clear.ml/docs/latest/docs/references/sdk/task/#taskcreate