AgitatedDove14 Yes exactly!

--args param.ovveride=value does not work with clearml-task

By the way, since if i create the task locally, reset it and enqueue it, it works. This is the workaround that i'm using right now

Yes the task is running on a remote agent with the --docker flag
this is the config on the machine(s) running the agent
`
agent {
venvs_cache: {
max_entries: 50
free_space_threshold_gb: -1
path: ~/.clearml/venvs-cache
}
extra_docker_arguments: [
"--network", "host",
"-v", "/home/ubuntu/.ssh:/root/.ssh:ro",
"-v", "/home/ubuntu/.cache:/root/.cache",
]

docker_internal_mounts {
    sdk_cache: "/clearml_agent_cache"

...

I used an env variable to avoid creating and endless loop of init/enqueue (using an argument like clearml.queue that would be captured and forwarded to the agent)

It only works with argparse

I tried like that clearml-task --script train.py --args overrides="log.clearml=True train.epochs=200 clearml.save=True" --project mornag-plain-dense --name mornag-plain-dense-training --queue tesla-t4 --skip-task-init

Related GitHub issue https://github.com/allegroai/clearml/issues/847

I opened the issue on github https://github.com/allegroai/clearml-web/issues/46

Yes it uses hydra and everything works fine without clearml. The script is similar to this one https://github.com/galatolofederico/lightning-template/blob/main/train.py

This is the full print(cfg)
` {'dataset': {'name': '<my-dataset-name>', 'path': '', 'target_shape': [128, 128]}, 'model': 'unet', 'models': {'unet': {'dim': 8, 'dim_mults': [1, 2, 4], 'num_blocks_per_stage': [2, 2, 2], 'num_self_attn_per_stage': [0, 0, 1], 'nested_unet_depths': [0, 0, 0], 'nested_unet_dim': 16, 'use_convnext': False, 'resnet_groups': 2, 'consolidate_upsample_fmaps': True, 'weight_standardize': False, 'attn_heads': 2, 'attn_dim_head': 16}}, 'train': {'accelerator': 'auto...

But not argument is passed to the scripts. Here i am printing sys.argv

This is what it looks like from the web app

Everything works fine i force those values using AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY env variables

Ok i did some investigations and the bug appear from version 1.8.0. In version 1.7.0 there is not. I open a issue in GitHub

The debug samples are correctly uploaded in the bucket (is a minio bucket) i can see them from the minio webapp. I have used logger.report_image

if i print config_list from def from_config(cls, s3_configuration): in file bucket_config.py line 121 i get
{'key': '', 'secret': '', 'region': '', 'multipart': True, 'use_credentials_chain': False, 'bucket': 'clearml', 'host': 's3.myhost.tld:443', 'token': '', 'extra_args': ConfigTree()}

Nevermind my aws config was not under sdk :face_palm:

The error is
2022-11-28 14:40:17,099 - clearml.storage - ERROR - Failed creating storage object Reason: Missing key and secret for S3 storage access ( )

No luck with --args overrides="['dataset.path=abcd']"

I installed clearml from source and printed the internal S3 configurations, basically key and secret are empty

No problem! Thank you for your amazing work!

I also save the models in the s3 bucket using output_uri=cfg.clearml.output_uri, in the Task.init

I specified the upload destination in the logger. Logger.current_logger().set_default_upload_destination(cfg.clearml.media_uri) . Yes Minio with no special config. The s3 config is in the clearml.conf

just tried 1.9.1 and it is affected

Also 1.9.1-312 is affected

1.9.0 is still affected

The file has been uploaded correctly in the bucket

i printed cfg in the script and the config has not been overwritten 😢