how does your model look like?

Hi OutrageousSheep60 ! Regarding your questions:
No it's not. We will have a RC that fixes that ASAP, hopefully by tomorrow You can use add_external_files which you already do. If you wish to upload local files to the bucket, you can specify the output_url of the dataset to point the bucket you wish to upload the data to. See the parameter here: https://clear.ml/docs/latest/docs/references/sdk/dataset/#upload . Note that you CAN mix external_files and regular files. We don't hav...

thank you! we will take a look and come back to you

Hi @<1578555761724755968:profile|GrievingKoala83> ! Can you share the logs after setting NCCL_DEBUG=INFO of all the tasks? Also, did it work for you 5 months ago because you were on another clearml version? If it works with another version, can you share that version number?

basically, I think that the pipeline run starts from __ main_ _ and not the pipeline function, which causes the file to be read

@<1545216070686609408:profile|EnthusiasticCow4> yes, that's true. I would aggregate the tasks by tags (the steps will be tagged with opt: ID ), None then get the metrics to get the losses None , and look into the tasks config to get the term you wanted to optimize [None](https://clear.ml/docs/latest/docs/references/sdk/task/#get_last...

Hi DangerousDragonfly8 ! The file is there to test the upload to the bucket, as the name suggests. I don't think deleting it is a problem, and we will likely do that automatically in a future version

Basically, it looks like the agent installs an outdated pip version and this should fix it, and hopefully install your packages correctly

you could also try using gloo as the backend (it uses CPU) just to check that the subprocesses spawn properly

Hi @<1570220858075516928:profile|SlipperySheep79> ! What happens if you do this:

import yaml
import argparse
from my_pipeline.pipeline import run_pipeline
from clearml import Task

parser = argparse.ArgumentParser()
parser.add_argument('--config', type=str, required=True)

if __name__ == '__main__':
    if not Task.current_task():
      args = parser.parse_args()
      with open(args.config) as f:
          config = yaml.load(f, yaml.FullLoader)
    run_pipeline(config)

Hi @<1545216070686609408:profile|EnthusiasticCow4> ! Can't you just get the values of the hyperparameters and the losses, then plot them with something like mathplotlib then just report the plot to ClearML?

@<1654294828365647872:profile|GorgeousShrimp11> Any change your queue is actually named megan-testing and not megan_testing ?

PanickyMoth78 there is no env var for sdk.google.storage.pool_connections/pool_maxsize . We will likely add these env vars in a future release.
Yes, setting max_workers to 1 would not make a difference. The docs look a bit off, but it is specified that 1: if the upload destination is a cloud provider ('s3', 'gs', 'azure') .
I'm thinking now that the memory issue might also be cause because of the fact that we prepare the zips in the background. Maybe a higher max_workers wou...

1.10.2 should be old enough

Hi @<1715900760333488128:profile|ScaryShrimp33> ! You can set the log level by setting the CLEARML_LOG_LEVEL env var before importing clearml. For example:

import os
os.environ["CLEARML_LOG_LEVEL"] = "ERROR"  # or str(logging.CRITICAL/whatever level) also works 

Note that the ClearML Monitor warning is most likely logged to stdout, in which case this message can't really be suppressed, but model upload related message should be

Hi @<1533257278776414208:profile|SuperiorCockroach75> Try setting packages in your pipline component to your requirements.txt or simply add the list of packages (with the specific versions). None

this is a bug, we will fix this asap

hi OutrageousSheep60 ! We didn't release an RC yet, we will a bit later today tho. We will ping you when it's ready, sorry for the delay

Hi @<1555000563244994560:profile|OutrageousSealion55> ! How do you pass base_task_id in the HyperParamterOptimizer ?

QuaintJellyfish58 We will release later today an RC that adds the region to boto_kwargs . We will ping you when it's ready to try it out

Hi @<1597762318140182528:profile|EnchantingPenguin77> ! You should be able to see the overrides unde CONFIGURATION->HYPERPARAMETERS->Args->overrides:

Hi @<1523708920831414272:profile|SuperficialDolphin93> ! What if you do just controller.start() (to start it locally). The task should not quit in this case.

Hi DangerousDragonfly8 ! At the moment, this is not possible, but we do have it in plan (we had some prior requests for this feature)

Hi @<1566596968673710080:profile|QuaintRobin7> ! Sometimes, ClearML is not capable of transforming matplotlib plots to plotly , so we report the plot as an image to Debug Samples. Looks like report_interactive=True makes the plot unparsable

Hi @<1524560082761682944:profile|MammothParrot39> ! Do you see the parameter do_stuff under your pipeline controller's task?

this is likely an UI bug. We should have a fix soon. In the meantime, yes, you can edit the configuration under the pipeline task to achieve the same effect

Hi @<1523701504827985920:profile|SubstantialElk6> !
Regarding 1: pth files get pickled.
The flow is like this:

• The step is created by the controller by writing some code to a file and running that file in python
• The following line is ran in the step when returning values: None
• This is eventually ran: [None](https://github.com/allegroai/clearml/blob/cbd...

Hi @<1578918167965601792:profile|DistinctBeetle43> ! This is currently not possible. A different task will be created for each instance

Perfect! Can you please provide the sizes of the files of the other 2 chunks as well?

hi QuaintJellyfish58 ! How does your clearml.conf look like? How do you run minio ? Can you download files using boto3 rather than clearml ? Could you provide a script that could help us reproduce the issue?