It runs directly but leads to the above error with clearml

Both manually (i.e. calling Task.init and running it without agent, and with agent ? same exact behavior ?

Thank you!

Hmm interesting...
of course you can do:
dataset._task.connect(...)But maybe it should be public?!

How are you using that (I mean in the context of a Dataset)?

GrievingTurkey78 I'm not sure I follow, are you asking how to add additional scalars ?

Ohh no I see, yes that makes sense, and I was able to reproduce m thanks!

I thought about the fact that maybe we need to write everything in one place

It will be in the same place, under the main Task
Should work out of the box

Can you reproduce this behavior outside of lightning? or in a toy example (because I could not)

total size 5.34 GB, 1 chunked stored (average size 5.34 GB)PanickyAnt52 The issue itself the Dataset will not break files (it will package into multiple zip files a large folder, but not break the file itself).
The upload itself is limited by the HTTP interface (i.e. 2GB file size limit)
I would just encode it into multiple Arrow files
does that make sense ?

SUCCESS!!!

Oh I see, this seems like Triton configuration issue, usually dim -1 means flexible. I can also mention that serving 1.1 should be released later this week with better multiple input support for triton. Does that make sense?

Should I map the poetry cache volume to a location on the host?

Yes, this will solve it! (maybe we should have that automatically if using poetry as package manager)
Could you maybe add a github issue, so we do not forget ?
Meanwhile you can add the mapping here:
https://github.com/allegroai/clearml-agent/blob/bd411a19843fbb1e063b131e830a4515233bdf04/docs/clearml.conf#L137
extra_docker_arguments: ["-v", "/mnt/cache/poetry:/root/poetry_cache_here"]

but could you try with the latest RC?

Yey!

Thank you so much @<1572395184505753600:profile|GleamingSeagull15> !

looks like your

faq.clear.ml

site is missing from your main sites sitemap files,

Thank you for noticing! I'll check with the webdevs

Also missing the

robots

meta tag on that site,

🙏

Last tip is to add a link on the

faq.clear.ml

site back to

clear.ml

for search index relevancy ( connects the two sites as being related in content...

pip3 install clearml==0.17.5rc5

Will do! 🙂

PungentLouse55 hmmm
Do you have an idea on how we could quickly reproduce it?

Could you test if this is working:
https://github.com/allegroai/clearml/blob/master/examples/reporting/matplotlib_manual_reporting.py

Hi @<1545216070686609408:profile|EnthusiasticCow4>
Oh dear, I think this argument is not exposed 😞

• You can open a GH
• If you want to add a PR this is very simple:None

include_archived=False,
):
   if not include_archived:
        system_tags = ["__$all", cls.__tag, "__$not", "archived"]
    else:
        system_tags = [cls.__tag]

...
   system_tag...

Okay give me a minute

oh the pipeline logic itself holds one "job" on the worker, and this is why you do not have any other spare workers to run the components of the pipeline.
Run your worker with --services-mode , it will launch multiple Tasks at the same time, it should solve the issue

What's the clearml-server version ?

RipeGoose2 you can put ut before/after the Task.init, the idea is for you to set it before any of the real training starts.
As for not effecting anything,
Try to add the callback and just have it returning None (which means skip over the model log process) let me know if this one works

Hi DepressedChimpanzee34
Why do you need to have the configuration added manually ? isn't the cleaml.conf easier ? If not I think OS environments are easier no? I run run above code, everything worked with no exception/warning... What is the try/except solves exactly ?

You can see the class here:
https://github.com/allegroai/clearml/blob/9b962bae4b1ccc448e1807e1688fe193454c1da1/clearml/binding/frameworks/init.py#L52

Basically you do:
` def my_callback(load_or_save, model):
# type: (str, WeightsFileHandler.ModelInfo) -> WeightsFileHandler.ModelInfo
assert load_or_save not in ('load', 'save')
# do something
if skip:
return None
return model

WeightsFileHandler.add_pre_callback(my_callback) `

Hmm that is odd.
Can you verify with the latest from GitHub?
Is this reproducible with the pipeline example code?

I have to assume that I do not know the dataset ID

Sorry I mean:

datasets = Dataset.list_datasets(dataset_project="some_project") 
for d in datasets:
  d["version"] = Dataset.get(dataset_id=d["id"]).version

wdyt?

Hi @<1547028031053238272:profile|MassiveGoldfish6>

The issue I am running into is that this command does not give me the dataset version number that shows up in the UI.

Oh no, I think you are correct, it will not return the version per dataset 😞 (I will make sure we add it)
But with the dataset ID you can grab all the properties:
Dataset.get(dataset_id="aabbcc").version
wdyt

Hi ProudChicken98
task.connect(input) preserves the types based on the "input" dict types, on the flip side get_parameters returns the string representation (as stored on the clearml-server).
Is there a specific reason for using get_parameters over connect ?

let me check