we will have a dedicate vm to hold trains related docker, do I need to setup some file server? (i saw earlier thread mention minio)

I see, I will look into the documentation of it, thanks Jake.

Also I am unclear what is the difference of storageManager and StorageHelper, is there an example that integrate that with model training.

I go through the doc and seems it doesn't mention downloading from artifact (programatically)?

Ok, will prepare a PR and script to reproduce the error

I have been using this line to prevent experiments won't accidentally sent to the public server (I have my custom self-hosted server)
Task.set_credentials("PLACEHOLDER", "PLACEHOLDER","PLACEHOLDER")

However, when I upgraded from 0.17.5 -> > 1.0.0. Weird stuff happen.
Since upgrade from v0.17.5 -> > 1.0.0, it has issue replacing the credentials.

Expected Behavior:

Conf should replace the "PLACEHOLDER" is the conf file exist. Else it should fails the experiment.

What happened:

The ...

Oh I did not realize I asked this in a old thread, sorry about that.

I can confirm this seems to fix this issue, and I have reported this issue to kedro team see what's their view on this. So it seems like it did remove the TaskHandler from the _handler_lists

lol...... mine is best_model_20210611_v1.pkl and better_model_20210611_v2.pkl or best_baseline_model_with_more_features.pkl

i.e. some files in a shared drive, then someone silently updated the files and all the experiments become invalid and no one knows when did that happened.

Ok, sorry, this is my mistake, it's actually inside a loop, so this make sense.

I couldn't report it to demo server, since this involve internal stuff...

ok, it makes sense. Is there a way to let trains save it without blocking the program ?

GrumpyPenguin23 yes, those features seems to related to other infrastructure, not Trains (ML experiment management)

I am abusing the "hyperparameters" to have a "summary" dictionary to store my key metrics, due to the nicer behaviour of diff-ing across experiments.

Thanks Jake, I just upgraded our server to ClearML It works well and fixed some of the annoying UI bugs and enhancements. :)

VivaciousPenguin66 What's your thought on Prefect? There are so many pipeline library and I wasn't so sure how different are they. I have experience with Airflow. With Kedro, we were in hope that data scientist will write the pipeline themselves with minimal effort to handover to another engineer to work on. For serious production (need to scale), we consider convert Kedro pipeline to Airflow, there are plugin to do that, tho I am not sure how mature they are.

Hi, I think I can confirm this is a bug of Trains. Is that ok if I submit a PR to fix this?

for the most common workflow, I may have some csv, which may be updated from time to time

I wonder what's the extra features is offered in the enterprise solution tho

AgitatedDove14
are the data versioning completely different from the Trains Artifact/storage solution? or it's some enhanced feature.

No, I mean it capture the plot somehow, as you can see the left side there are a list of plot, but it does not show up.

matplotlib.version
'3.1.3'

This will make the plotting fail

potentially both, but let just say structure data first, like CSV, pickle (may not be a table, could be any python object), feather, parquet, some common data format

I think it's related to the fix that use "incremental: true", this seems to fix 1 problem, but at the same time it will ignore all other handlers.

I am not sure what are those example/1/2/3 are, I only have one chart

AgitatedDove14 I believe you mean plt.savefig? I used this function to save my charts, but it does not show up as well.

AgitatedDove14 No, unless I close the window manually.

Disable the matplotlib GUI does work.

and the 8 charts are actually identical

good to know~