I think the main difference is that I can see a value of having access to the raw format within the cloud vendor and not only have it as an archive

I see it does make sense.
Two options, one, as you mentioned use the ClearML StorageManager to upload the files, then register them as external links with Dataset.
Two, I know the enterprise tier has HyperDatasets, that are essentially what you describe, with version control over the "metadata" and "raw storage" on the GCP, including the ab...

TenseOstrich47 Nice!

My question is, which version do you need docker compose?

Ohh sorry, there is no real restriction, we just wanted easy copy-paste for the installation process.

Where is darts reporting scalars ?

Ok i did a pip install -r requirements.txt and NOW it picks them up correctly
So packages have to be installed and not just be mentioned in requirements / imported?

Yes, it looks for them locally so it has all the specific versions you need.
If the "installed packages" is totally empty the agent will revert to looking for requirements.txt inside the repository.

Thanks StaleButterfly40 !

Let me check

GrittyKangaroo27 any chance you can open a GitHub issue so this is not forgotten ?
(btw: we I think 1.1.6 is going to be released later today, then we will have a few RC with improvements on the pipeline, I will make sure we add that as well)

MysteriousBee56 that is very strange definitely explains it, kudos on debugging it !!!

Hi GrittyKangaroo27

How could I turn off model logging when running this training step?

This is a good point! I think we cannot pass these arguments.
Would this make sense to you?
PipelineDecorator.component(..., auto_connect_frameworks)
wdyt?

Hi RotundHedgehog76
I think it should work out of the box, I mean at the end both spin jupyter notebooks, which is what clearml interacts with. Are you getting any errors?

looks like a great idea, I'll make sure to pass it along and that someone reply 🙂

RoundMosquito25 this is a good point, I mean in theory it could be done, the question is the actual Bayesian optimization you are using.
Is it optuna (OptimizerOptuna) or OptimizerBOHB?

Nice! !!!
🎊

Can you login to it?

So it should cache the venvs right?

Correct,

path: /clearml-cache/venvs-cache

Just making sure, this is the path to the host cache folder

ClumsyElephant70 I think I lost track of the current issue 😞 what's exactly not being cached (or working)?

GrievingTurkey78 can you send the entire log?

No by definition the agent will only execute one Task at a time, you can spin a second agent on the same GPU :)

Hi IrritableGiraffe81

I have a package called

feast[redis]

in my requirements.txt file.

This means feast is installing additional packages, once the agent is done installing everything, it basically calls pipe freeze and stores back All the packages including versions
Now the question is, how come redis is not installed.
Notice that the Task already has the autodetected packages (it basically ignores requirem,ents.txt as it is often not full missing or just wrong)
...

I see.
You can get the offline folder programmatically then copy the folder content (it's the same as the zip, and you can also pass a folder instead of zip to the import function)
task.get_offline_mode_folder()You can also have a soft link of the offline folder (if you are working on a linux machine:
ln -s myoffline_folder ~/.trains/cache/offline

HugePelican43 sure you can, usually the limiting factor is memory, as it cannot be shared among processes, so if one allocated all memory the second process will crash with out of memory error

. Could you clarify the question for me, please?
...
Could you please point me to the piece of ClearML code related to the downloading process?

I think I mean this part:
https://github.com/allegroai/clearml/blob/e3547cd89770c6d73f92d9a05696018957c3fd62/clearml/datasets/dataset.py#L2134

Or a nicer one here:
https://demoapp.trains.allegro.ai/projects/97f6b5b53a0243c196d6f49c221cbdca/compare-experiments;ids=cdc2cc156ae042f08dab2b66756f468a,bb76b70520e046ebbcc21613926e7316,189a495824544718b4c271ce9575f32c/hyper-params/graph?hyper-params=graph

Hi CleanPigeon16
I think now the issue is missing git credentials, did you pass git_user / git_pass to the AWS autoscaler ?

Could you test with 4.7.5 ?

Also I can’t call the “preprocess” function since there is no valid endpoint to be hitting

Wait now I'm confused, when you are calling " None " you are actually calling the preprocess function running on the inference container, and this one in turn (automatically) calls the Triton container.

Are you calling the Triton manually?
Could you share your preprcoess.py , and the command line you have used to register the two model versions ?
(based on ...

WhimsicalLion91

What would you say the use case for running an experiment with iterations

That could be loss value per iteration, or accuracy per epoch (iteration is just a name for the x-axis in a sense , this is equivalent to time series)
Make sense?

Hi EnviousPanda91
You mean like collect plots, then generate a pdf?

Yep, but a funny hack nonetheless.
No idea why they have it there...

MysteriousBee56 Okay, let's try this one:
docker run -t --rm nvidia/cuda:10.1-base-ubuntu18.04 bash -c "echo 'Binary::apt::APT::Keep-Downloaded-Packages \"true\";' > /etc/apt/apt.conf.d/docker-clean && apt-get update && apt-get install -y git python3-pip && python3 -m pip install trains-agent && echo done"