JitteryCoyote63 is this still an issue?

You will be able to set it.
You will just not see the output in the console log , but everything is running and being executed

so the thing with IAM roles, they are designed to allow AWS instances to get "automatic" permission (based on the IAM role). They are not actually designed to generate key/secret as I think the lifetime is be default relatively short. Since the actual request to the S3 comes from the client browser (i.e. outside of AWS cluster) the IAM role cannot apply, and you have to provide the key/secret. The easiest way is to generate S3 keys regardless of the IAM roles, to be used with the clients (sp...

Yea I know, I reported this

LOL, apologies these days it a miracle I still remember my login passwords 😉

well cudnn is actually missing from the base image...

Hmm, I think I need more to try and reproduce, what exactly did you do, what was the expected behavior vs reality ?

Hmm, in the credentials popup there should be a "secure connect" checkbox, it tells it to use https instead of http. Can you verify?

can the ClearML File server be configured to any kind of storage ? Example hdfs or even a database etc..

DeliciousBluewhale87 long story short, no 🙂 the file server, will just store/retrieve/delete files from a local/mounted folder

Is there any ways , we can scale this file server when our data volume explodes. Maybe it wouldnt be an issue in the K8s environment anyways. Or can it also be configured such that all data is stored in the hdfs (which helps with scalablity).I would su...

Hmm, how does your preprocessing code looks like?

Thanks IntriguedRat44 !
I'll follow up on GitHub 🙂

What you actually specified is torch the @ is kind of pip remark, pip will not actually parse it 🙂
use only the link https://download.pytorch.org/whl/cu100/torch-1.3.1%2Bcu100-cp36-cp36m-linux_x86_64.whl

Thanks a lot. I meant running a bash script after cloning the repository and setting the environment

Hmm that is currently not supported 😞
The main issue in adding support is where to store this bash script...

Perhaps somewhere inside clear ml there is an order of actions for starting that can be changed?

Not that I can think of,
but let's assume you could have such a thing, what would you have put in the bash script (basically I want to see maybe there is a worka...

Hmm could it be this is on the "helper functions" ?

Verified, you are correct there is no way to override with OS environment 😞

You might be able to write a script to override the links ... wdyt?

SweetGiraffe8
That might be it, could you test with the Demo server ?

Great, please feel free to share your thoughts here 🙂

Notice you should be able to override them in the UI (under Args seciton)

wdyt?
https://clear.ml/docs/latest/docs/clearml_serving/clearml_serving
https://clear.ml/docs/latest/docs/clearml_serving/clearml_serving_cli#metrics
https://clear.ml/docs/latest/docs/clearml_serving/clearml_serving_tutorial#model-monitoring-and-performance-metrics

BattyLion34 is this running with an agent ?
What's the comparison with a previously working Task (in terms of python packages) ?

Nice debugging experience

Kudos on the work !
BTW, I feel weird to add an issue on their github, but someone should, this generic setup will break all sorts of things ...

Can clearml-serving does helm install or upgrade?

Not sure I follow, how would a helm chart install be part of the ml running ? I mean clearml-serving is installed via helm chart, but this is a "one time" i.e. you install the clearm-serving and then you can via CLI / python send models to be served there, this is not a "deployed per model" scenario, but a deployment for multiple models, dynamically loaded

👍

Yes, experiments are standalone as they do not have to have any connecting thread.
When would you say a new "run" vs a new "experiment" ? when you change a parameter ? change data ? change code ?
If you want to "bucket them" use projects 🙂 it is probably the easiest now that we have support for nested projects.

SarcasticSquirrel56 when the process dies (i.e. killed) it does not have time not update the state, then the server watchdog will set the state to aborted after X amount of time of inactivity (default is 2 hours)

Now I’m just wondering if I could remove the PIP install at the very beginning, so it starts straightaway

AbruptCow41 CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL=1 does exactly that 🙂 BTW, I would just set the venv cache and this means it will just be able to restore the entire thing (even if you have changed the requirements
https://github.com/allegroai/clearml-agent/blob/077148be00ead21084d63a14bf89d13d049cf7db/docs/clearml.conf#L115

Hi AverageBee39
What's the clearml-server and clearml packge you are using ?
(I looks like some capability that is missing from the server, i.e. needs upgrade ?!)

Hi VexedCat68
Could it be the python version is not the same? (this is the only reason not to find a specific python package version)

This is also set in the command line.
--cpu-only or maybe without any --gpus flag at all

Now I suspect what happened is it stayed on another node, and your k8s never took care of that