Looks great, let me see if I can understand what's missing, because it should have worked ...

Hi LackadaisicalOtter14

However, whenever we spin up a session, 

 always gets run and overwrites our configs

what do you mean by that?
The what config are being overwritten? (generally speaking, it just add the OS environment it needs to for the setup process)

I want to be able to delete only the logs since they are taking a lot of space in my case.

I see... I do not think this is possible 😞
You can disable the auto logging though ... pass auto_connect_streams=False to Task.init

It said the command --aux-config got invalid input

This seems like an interface bug.. let me see if we can fix that 🙂

BTW: this seems like a triton LSTM configuration issue, we might want to move the discussion to the Triton server issue, wdyt?

Definitely!

Could you start an issue https://github.com/triton-inference-server/server/issues , and I'll jump join the conversation?

. Is there any reference about integrating kafka data streaming directly to clearml-serving...

one of the two experiments for the worker that is running both experiments

So this is the actual bug ? I need some more info on that, what exactly are you seeing?

I have timeseries dataset with dimension 1,60,1 which the first dimension is number of data, the second one is timestep

I think it should be --input-size 1 60 ` if the last dimension is the batch size?
(BTW: this goes directly to Triton configuration, it is the information Triton needs in order to run the model itself)

Hmm, how does your preprocessing code looks like?

wdyt?
https://clear.ml/docs/latest/docs/clearml_serving/clearml_serving
https://clear.ml/docs/latest/docs/clearml_serving/clearml_serving_cli#metrics
https://clear.ml/docs/latest/docs/clearml_serving/clearml_serving_tutorial#model-monitoring-and-performance-metrics

So I would try: [-1, 60, 1]

Right, you need to pass "repo" and direct it to the repository path
(BTW, what's the cleaml version)

If there is new issue will let you know in the new thread

Thanks! I would really like to understand what is the correct configuration

the trend step artifact used to keep track the time of the data so we know the expected trend of the input data. For example, on the first data which is trend_step = 1 the trend value is 10, then if the trend_step = 10 (the tenth data) our regressor will predict the trend value of the selected trend_step. this method is still in research to make it more efficient so it doesn't need to upload artifact every request

Make sense! I would suggest you add a GitHub issue with feature request ...

EmbarrassedPeacock82 are you using keras/pytorch etc for serving (i.e. Triton) ?

Correct 🙂

Run ifconfig

Very odd, I still can't reproduce. This is just the cleanup service running without anything else ?
What's the clearml version it is using ?

Hi IrritableJellyfish76
If you are running a code that uses clearml from kubeflow, you have out of the box integration between the two, what am I missing?

No sure I follow, you mean to launch it on the kubernretes cluster from the ClearML UI?
(like the clearml-k8s-glue ?)

Hi TrickySheep9
Hmm I think you are correct, exit remotely will not work inside a jupyter notebook because it will not be able to close it.

I was just revising workflows that might be similar, wdyt?
https://clearml.slack.com/archives/CTK20V944/p1620506210463400?thread_ts=1614234125.066600&cid=CTK20V944

I just called exit(0) in a notebooke and it closed it (the kernel) no exception

This is a Sagemaker notebook instances

Yes I think this is the issue

Jupyter server v 6.0.3

You cannot call exit(0) and kill the kernel from the SageMake notebook

Just a bit of background, the execute)remotely will kill the current process (after the Task is synced) and enqueue the Task that was created for remote execution. What seems to fail is actually killing the current process. You can just pass exit_process=False

No worries 🙂

Hi @<1684010629741940736:profile|NonsensicalSparrow35>

however for the remote file it always creates the name with the following pattern:

{filename_prefix}checkpoint{n}.pt

..

Is this the main issue?
Notice that the model name (i.e. the entry on the Task itself) is not directly connected with the stored file name on the target file server (or S3)

Hi IrritableGiraffe81
PipelineDecorator.debug_pipeline() runs everything as regular python functions, but "PipelineDecorator.run_locally()" is actually sumulating all the steps on the same local machine (so that it is easier to debug the "real" pipeline running on multiple machines)
What I think is happening is that the casting of the arguments passed to the component fail.
Basically the type hints are currently ignored (we are working on using them for casting in the next version)
but righ...

You mean to add these two to the model when deploying?

    │   ├── model_NVIDIA_GeForce_RTX_3080.plan
    │   └── model_Tesla_T4.plan

Notice the preprocess.py is Not running on the GPU instance, it is running on a CPU instance (technically not the same machine)

No

Hi RipeAnt6

What would be the best way to add another model from another project say C to the same triton server serving the previous model?

You can add multiple call to cleaml-serving , each one with a new endpoint and a new project/model to watch, then when you launch it it will setup all endpoints on a single Triton server (the model optimization loading is taken care by Triton anyhow)