BTW:
Just making sure, 74 was not supposed to be the last checkpoint (in other words it is not stuck on leaving the training process, but actually in the middle)

It will store the entire content of the file, then you can edit it in the UI, and in remote it will return a new local copy of the file (based on the data in the UI) for you to read.

Instead you can do: TRAINS_WORKER_NAME = "trains-agent":$DYNAMIC_INSTANCE_ID
Then the Worker ID will running instance appended to the worker name. This means that even if you use the same $DYNAMIC_INSTANCE_ID twice, you will not have two agent registering on the same name.

Also what do you have in the "Configuration" section of the serving inference Task?

So clearml server already contains an authentication layer (JWT Token), and you do have a full user management on top:
https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_config#web-login-authentication
Basically what I'm saying if you add httpS on top of the communication, and only open the 3 ports, you should be good to go. Now if you really need SSO (AD included) for user login etc, unfortunately this is not part of the open source, but I know they have it in the scale/ent...

right now I can't figure out how to get the session in order to get the notebook path

you mean the code that fires "HTTPConnectionPool" ?

Hi ScaryLeopard77
You can probably do:
Task.init(...,continue_last_task='task_id_here')This will continue a previously executed Task and log both steps in the same place.
Does that help?
BTW: you can also of course manually report to any Task as it is still running with:
aux_task = Task.get_task(task_id_here) aux_task.get_logger().report_scalar(...)

Guys, any chance you can verify the RC solves the issue?
pip install clearml==1.0.2rc0

WickedGoat98 until the next RC release (should not take long) this will solve it:
df = pd.concat([tickerDf.Close, tickerDf_Change.Close_pcent], axis=1) df = df[1:] df.index = df.index.astype(str) setattr(df, 'ticker', args.symbol)Basically removing the nan and converting the datetime to string representation (so plotly.js likes it)

is it possible to change an existing model's URL?

Edit the DBs ... That's basically the only way 😞

Hi TrickyRaccoon92 , TB is automatically collected and converted into data stored on the system The UI uses plotly to display the data itself (on your web browser).
You still have the original TB protobuf file, if you want to dive deeper and debug the data (it is not automatically uploaded, but some users do upload it as additional artifact on the experiment)
Make sense ?

SoreDragonfly16 . In the hyper parameters Tab, you have "parallel coordinates" (next to the "add experiment" the button saying "values" press on it and there should be " parallel coordinates")
Is that it?

EnviousStarfish54 Yes i'm not sure what happens there we will have to dive deeper, but now that you got us a code snippet to reproduce the issue it should not be very complicated to fix (I hope 🤞 )

EnviousStarfish54 following on this issue, the root cause is that dictConfig will clean All handlers if Not passed "incremental": True
conf_logging = { "incremental": True, ... }Since you pointed that Kedro is internally calling logging.config.dictConfig(conf_logging) ,
this seems like an issue with Kedro as this call will remove All logging handlers, which seems problematic. wdyt ?

My pleasure, and apologies 🙂

MoodyCentipede68 from your log

clearml-serving-triton | E0620 03:08:27.822945 41 model_repository_manager.cc:1234] failed to load 'test_model_lstm2' version 1: Invalid argument: unexpected inference output 'dense', allowed outputs are: time_distributed

This seems the main issue of triton failing to.load
Does that make sense to you? how did you configure the endpoint model?

NICE! MoodyCentipede68 this is awesome 🙂

BTW: in your code, you should probably replace
dataset_task = Task.get_task(task_id=dataset.id)with:
dataset_task = dataset._task

Hi StickyBlackbird93
Yes, this agent version is rather old ( clearml_agent v1.0.0 )
it had a bug where pytorch wheel aaarch broke the agent (by default the agent in docker mode, will use the latest stable version, but not in venv mode)
Basically upgrade to the latest clearml-agent version it should solve the issue:
pip3 install -U clearml-agemnt==1.2.3BTW for future debugging, this is the interesting part of the log (Notice it is looking for the correct pytorch based on the auto de...

You need to use tf.summary.image and not summary_ops_v2.image

Fixed on main branch (see github issue), RC later today

Image needs to be in range [0, 1] and not [0, 255] (matplotlib and tensorboard can handle either one)

Is there a code to reproduce ?

And when retrieve just this file? is it working ?
(Maybe for some reason the file is corrupted) ?

DepressedChimpanzee34 something along the lines of:
from multiprocessing.pool import ThreadPool p = ThreadPool() def get_last_metric(t): return t.get_last_scalar_metrics() task_scalars_list = p.map(get_last_metric, top_tasks) p.close()We parallelized network connection as I'm assuming the delay is fetching

You can try direct API call for all the Tasks together:
Task._query_tasks(task_ids=[IDS here], only_fields=['last_metrics'])

Hi MagnificentSeaurchin79
Could you test with the tesnorflow toy example?
https://github.com/allegroai/clearml/blob/master/examples/frameworks/tensorflow/tensorboard_toy.py

So it seems to get the "hint" from the type:
This will work
tf.summary.image('toy255', (ex * 255).astype(np.uint8), step=step, max_outputs=10)wdyt, should it actually check min/max and manually cast it ?

FrothyShark37 any chance you can share snippet to reproduce?

https://github.com/allegroai/trains-web

Hi OddShrimp85

right place to ask about clearml serving.

It is 🙂

I did not manage to get clearml serving work with my own clearml server and triton setup.

Yes it should have been updated already, apologies.
Until we manage to sync the docs, what seems to be your issue, maybe we can help here?

Perhaps this is something that can be made clearer when updating the docu?

Hmm that is a good point, let's open a git issue and explain there, then update the docs, wdyt?

Done!

Thanks

fatal: unable to find a suitable socket path; use --socket

 )

Look here https://stackoverflow.com/questions/5015732/why-do-i-get-unable-to-connect-a-socket-when-i-try-to-clone-via-a-git-url

I think that's the root cause, we should probably also add https://github.com/allegroai/trains-agent/issues/16