Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Profile picture
StrangePelican34
Moderator
6 Questions, 19 Answers
  Active since 10 January 2023
  Last activity 12 months ago

Reputation

0

Badges 1

19 × Eureka!
0 Votes
9 Answers
975 Views
0 Votes 9 Answers 975 Views
After I Finish Training A Model, I Want To Call Logger.Report_Scalars To Help Monitor Inferencing Status (We Do A Lot Of Batch) But After The Model Finishes Training, Scalars Are No Longer Accepted By The Task As It Is Considered Completed. Help!
After I finish training a model, I want to call logger.report_scalars to help monitor inferencing status (we do a lot of batch) but after the model finishes ...
one year ago
0 Votes
5 Answers
982 Views
0 Votes 5 Answers 982 Views
It Appears That When I Use Poetry, It Recreates The Environment (Doesn'T Persist The Venv) And Redownloads All The Python Packages (Doesn'T Use A Cache). Is This How The System Currently Works, Or Are There Specific Flags And Such That I Have Not Enabled
It appears that when I use poetry, it recreates the environment (doesn't persist the venv) and redownloads all the python packages (doesn't use a cache). Is ...
3 years ago
0 Votes
10 Answers
974 Views
0 Votes 10 Answers 974 Views
I Am Using Opennmt-Tf (2.18.1) And Clearml (1.1.2) For Training And Testing My Translation Models. I Am Wanting To Register The Incremental Bleu Scores And Final Test Data With Clearml (For Plotting, Comparison, Etc.), But It Is Not Working. I Cannot Fi
I am using opennmt-tf (2.18.1) and clearml (1.1.2) for training and testing my translation models. I am wanting to register the incremental bleu scores and f...
3 years ago
0 Votes
7 Answers
840 Views
0 Votes 7 Answers 840 Views
For Clearml Serving, If I Am Trying To Deploy 100 Models On A Gpu That Can Handle 5 Concurrently, But Each One Will Be Sporadically Used (Fine Tuned Models Trained For Different Customers), Can Clearml-Serving Automatically Load And Unload Models Based Up
For ClearML serving, if I am trying to deploy 100 models on a GPU that can handle 5 concurrently, but each one will be sporadically used (fine tuned models t...
one year ago
0 Votes
2 Answers
664 Views
0 Votes 2 Answers 664 Views
Is There Documentation On The Clearml Slurm Enterprise Integration?
Is there documentation on the ClearML Slurm enterprise integration?
12 months ago
0 Votes
5 Answers
1K Views
0 Votes 5 Answers 1K Views
We Had A Problem Moving From Google Drive To Bucket Storage (S3, Google Storage, Etc.) In That We Still Wanted To Be Able To Mount The Bucket As A Network Drive. We Were Able To Find A Stable, Free, Open Source, Multiplatform Way To Do This. The Instruc
We had a problem moving from Google Drive to Bucket storage (S3, Google Storage, etc.) in that we still wanted to be able to mount the bucket as a network dr...
3 years ago
0 After I Finish Training A Model, I Want To Call Logger.Report_Scalars To Help Monitor Inferencing Status (We Do A Lot Of Batch) But After The Model Finishes Training, Scalars Are No Longer Accepted By The Task As It Is Considered Completed. Help!

It seems that once the job as completed once, it doesn't accept any new report...

one year ago
0 After I Finish Training A Model, I Want To Call Logger.Report_Scalars To Help Monitor Inferencing Status (We Do A Lot Of Batch) But After The Model Finishes Training, Scalars Are No Longer Accepted By The Task As It Is Considered Completed. Help!

we are at Server: 3.18.2-1126 and pypi version 1.12.2.

one year ago
0 I Am Using Opennmt-Tf (2.18.1) And Clearml (1.1.2) For Training And Testing My Translation Models. I Am Wanting To Register The Incremental Bleu Scores And Final Test Data With Clearml (For Plotting, Comparison, Etc.), But It Is Not Working. I Cannot Fi

It worked! I added this call shortly after Task.init :
tf.summary.create_file_writer("C:/mypath/logs")

3 years ago
0 I Am Using Opennmt-Tf (2.18.1) And Clearml (1.1.2) For Training And Testing My Translation Models. I Am Wanting To Register The Incremental Bleu Scores And Final Test Data With Clearml (For Plotting, Comparison, Etc.), But It Is Not Working. I Cannot Fi

So, accordintg to the article (and the code as far as I could tell), OpenNmt-tf automatically enabled TensorBoard. That is, it auto-logs the relevant features through tf.summary ( https://www.tensorflow.org/api_docs/python/tf/summary ). This is output on the cmd line with the likes of:
` INFO:tensorflow:Evaluation result for step 9000: loss = 1.190986 ; perplexity = 3.290324 ; bleu = 63.569644
INFO:tensorflow:Step = 9100 ; steps/s = 2.17, source words/s = 28293, target words/s = 39388 ; Lea...

3 years ago
0 I Am Using Opennmt-Tf (2.18.1) And Clearml (1.1.2) For Training And Testing My Translation Models. I Am Wanting To Register The Incremental Bleu Scores And Final Test Data With Clearml (For Plotting, Comparison, Etc.), But It Is Not Working. I Cannot Fi

In Tensorflow's init .py, tensorboard appears to be initialized (including tf.summary):
` # Hook external TensorFlow modules.

Import compat before trying to import summary from tensorboard, so that

reexport_tf_summary can get compat from sys.modules. Only needed if using

lazy loading.

_current_module.compat.v2 # pylint: disable=pointless-statement
try:
from tensorboard.summary._tf import summary
_current_module.path = (
[_module_util.get_parent_dir(summary)] + _current_m...

3 years ago
0 For Clearml Serving, If I Am Trying To Deploy 100 Models On A Gpu That Can Handle 5 Concurrently, But Each One Will Be Sporadically Used (Fine Tuned Models Trained For Different Customers), Can Clearml-Serving Automatically Load And Unload Models Based Up

That is great to hear! Is there any documentation on how it works, and if it can be configured?

one year ago
0 After I Finish Training A Model, I Want To Call Logger.Report_Scalars To Help Monitor Inferencing Status (We Do A Lot Of Batch) But After The Model Finishes Training, Scalars Are No Longer Accepted By The Task As It Is Considered Completed. Help!

@<1523701205467926528:profile|AgitatedDove14> - for some reason none of those solutions are working. I am forcing "mark_started" - but it doesn't register. Models don't have the report_* endpoints and even trying with the artifact - once the job finishes, the artifact will no longer update.

one year ago
0 I Am Using Opennmt-Tf (2.18.1) And Clearml (1.1.2) For Training And Testing My Translation Models. I Am Wanting To Register The Incremental Bleu Scores And Final Test Data With Clearml (For Plotting, Comparison, Etc.), But It Is Not Working. I Cannot Fi

The only place I see subprocess being called in opennmt is to determine the batch size, but not for the primary training task.

3 years ago
0 Is There Documentation On The Clearml Slurm Enterprise Integration?

Thanks! I'll look forward to it. If there is a draft that can give a general scope and function to the integration, that would also be helpful for this phase of our project.

12 months ago
0 After I Finish Training A Model, I Want To Call Logger.Report_Scalars To Help Monitor Inferencing Status (We Do A Lot Of Batch) But After The Model Finishes Training, Scalars Are No Longer Accepted By The Task As It Is Considered Completed. Help!

I opened an issue here: None

one year ago
0 After I Finish Training A Model, I Want To Call Logger.Report_Scalars To Help Monitor Inferencing Status (We Do A Lot Of Batch) But After The Model Finishes Training, Scalars Are No Longer Accepted By The Task As It Is Considered Completed. Help!

@<1523701205467926528:profile|AgitatedDove14> - after the model_trainer.train step it is marked as complete. This is done using our own repo - None . The extra reporting steps are not added here (I am working on that locally) but it is calling the job complete.

one year ago
0 It Appears That When I Use Poetry, It Recreates The Environment (Doesn'T Persist The Venv) And Redownloads All The Python Packages (Doesn'T Use A Cache). Is This How The System Currently Works, Or Are There Specific Flags And Such That I Have Not Enabled

But with poetry running in a docker container, the cache is not persistent. Should I map the poetry cache volume to a location on the host?

3 years ago
0 It Appears That When I Use Poetry, It Recreates The Environment (Doesn'T Persist The Venv) And Redownloads All The Python Packages (Doesn'T Use A Cache). Is This How The System Currently Works, Or Are There Specific Flags And Such That I Have Not Enabled

https://github.com/allegroai/clearml-agent/issues/74

3 years ago
0 I Am Using Opennmt-Tf (2.18.1) And Clearml (1.1.2) For Training And Testing My Translation Models. I Am Wanting To Register The Incremental Bleu Scores And Final Test Data With Clearml (For Plotting, Comparison, Etc.), But It Is Not Working. I Cannot Fi

No TB (Tesnorboard) is not enabled. I just googled it and found this: https://forum.opennmt.net/t/running-tensorboard/4242 . I will try enabling TB and see if that fixes it.

3 years ago
0 For Clearml Serving, If I Am Trying To Deploy 100 Models On A Gpu That Can Handle 5 Concurrently, But Each One Will Be Sporadically Used (Fine Tuned Models Trained For Different Customers), Can Clearml-Serving Automatically Load And Unload Models Based Up

I checked Triton and found these references:

  • None
  • NoneIt appears that "they sell that" as Triton Management Service, part of None . It is possible to do through their API, but would need to be explicit. Moreover, there are likely a few different algorithms that could be us...
one year ago
0 We Had A Problem Moving From Google Drive To Bucket Storage (S3, Google Storage, Etc.) In That We Still Wanted To Be Able To Mount The Bucket As A Network Drive. We Were Able To Find A Stable, Free, Open Source, Multiplatform Way To Do This. The Instruc

At least I made S3 enough like Google Drive to make our team happy.

3 years ago
0 We Had A Problem Moving From Google Drive To Bucket Storage (S3, Google Storage, Etc.) In That We Still Wanted To Be Able To Mount The Bucket As A Network Drive. We Were Able To Find A Stable, Free, Open Source, Multiplatform Way To Do This. The Instruc

Move from gdrive to S3 bucket storage. This mounts the S3 bucket storage as a network drive. ClearML does not work easily with Google Drive.

3 years ago
0 For Clearml Serving, If I Am Trying To Deploy 100 Models On A Gpu That Can Handle 5 Concurrently, But Each One Will Be Sporadically Used (Fine Tuned Models Trained For Different Customers), Can Clearml-Serving Automatically Load And Unload Models Based Up

If ClearML does not implement this, we may have to ourselves - None .

one year ago
0 For Clearml Serving, If I Am Trying To Deploy 100 Models On A Gpu That Can Handle 5 Concurrently, But Each One Will Be Sporadically Used (Fine Tuned Models Trained For Different Customers), Can Clearml-Serving Automatically Load And Unload Models Based Up

Let's see if I understand:

  • Triton server deployments only have manual, static deployment of models for inferencing (without enterprise)
  • ClearML can load and unload models based upon usage, but has to do so from the hard drive
  • Triton server does not support saving models off to normal RAM for faster loading/unloading
  • Therefore, currently, we can deploy 100 models when only 5 can be concurrently loaded, but when they are unloaded/loaded (automatically by ClearML), it will take a few sec...
one year ago