Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
I Have A Problem With Clearml Today, It Seems To Be Waiting For Something (Not Sure For What) And Writing To The Console "Clearml.Model - Info - Selected Model Id: ....." Every Few Seconds, Without Making Progress In Training. Do You Know What Can Be The

I have a problem with clearml today, it seems to be waiting for something (not sure for what) and writing to the console "clearml.model - INFO - Selected model id: ....." every few seconds, without making progress in training. Do you know what can be the reason?

  
  
Posted 11 months ago
Votes Newest

Answers 20


@<1523703472304689152:profile|UpsetTurkey67> great, thank you! We are taking a look

  
  
Posted 11 months ago

@<1523703472304689152:profile|UpsetTurkey67> can you expand on how exactly you're using joblib?

  
  
Posted 11 months ago

@<1523701435869433856:profile|SmugDolphin23> I have checked that when setting auto_connect_frameworks=False it works, but disabling just joblib is not enough.

  
  
Posted 11 months ago

@<1523703472304689152:profile|UpsetTurkey67> can you please open a Github issue as well, so we can better track this one?

  
  
Posted 11 months ago

@<1523701435869433856:profile|SmugDolphin23> None

  
  
Posted 11 months ago

@<1523701435869433856:profile|SmugDolphin23> it took some time, but I was able to cut 90% of the code, just dataloading remains and the problem persists (which is fortunate, as it makes it easy to replicate). Please have a look.

  
  
Posted 11 months ago

@<1523701435869433856:profile|SmugDolphin23> let me know if you need any help in reproducing

  
  
Posted 11 months ago

@<1523701087100473344:profile|SuccessfulKoala55> FYI

  
  
Posted 11 months ago

to avoid loading and cropping a big image

  
  
Posted 11 months ago

@<1523701435869433856:profile|SmugDolphin23> it did not help, shall I create smallest example when it does not work and paste it here?

  
  
Posted 11 months ago

@<1523703472304689152:profile|UpsetTurkey67> It would be great if you could write a script we could use to reproduce

  
  
Posted 11 months ago

@<1523701087100473344:profile|SuccessfulKoala55> any ideas what can be the cause?

  
  
Posted 11 months ago

but I do agree that some kind of autoconnect may be the issue

  
  
Posted 11 months ago

The problem started appearing when I started to use joblib with a simple memory caching mechanism.

  
  
Posted 11 months ago

@<1523701087100473344:profile|SuccessfulKoala55> I am using it as follows:

after calling clearml.Task.init() I create an object:

cache = Cache('/scidata/marek/diskcache')

and then in the loading function I do:

   if cache_arg in load_and_crop.cache:                                                                                                                                                                     
         return load_and_crop.cache[cache_arg]             
  
  
Posted 11 months ago

@<1523701435869433856:profile|SmugDolphin23> will send later today

  
  
Posted 11 months ago

Hi @<1523703472304689152:profile|UpsetTurkey67> ! What if in Task.init you set auto_connect_frameworks={"joblib": False} . Do you still have this issue?

  
  
Posted 11 months ago

@<1523701087100473344:profile|SuccessfulKoala55> I have the same problem with diskcache

  
  
Posted 11 months ago

I am seeing such warnings clearml.model - WARNING - 9 model found when searching

  
  
Posted 11 months ago

BTW, why do you need the cache?

  
  
Posted 11 months ago
564 Views
20 Answers
11 months ago
11 months ago
Tags