Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Does Anyone Have

does anyone have working example of how to properly use pagination with ClearML REST API?
I'm trying to use projects.get_all to get an json with all my project, but when I run in a while loop, I keep getting the same first 500 results (which is the offset limit)

  
  
Posted 2 months ago
Votes Newest

Answers 12


Hi @<1523701842515595264:profile|PleasantOwl46> , I think you can add a PR here - None

  
  
Posted 2 months ago

OK I got everything to work
I think this script can be useful to other people and will be happy to share

@<1523701070390366208:profile|CostlyOstrich36> is there some repo I fork and contribute?

  
  
Posted 2 months ago

ohhh severe error here 🙂
I was mixed between other API I worked on .. and did not read carefully the right flag

simple adding page to the body did the work
thanks again @<1724235687256920064:profile|LonelyFly9>

  
  
Posted 2 months ago

I think this is the right approach, let me have a deeper look
thanks @<1724235687256920064:profile|LonelyFly9>

  
  
Posted 2 months ago

Are you using page_size and page keys? page should be incremented by 1 regardless of page size, then just check if the response contains less than those 500 responses, then you can break out.

  
  
Posted 2 months ago

when running in debug and watch the values I get

    data = response.json()
    projects = data['data']['projects']
    all_data.extend(projects)

in each loop iteration
projects are same 500 values
all_data gets append for same 500 values in endless loop

I have bug in my code and can't find where just yet

  
  
Posted 2 months ago

hmmm that's not it, same endless loop when using body

  
  
Posted 2 months ago

Why are you using query parameters? The documentation shows that you should be using the request body for all that. None

  
  
Posted 2 months ago

the way I do pagination is wrong
@<1523701070390366208:profile|CostlyOstrich36> might throw some champions tip over here 🙂

  
  
Posted 2 months ago

trying to use projects.get_all to pull all my projects into single file
and there are more then 500 ...

  
  
Posted 2 months ago

I know the 500 limit and using it

but my while loop keeps pulling the same 500 ... and running endless

  
  
Posted 2 months ago

offset = 0
limit = 500
all_data = []

while True:
    params = {
        'offset': offset,
        'limit': limit
    }
    
    response = requests.get(url, headers=headers, params=params,verify=False)
    
    data = response.json()
    projects = data['data']['projects']

    print(f"pulled {len(projects)} projects.")

    if len(projects) == 0:
        print("no project found - exiting ...")
        break    
    
    all_data.extend(projects)
    offset += limit

print(f"pulled {len(projects)} projects, total of: {len(all_data)}")

it keeps pulling 500 and looping
who can help me find my bug 🙂

  
  
Posted 2 months ago