the way I do pagination is wrong
@<1523701070390366208:profile|CostlyOstrich36> might throw some champions tip over here 🙂

when running in debug and watch the values I get

    data = response.json()
    projects = data['data']['projects']
    all_data.extend(projects)

in each loop iteration
projects are same 500 values
all_data gets append for same 500 values in endless loop

I have bug in my code and can't find where just yet

I think this is the right approach, let me have a deeper look
thanks @<1724235687256920064:profile|LonelyFly9>

I know the 500 limit and using it

but my while loop keeps pulling the same 500 ... and running endless

trying to use projects.get_all to pull all my projects into single file
and there are more then 500 ...

hmmm that's not it, same endless loop when using body

OK I got everything to work
I think this script can be useful to other people and will be happy to share

@<1523701070390366208:profile|CostlyOstrich36> is there some repo I fork and contribute?

ohhh severe error here 🙂
I was mixed between other API I worked on .. and did not read carefully the right flag

simple adding page to the body did the work
thanks again @<1724235687256920064:profile|LonelyFly9>

Why are you using query parameters? The documentation shows that you should be using the request body for all that. None

offset = 0
limit = 500
all_data = []

while True:
    params = {
        'offset': offset,
        'limit': limit
    }
    
    response = requests.get(url, headers=headers, params=params,verify=False)
    
    data = response.json()
    projects = data['data']['projects']

    print(f"pulled {len(projects)} projects.")

    if len(projects) == 0:
        print("no project found - exiting ...")
        break    
    
    all_data.extend(projects)
    offset += limit

print(f"pulled {len(projects)} projects, total of: {len(all_data)}")

it keeps pulling 500 and looping
who can help me find my bug 🙂

Hi @<1523701842515595264:profile|PleasantOwl46> , I think you can add a PR here - None

Are you using page_size and page keys? page should be incremented by 1 regardless of page size, then just check if the response contains less than those 500 responses, then you can break out.