offset = 0
limit = 500
all_data = []

while True:
    params = {
        'offset': offset,
        'limit': limit
    }
    
    response = requests.get(url, headers=headers, params=params,verify=False)
    
    data = response.json()
    projects = data['data']['projects']

    print(f"pulled {len(projects)} projects.")

    if len(projects) == 0:
        print("no project found - exiting ...")
        break    
    
    all_data.extend(projects)
    offset += limit

print(f"pulled {len(projects)} projects, total of: {len(all_data)}")

it keeps pulling 500 and looping
who can help me find my bug 🙂

I know the 500 limit and using it

but my while loop keeps pulling the same 500 ... and running endless

trying to use projects.get_all to pull all my projects into single file
and there are more then 500 ...

the way I do pagination is wrong
@<1523701070390366208:profile|CostlyOstrich36> might throw some champions tip over here 🙂

Why are you using query parameters? The documentation shows that you should be using the request body for all that. None

hmmm that's not it, same endless loop when using body

when running in debug and watch the values I get

    data = response.json()
    projects = data['data']['projects']
    all_data.extend(projects)

in each loop iteration
projects are same 500 values
all_data gets append for same 500 values in endless loop

I have bug in my code and can't find where just yet

Are you using page_size and page keys? page should be incremented by 1 regardless of page size, then just check if the response contains less than those 500 responses, then you can break out.

I think this is the right approach, let me have a deeper look
thanks @<1724235687256920064:profile|LonelyFly9>

ohhh severe error here 🙂
I was mixed between other API I worked on .. and did not read carefully the right flag

simple adding page to the body did the work
thanks again @<1724235687256920064:profile|LonelyFly9>

OK I got everything to work
I think this script can be useful to other people and will be happy to share

@<1523701070390366208:profile|CostlyOstrich36> is there some repo I fork and contribute?

Hi @<1523701842515595264:profile|PleasantOwl46> , I think you can add a PR here - None