for some reason it's not in REST API docs, but I used
users.get_all

offset = 0
limit = 500
all_data = []

while True:
    params = {
        'offset': offset,
        'limit': limit
    }
    
    response = requests.get(url, headers=headers, params=params,verify=False)
    
    data = response.json()
    projects = data['data']['projects']

    print(f"pulled {len(projects)} projects.")

    if len(projects) == 0:
        print("no project found - exiting ...")
        break    
    
    all_data.extend(projects)
    offset += limit

print(f"pulled {le...

I know the 500 limit and using it

but my while loop keeps pulling the same 500 ... and running endless

trying to use projects.get_all to pull all my projects into single file
and there are more then 500 ...

when running in debug and watch the values I get

    data = response.json()
    projects = data['data']['projects']
    all_data.extend(projects)

in each loop iteration
projects are same 500 values
all_data gets append for same 500 values in endless loop

I have bug in my code and can't find where just yet

ohhh severe error here 🙂
I was mixed between other API I worked on .. and did not read carefully the right flag

simple adding page to the body did the work
thanks again @<1724235687256920064:profile|LonelyFly9>

the way I do pagination is wrong
@<1523701070390366208:profile|CostlyOstrich36> might throw some champions tip over here 🙂

I think this is the right approach, let me have a deeper look
thanks @<1724235687256920064:profile|LonelyFly9>

I manage clearml-server for many users
wrote a script that pulls tasks (for review - pending deletion) and limit for specific user (saw the REST can limit only by user id )
so user will need to provide the script his own user id in advance
I guess my options are:

• maintain stastic list of users (which I get from users.get_all endpoint)
• add to my script another query that will match user name to user id, before running the tasks_get.all for specific user id

okie so this works only if jobs run in parallel
first job create new task id
second job (initiated immediately after first job) do the reuse properly

if I wait for first job to finish - then run again new second job with same name, it will not do reuse

is this expected?

not really - I can try to run these in parallel

      {
        "id": "c2e33466fd9de1a0d2be8d803ad4fbed",
        "company": "d1bd92a3b039400cbafc60a7a5b1e52b",
        "name": "John Doe",
        "family_name": "John",
        "given_name": "Doe",
        "created": "2024-09-24T06:13:59.956000+00:00"
      }

this is how the object looks like

I have another instance with clearml-server 1.7 and I got same behavior
as I missing anything? I was under the assumption that jobs with same project/task names should be overwritten and not duplicated

unfortunately I couldn't fix this
the ES state in hectic, can't delete anything
clearml is still live, read-only mode, all existing indices are readable
new jobs can't write to this clearml server

same basic job not gets overwritten, but created new one every time

@<1523701070390366208:profile|CostlyOstrich36> unfortunately, this is not the behavior we are seeing
same exact issue happen tonight
on epoch number 53 ClearML were shut down, the job did not continue to epoch 54 and eventually got killed with watchdog timer

how do you know the ID?
I'm looking for this ...

yep that was my approached with no luck so far
hopefully someone from the ClearML dev team can give their input on this

got it, I don't really understand why it happens, quite certain I didn't see this in the past

SuccessfulKoala55 where does the SDK stores the cache?
we have an cluster with shared storage so all computes nodes that is running the jobs has same storage
should I assume it will use the cache and overwrites identical jobs?
trying to reproduce this but still every new and same jobs gets new task ID

yep, that's options number 2 🙂
I will implement that

api calls behaves much better
no problem to query tasks in other projects

so I think I'm in the right direction
adding verify= and pointing to my CA.pem looks like the right approach

now, how do I use it with ClearML API?

cleanup_service

for task in tasks:
    try:
      deleted_task = Task.get_task(task_id=task.id)
      print (deleted_task.name)
      deleted_task.delete(
               delete_artifacts_and_models=True,
               skip_models_used_by_other_tasks=True,
               raise_on_error=False
      )

it throw down the SSL error,...

@<1523701087100473344:profile|SuccessfulKoala55> looks OK (?)

>>> StorageHelper.get(Task._get_default_session().get_files_server_host())._container.session.verify
InsecureRequestWarning: Certificate verification is disabled! Adding certificate verification is strongly advised. See: 

True

I'm using rpm based machine, but I get your direction
put the cert in the right place for python to looks for it automatically

can I assume if it works smoothly with requests or urllib3 it will work for the ClearML API?

@<1523701435869433856:profile|SmugDolphin23> thanks for good pointers

it did not work on first attempt - requests did not validated the certs right
I have added this:

token_req = requests.get(api_server + "/auth.login", verify="<my_org_CA>", auth=(access_key, secret_key))``` 
print(token_req)

I got back

<Response [200]>

which I believe is good right?

when adding
token = token_req.json()["data"]["token"]

I got errors from json decoder, which I believe is exp...

let me dig in more and hopefully can share successful results
thanks!

the application is functional on localhost for sure

SuccessfulKoala55 any clue?

yep, again most jobs works .. the issue with when a job tries to upload artifacts to fileserver