Well, the data regarding reuse is usually stored in the home folder under .clearml/.session.json I think
okie so this works only if jobs run in parallel
first job create new task id
second job (initiated immediately after first job) do the reuse properly
if I wait for first job to finish - then run again new second job with same name, it will not do reuse
is this expected?
didn't do that test
I usually wait for first job to finish before I start new one
Another question - is there any specific reason for wanting to reuse the task?
and are there times when you try to run it in parallel?
from clearml import Task task = Task.init(project_name="Inbar2022/LanguageFactoryDanish/lions_test", task_name="lions3")python main.py --cuda --epoch 1
not really - I can try to run these in parallel
tried it from single workstation, but I get same unexpected behavior
same project/task names, same workstation running the job, anything else I should check to confirm those are identical jobs?
every new identical job start with ClearML Task: created new task id=...
the behaviour PleasantOwl46 the overwrite behaviour is only if you run it again on the same workstation - the SDK stored a cache and will overwrite if all details are identical and only if less then 72 hours have passed (see sdk.development.task_reuse_time_window_in_hours configuration setting)
same basic job not gets overwritten, but created new one every time
got it, I don't really understand why it happens, quite certain I didn't see this in the past
SuccessfulKoala55 where does the SDK stores the cache?
we have an cluster with shared storage so all computes nodes that is running the jobs has same storage
should I assume it will use the cache and overwrites identical jobs?
trying to reproduce this but still every new and same jobs gets new task ID
I have another instance with clearml-server 1.7 and I got same behavior
as I missing anything? I was under the assumption that jobs with same project/task names should be overwritten and not duplicated
But it will also consider the file name from which this was run