This is the hpo task log with the azure-storage-blob in version 12.14.1
Hi @<1523701070390366208:profile|CostlyOstrich36> , thanks for your answer! I just updated the 'azure_storage_blob' package to the newest version and got some strange behaviour. When running the BOHB hyperparameter optimization, there is only one job executed and not stopped. I aborted the job after 3500 epochs because I set the the max_iteration_per_job parameter to 1000 and the job seems to run infinitely long. I just downgraded the package back to version 12.14.1 and everything works as before and I again get the error messages as mentioned above.
Yes, I also find that very weird... I start the hyperparameter optimization via python code using the HyperParameterOptimizer class of clearml. Which configurations are you explicitely interested in?
When comparing the logs of the two hpo tasks it seems like no logs of the subtasks are getting to the hpo task. So maybe this is the reason for the infinitely long running subtask? But what does the azure package have to do with that?
This is the log of the hpo task with the newest azure azure-storage-blob version
Hi @<1526371965655322624:profile|NuttyCamel41> , try upgrading the Azure SDK package and try running again
So your HPO job is affected by the azure_storage_blob package? How are you running HPO? Can you provide logs & configurations for two such different runs?