Here is the kernprofiler results from putting @profile before dataset.get:
It took 37s to access a dataset that must be in local cache since I use the same datset multiple times. Dataset size is a few mb stored in azure data storage, is this normal time duration? (had to cut the report because of message size limitation)
Total time: 37.5027 s
File: /home/francisco/miniconda3/envs/py310/lib/python3.10/site-packages/clearml/datasets/dataset.py
Function: get at line 1559

Line # Hits ...

Thanks for the reply but now I am trying to submit new tasks including getting datasets and still takes ~5min (excluding running my code), any idea what could be happening/how to debug?

Ok, the files are 20mb or less. I will try to profile the code and I will get back to you with the results. Never had issues with the internet stability so it shouldn't be that.