Hi, we deploy ClearML in Azure and have used Azure Cache for Redis so far. But that's retiring and the new service is Azure Managed Redis. I've managed to connect clearml-server
and it works mostly fine except for cloning of tasks and deleting of archived tasks, where the following error pops up:
CROSSSLOT Keys in request don't hash to the same slot (context='', command='del', original-slot='2259', wrong-slot='5440', first-key='tags_d1bd92a3b039400cbafc60a7a5b1e52b_Task', violating-key='tags_d1bd92a3b039400cbafc60a7a5b1e52b_d12eae147f024490a3740fc1453b5076_Task')
This didn't happen before. The key differences between Azure Cache For Redis and Azure Managed Redis are that the former is Redis v6 and the latter is v7.
Anyone run ClearML and Redis > 6? According to any favourite AI bot the cluster slots rules enforcing got stricter so wondering if it's just an issue with moving to a later version of Redis?
If it's not Redis 6vs7, one possible related artefact: After changing the connection, clearml-server kept dying on startup with Redis Exception DB index out of range
. I had to explicitly set DB index to 0 in the config for the apiserver
and the workers
aliases. Is there by any chance some implicit - use db 0 for some stuff and db 1 for others? (this would be bit problematic because Azure Managed Redis only supports DB 0 at this point)
Despite the error popup the operation still succeeds.