[Urgent!!!!!!] worker will report cross slot error when working with redis cluster

[steven-zou]

Apr 11 22:53:03 172.18.0.1 jobservice[971]: message repeated 439 times: [ ERROR: worker.fetch - CROSSSLOT Keys in request don't hash to the same slot]
Apr 11 22:53:03 172.18.0.1 jobservice[971]: ERROR: requeuer.process - CROSSSLOT Keys in request don't hash to the same slot
Apr 11 22:53:03 172.18.0.1 jobservice[971]: ERROR: requeuer.process - CROSSSLOT Keys in request don't hash to the same slot
Apr 11 22:53:03 172.18.0.1 jobservice[971]: ERROR: worker.fetch - CROSSSLOT Keys in request don't hash to the same slot

https://stackoverflow.com/questions/38042629/redis-cross-slot-error

NOTES: jobservice is our component name.

[steven-zou]

@shdunning any comments?

[shdunning]

I suspect this has something to do with the Lua scripts referencing keys that aren't defined on that node where the Lua script is running -- like our Lua script for fetching the next job, which requires 5 keys: (1) job queue (2) in progress queue (3) pause key (4) lockKey (5) lock info key (6) set concurrency key.

We'll likely need to investigate how to make these Lua scripts cluster-safe.

[shdunning]

https://redis.io/topics/cluster-spec#keys-hash-tags

[austintaylor]

@steven-zou What commit are you running? I believe we were constructing keys in a lua script at one point, which would cause this error, but we fixed this in #52

[steven-zou]

Thanks, @austintaylor. After rough checking, there might be a mistake when importing work library into our project with dep. Probably, an older version is imported. I'll do more verification to make sure if the above issue existing.

[shdunning]

@steven-zou if it's any help, we've been having better luck with https://github.com/kardianos/govendor instead of dep.

[stefanoschrs]

Have the same issue on 0.5.1, when using an elasticache instance I get the

ERROR: requeuer.process - CROSSSLOT Keys in request don't hash to the same slot
ERROR: worker.fetch - CROSSSLOT Keys in request don't hash to the same slot

[sebcoetzee]

@stefanoschrs any luck fixing this on 0.5.1?

[stefanoschrs]

Nope, nothing..

[sebcoetzee]

@stefanoschrs I ended up switching out the clustered Redis instance for a normal one to get around this.

[stefanoschrs]

That's what I did also, but it's not a solution..

[steven-zou]

Yes, check with v0.5.1, the issue still is there. Any bits of advice? @shdunning @austintaylor

[steven-zou]

Just push up:

@shdunning @austintaylor

[shdunning]

@steven-zou i would need to do some digging. this (normally) occurs if we are dynamically referencing a key in the lua scripts instead of explicitly passing them in via the KEYS argv. I'll try to spend some time this week digging into the lua scripts to see if I can track down where this might be occuring.