Add timeout on lifecycler heartbeat

[anna-tran]

What this PR does:
Adds a timeout context to the lifecycler heartbeat to prevent requests to the ring from waiting indefinitely in case of a network issue.

Which issue(s) this PR fixes:
Fixes #6211

Checklist

• [ N/A ] Tests updated
• [ X ] Documentation added
• [ X ] CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[alanprot]

I think we can just add a context with timeout on the heartbeat call here?

cortex/pkg/ring/lifecycler.go

Lines 544 to 549 in f74b4cd

	func (i *Lifecycler) heartbeat() {
	i.lifecyclerMetrics.consulHeartbeats.Inc()
	if err := i.updateConsul(context.Background()); err != nil {
	level.Error(i.logger).Log("msg", "failed to write to the KV store, sleeping", "ring", i.RingName, "err", err)
	}
	}

Something like:

func (i *Lifecycler) heartbeat(ctx context.Context) {
	i.lifecyclerMetrics.consulHeartbeats.Inc()
	ctx, _ = context.WithTimeout(ctx, i.cfg.HeartbeatPeriod)
	if err := i.updateConsul(context.Background()); err != nil {
		level.Error(i.logger).Log("msg", "failed to write to the KV store, sleeping", "ring", i.RingName, "err", err)
	}
}

This would make sure we always start a new heartbeat?

Looking where the go routine got stuck, this would fix the problem:

https://github.com/golang/go/blob/7529d09a11496a77ccbffe245607fbd256200991/src/net/http/transport.go#L2734-L2738

I think if we do that we dont need any extra flag at all and this would work for all kv client

[yeya24]

Looks good. Let's update the PR title and description based on the latest change.