I was having the exact same problem. And I don't think this is a proper workaround answer, but I have found that using the iFi Go Link Max does not cause this issue. 😅
* Edit:
I just encountered a (fortunately low volume) noise issue while playing Apple Music (with loseless music) that was resolved by unplugging and plugging in again.