Record Track Downloads via Entity Manager #9103

rickyrombo · 2024-07-11T21:52:27Z

Description

Quick EntityManager implementation of recording track downloads across the network.

Ideal solution would be to do this from content or discovery rather than the client, but this is a quick and dirty way of getting it done fast.

Note the commit messages - might help going commit by commit.

Things I would change given more time:

Send the transaction from Discovery or Content rather than relying on the client
Include the download count in track aggregates, or generally have some sort of endpoint to expose the counts
Notifs triggered on new rows for downloads

How Has This Been Tested?

Tested locally w/ download all on local dev client

changeset-bot · 2024-07-11T21:52:32Z

🦋 Changeset detected

Latest commit: 71159a7

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages

Name	Type
@audius/sdk	Patch
@audius/sp-actions	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

rickyrombo · 2024-07-11T21:53:24Z

packages/discovery-provider/plugins/pedalboard/apps/relay/src/walletManager.ts

 import { logger } from "./logger";
 import { Wallet, providers } from "ethers";

+const localDevWallets = [


these all came from POA ganache

rickyrombo · 2024-07-11T21:54:15Z

packages/libs/src/sdk/services/EntityManager/types.ts

   * The numeric user id
   */
-  userId: number
+  userId?: number


Unsure if downloads are gated on being signed in at a protocol level, so made user ID optional

sliptype · 2024-07-11T22:07:43Z

this is AWESOME! I'm imagining notifications when someone downloads your track 👀

alecsavvy

relay changes LGTM!

alecsavvy · 2024-07-11T22:34:00Z

packages/discovery-provider/plugins/pedalboard/apps/relay/src/middleware/validator.ts

    if (isUserDeactivate(isDeactivated, encodedABI)) {
      logger.info("user deactivation")
-      createOrDeactivate = true
+      isAnonymousAllowed = true


thanks for making this flag more general

packages/discovery-provider/src/tasks/entity_manager/entities/track.py

audius-infra · 2024-07-11T23:10:35Z

Preview this change https://demo.audius.co/mjp-track-recorded-downloads

isaacsolo

awesome! would be nice to have access validation for recording a download, more reason to move this to content in the future

raymondjacobson · 2024-07-16T22:25:04Z

packages/discovery-provider/ddl/migrations/0080_track_downloads.sql

+    track_id INTEGER NOT NULL,
+    user_id INTEGER,
+    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    CONSTRAINT track_downloads_blocknumber_fkey FOREIGN KEY (blocknumber) REFERENCES blocks("number") ON DELETE CASCADE,


do we have these constraints elsewhere now? especially the on delete cascade part. just curious if this is in line with the current strategy for these tables

Yeah we do it for eg track routes i believe and others - cascade deletes helps w/ reverts and replays. CC: @isaacsolo to confirm I set this up correctly

raymondjacobson · 2024-07-16T22:27:18Z

packages/discovery-provider/src/tasks/entity_manager/entities/track.py



+def download_track(params: ManageEntityParameters):
+    validate_track_tx(params)


do we need to validate that the params.user_id is the signer too?

Good q. validate_track_tx calls validate_signer, so this does. Do we need that? Arguably no. I designed everything else assuming that there isn't a user here, so if I wanted to be consistent I would need to make sure that a user not being found is allowed.

HOWEVER, in the client, we don't allow downloads unless signed in. So I think my assumption was off - despite the download API not requiring a user (iirc), the client does. Maybe the right thing to do is require an account for the download API and then keeping everything requiring a user?

chatted offline. kinda whatever you want to do here. require user is fine

packages/web/src/common/store/social/tracks/sagas.ts

audius-infra · 2024-07-16T23:57:29Z

Preview this change https://demo.audius.co/mjp-track-recorded-downloads

dharit-tan · 2024-07-24T17:34:09Z

packages/discovery-provider/src/tasks/entity_manager/entities/track.py

+
+    track_id = params.entity_id
+    existing_track = params.existing_records["Track"][track_id]
+    if (


i don't quite understand this - wouldn't track_id never change on an update?
and how is new_records["Track"][track_id][-1] an array...
maybe the comment wording is throwing me off.

this is all copy-pasted, but I think I understand it?

track_id isn't changing, but the track might be (ie if a couple of edits come in the same block). We want to work with the most recent changed track or we might have problems.

existing_records is a dict mapping ids => records
new_records is a dict mapping ids => record lists

dharit-tan

lgtm

…wnloads

gitguardian · 2024-08-09T00:03:23Z

⚠️ GitGuardian has uncovered 8 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secrets in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
11648676	Triggered	Generic High Entropy Secret	`9beb0ed`	core/infra/dev_config/discovery-one.docker.env	View secret
11648676	Triggered	Generic High Entropy Secret	`9beb0ed`	core/infra/dev_config/discovery-one.env	View secret
11648678	Triggered	Generic High Entropy Secret	`9beb0ed`	core/infra/dev_config/content-three.env	View secret
11648678	Triggered	Generic High Entropy Secret	`9beb0ed`	core/infra/dev_config/content-three.docker.env	View secret
11648679	Triggered	Generic High Entropy Secret	`9beb0ed`	core/infra/dev_config/content-two.docker.env	View secret
11648679	Triggered	Generic High Entropy Secret	`9beb0ed`	core/infra/dev_config/content-two.env	View secret
11648680	Triggered	Generic High Entropy Secret	`9beb0ed`	core/infra/dev_config/content-one.env	View secret
11648680	Triggered	Generic High Entropy Secret	`9beb0ed`	core/infra/dev_config/content-one.docker.env	View secret

🛠 Guidelines to remediate hardcoded secrets

Understand the implications of revoking this secret by investigating where it is used in your code.
Replace and store your secrets safely. Learn here the best practices.
Revoke and rotate these secrets.
If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

following these best practices for managing and storing secrets including API keys and other credentials
install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}

audius-infra · 2024-08-09T00:22:19Z

Preview this change https://demo.audius.co/mjp-track-recorded-downloads

[876e321] [QA-1459] Fix 0087_fix_slugs_with_slash.sql migration (#9411) Raymond Jacobson [c34cd83] Record Track Downloads via Entity Manager (#9103) Marcus Pasell [d38870a] Small repairer clean up (#9405) Michelle Brier [8cd3d6c] Payment router exception for when no geo_data found in memo (#9404) Reed [53bf3e2] [PAY-3321][PAY-3333] Add network cut feature flag and update staking bridge and payment router addresses (#9393) Saliou Diallo

[7121301] Set is_custom_musical_key (#9418) Sebastian Klingler [ee99fd4] add is_custom_musical_key (#9416) Michelle Brier [f30a637] [PAY-3332] Remixers users endpoint (#9401) Reed [f2eabb1] [QA-1491] Fix balance polling bug (#9402) Raymond Jacobson [c5e7348] Fix lint (#9412) Raymond Jacobson [5598656] [QA-1356] Improve notification UI (#9390) Raymond Jacobson [dcb74eb] Fix useClickOutside (#9409) Andrew Mendelsohn [975b630] Add better error messages for Claimable Tokens Program (#9369) Marcus Pasell [17904fb] [PAY-3335] Add priority fees to SDK Solana usages (#9399) Marcus Pasell [c34cd83] Record Track Downloads via Entity Manager (#9103) Marcus Pasell [e59dfc8] [QA-1357] Remove excessive polling and amplitude events in unlock (#9391) Raymond Jacobson [26b2fc9] [QA-1496] fix unsafe read of audio analysis field (#9406) Randy Schott [abc1b37] [C-4934] Search bar fixes (#9403) Sebastian Klingler [d525a1b] [C-4930, C-4931] Add IconText, ArtistPick, and Identifier components to harmony and harmony native (#9400) Kyle Shanks [53bf3e2] [PAY-3321][PAY-3333] Add network cut feature flag and update staking bridge and payment router addresses (#9393) Saliou Diallo [4d0fbc2] [PAY-3294] Hide save/repost for locked albums on mobile (#9333) Reed [7e3a75c] [C-4936] FilterButton fixes (#9397) Sebastian Klingler [517251d] [PAY-3286] Upload/edit confirmation drawer & modal rework (#9326) Raymond Jacobson [dce9903] [C-4920] Only show bpm decimals if necessary (#9398) Sebastian Klingler [d6dbab0] [C-4937] Adding some pagination buttons (#9394) JD Francis [f183f7e] [PAY-3336] fix isPlaying logic for details screens (#9395) Randy Schott [5ed523f] Add comment actions: edit, delete, react (#9372) Isaac Solo [88e24b5] [C 4922] Update mobile filter button (#9385) Dylan Jeffers [901d811] [C-4923] Persist verified filter & general cleanup (#9384) Sebastian Klingler [bab49d6] Add Pin icon and Timestamp component to harmony and harmony-native (#9383) Kyle Shanks [53a1a32] [QA-1485] Redir to signup/in on direct /feed link (#9373) Raymond Jacobson [c618eb9] Improve useClickOutside (#9377) Sebastian Klingler [a97deb8] [PAY-2267] Update stage payment router and program addresses (#9387) Raymond Jacobson [def034f] [QA-1446] Fix popup alignment (#9389) Raymond Jacobson [c86bc70] Add priority fees to purchases (#9392) Marcus Pasell [66f1a0e] [PAY-3314] Track select dropdown for purchase/remix targeting (#9386) Andrew Mendelsohn [5fe7681] Comments Read/Create E2E - Changes to DB/Discovery/SDK/Desktop UI (#9319) JD Francis [1ccfcd4] [PAY-3331] Purchase/sale endpoint filtering by content id (#9376) Reed [fc59a10] [PAY-3330] Fix spacing on RepostsFavoritesStats (#9375) Raymond Jacobson [7712005] Fix short trending bug (#9378) Raymond Jacobson [196fdba] [QA-1114] Include menu items even if buttons are present (#9381) Raymond Jacobson [8e6b1e8] [QA-1486] Fix sign up select artists screen (#9371) Sebastian Klingler [34d4417] [PAY-3313] Targeted DMs CTA and modal (#9364) Andrew Mendelsohn [3a19003] [C-4921] Fix search v2 navigating back from profile (#9370) Sebastian Klingler [d266394] [C-4918] Improve harmony web FilterButton (#9366) Dylan Jeffers [4ef81a7] Bump mobile versions to 110 (#9367) Dylan Jeffers

rickyrombo added 6 commits July 11, 2024 14:45

Add track_downloads table

0c0f342

Index track download entity manager txs

56f0b37

Allow track download entity manager txs through relay

09a4847

Add tracks.recordTrackDownload to SDK

182d953

Record track downloads from the client

d942a61

Fix transaction underpriced errors in local dev

ddc6dd3

rickyrombo requested review from alecsavvy, dharit-tan, isaacsolo and raymondjacobson July 11, 2024 21:52

pull-request-size bot added the size/L label Jul 11, 2024

rickyrombo commented Jul 11, 2024

View reviewed changes

alecsavvy approved these changes Jul 11, 2024

View reviewed changes

alecsavvy reviewed Jul 11, 2024

View reviewed changes

rickyrombo added 2 commits July 11, 2024 15:37

format

ab08cff

fix types

ba49658

alecsavvy reviewed Jul 11, 2024

View reviewed changes

packages/discovery-provider/src/tasks/entity_manager/entities/track.py Show resolved Hide resolved

ensure track being downloaded exists

4315bfd

isaacsolo approved these changes Jul 16, 2024

View reviewed changes

raymondjacobson approved these changes Jul 16, 2024

View reviewed changes

console error

3f40259

dharit-tan reviewed Jul 24, 2024

View reviewed changes

dharit-tan approved these changes Jul 24, 2024

View reviewed changes

Merge remote-tracking branch 'origin/main' into mjp-track-recorded-do…

9beb0ed

…wnloads

changeset

71159a7

rickyrombo enabled auto-merge (squash) August 9, 2024 00:05

rickyrombo merged commit c34cd83 into main Aug 9, 2024

rickyrombo deleted the mjp-track-recorded-downloads branch August 9, 2024 00:47



		def download_track(params: ManageEntityParameters):
		validate_track_tx(params)

Record Track Downloads via Entity Manager #9103

Record Track Downloads via Entity Manager #9103

Uh oh!

Conversation

rickyrombo commented Jul 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

How Has This Been Tested?

Uh oh!

changeset-bot bot commented Jul 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sliptype commented Jul 11, 2024

Uh oh!

alecsavvy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

audius-infra commented Jul 11, 2024

Uh oh!

isaacsolo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

audius-infra commented Jul 16, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dharit-tan left a comment

Choose a reason for hiding this comment

Uh oh!

gitguardian bot commented Aug 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ GitGuardian has uncovered 8 secrets following the scan of your pull request.

Uh oh!

audius-infra commented Aug 9, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

rickyrombo commented Jul 11, 2024 •

edited

Loading

changeset-bot bot commented Jul 11, 2024 •

edited

Loading

gitguardian bot commented Aug 9, 2024 •

edited

Loading