Test custom pg schema - #457

DZakh · 2025-02-13T12:36:21Z

No description provided.

JonoPrest

Thanks!

JonoPrest · 2025-02-14T08:39:28Z

I think we need to add the schema name to the persisted state to ensure envio dev reruns codegen if it changes

DZakh · 2025-02-18T14:45:42Z

-        //TODO: put the start script in the generated package.json
-        //and run from there.
-        if should_use_raw_events_worker {
-            args.push("--");
-            args.push("--sync-from-raw-events");
-        }


Removed redundant code

DZakh · 2025-02-18T14:47:37Z

+    let sql = Db.sql
+    let needsRunUpMigrations = await sql->Migrations.needsRunUpMigrations
+    if needsRunUpMigrations {
+      let _ = await Migrations.runUpMigrations(~shouldExit=false)
+    }


Added a run up migration here when a db schema doesn't exist

DZakh · 2025-02-18T14:48:50Z

+        | Custom(name) => `${Env.Db.publicSchema}.${name}`
+        | _ => fieldType :> string
+        }}${isArray ? "[]" : ""}${switch defaultValue {


Had to add schema name here as well

DZakh · 2025-02-18T14:50:32Z

+          }
+          "configuration": {
+            // Otherwise the entity in gql will be prefixed with the schema name (when it's not public)
+            "custom_name": tableName


Remove the prefix schema and optimise multiple requests into one.

JonoPrest · 2025-02-20T12:42:55Z

      {
-        `"${fieldName}" ${(fieldType :> string)}${isArray ? "[]" : ""}${switch defaultValue {
+        `"${fieldName}" ${switch fieldType {
+          | Custom(name) if !(name->Js.String2.startsWith("NUMERIC(")) => `"${Env.Db.publicSchema}".${name}`


This seems a bit fragile and I'm not following. When does custom start with "NUMERIC(" ?

I kind of think we should rather separate these into two variant constructors. Maybe Enum and Custom for eg.

Numeric is for precision. This is actually should be a separate constructor. I just don't want to spend time on this now. And besides the Numeric custom is used only for enums

Ok cool 👍🏼 maybe just add a comment explaining then.

JonoPrest · 2025-02-20T12:46:10Z

+    `DO $$ 
+    BEGIN
+      ${reset ? `DROP SCHEMA IF EXISTS ${Env.Db.publicSchema} CASCADE;` : ``}
+      IF NOT EXISTS(SELECT 1 FROM information_schema.schemata WHERE schema_name = '${Env.Db.publicSchema}') THEN
+        CREATE SCHEMA ${Env.Db.publicSchema};
+        GRANT ALL ON SCHEMA ${Env.Db.publicSchema} TO postgres;
+        GRANT ALL ON SCHEMA ${Env.Db.publicSchema} TO public;
+      END IF;
+    END $$;`,
+  )


Why does any of this run on the non reset case? I see it only drops in this case but what about the rest?

For example, you run pnpm start with a different schema. In this case it won't trigger the down migration (since pnpm start doesn't have the logic), but it still should create the schema if it's missing, so it's able to work correctly.

Ah got you 👍🏼

JonoPrest · 2025-02-20T12:49:33Z

-  | _ => Failure
-  }
-
-  process->exit(exitCode)


It looks like we no longer call process.exit with exit code. This could be a problem for the hosted service

We call it in runUpMigrations, but the first await sql->unsafe( actually don't have it. Should I add it back? Won't it exit with 1 automatically if the query fails?

All of the other potential failures are caught and only returned in the exit code. We definitely need the exit code to work accurately for the hosted service so I would suggest we keep the same logic.

JonoPrest · 2025-02-20T12:56:31Z

+  let tableNames =
+    [Db.allStaticTables, Db.allEntityTables]
+    ->Belt.Array.concatMany
+    ->Js.Array2.map(({tableName}) => tableName)
+  await trackTables(~tableNames)


I guess once the tables are all tracked the relationships and permisions can also be setup concurrently as well?

JonoPrest · 2025-02-20T12:58:10Z

  TablesStatic.PersistedState.table,
  TablesStatic.EndOfBlockRangeScannedData.table,
  TablesStatic.RawEvents.table,
-  TablesStatic.DynamicContractRegistry.table,


Is this only affecting hasura visibility? Even so, Could be a breaking change unfortunately

It's not, since the table is a part of Entity tables. So this was duplicated here.

Oh I see 👍🏼

JonoPrest · 2025-02-20T13:01:48Z

+    SELECT 1 
+    FROM pg_type 
+    WHERE typname = '${name->Js.String2.toLowerCase}' 
+    AND typnamespace = (SELECT oid FROM pg_namespace WHERE nspname = '${Env.Db.publicSchema}')


Wow, devils here hey, was this caught by a test? :)

JonoPrest · 2025-02-20T13:03:13Z

-      IF EXISTS(SELECT 1 FROM information_schema.schemata WHERE schema_name = '${Env.Db.publicSchema}') THEN
-        DROP SCHEMA ${Env.Db.publicSchema} CASCADE;
-      END IF;
+      DROP SCHEMA IF EXISTS ${Env.Db.publicSchema} CASCADE;


Does this work if it doesn't previously exist?

Yes, it doesn't do anything in this case.

JonoPrest approved these changes Feb 13, 2025

View reviewed changes

DZakh enabled auto-merge (squash) February 13, 2025 15:14

DZakh added 3 commits February 13, 2025 17:41

Test custom pg schema

a56d24f

Fix test

884454c

Fix custom schema test

17acae6

DZakh force-pushed the integration-testing-add-custom-schema branch from be58e4a to 17acae6 Compare February 13, 2025 15:41

DZakh added 3 commits February 18, 2025 13:02

Track tables in one request

542f098

Create a new schema on indexer start

9cef12d

Fix table tracking and enum creation

653b871

DZakh commented Feb 18, 2025

View reviewed changes

DZakh added 7 commits February 18, 2025 19:17

Fixes

c992499

Fix enum and table creation with custom schema

abe0788

Sync chains

5d97d1f

Sync chains

a511bf3

Fix chains order

a1d02f6

Revert old test script

5cd457c

Sync chains

b79e146

DZakh merged commit 32d71fe into main Feb 20, 2025

DZakh deleted the integration-testing-add-custom-schema branch February 20, 2025 11:15

JonoPrest reviewed Feb 20, 2025

View reviewed changes

Uh oh!

Conversation

DZakh commented Feb 13, 2025

Uh oh!

JonoPrest left a comment

Choose a reason for hiding this comment

Uh oh!

JonoPrest commented Feb 14, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants