gRPC support for ArcadeDB#2462

oc007us · 2025-08-31T17:09:10Z

What does this PR do?

This PR introduces full gRPC support for ArcadeDB, providing a strongly-typed, high-performance API for remote access to the database.

Key highlights include:

Streaming query execution with multiple retrieval modes:
- CURSOR – existing behavior, iterating and streaming results as they are retrieved.
- MATERIALIZE_ALL – materializes all results before streaming in batches.
- PAGED – reissues queries per batch using LIMIT/SKIP for efficient paging.
Insert operations:
- BulkInsert – single request for large batch inserts.
- InsertStream – client-streaming bulk insert with server-side batching and acknowledgements.
- InsertBidirectional – full duplex insert mode with per-batch acknowledgements and commit/rollback control.
Record operations:
- Create, update, and lookup records with schema-aware property handling (including embedded and list properties).
Transaction support:
- Added TransactionContext flags for begin, commit, and rollback to coordinate transactional operations across requests.
- Enhanced type conversion utilities for safe proto ↔︎ Java object mapping, including embedded documents and collections.
Client-side API:
- Full client-side implementation in RemoteGrpcDatabase with a simple, extensible API.

Motivation

To provide a native, efficient, and type-safe remote API for ArcadeDB that goes beyond REST/HTTP performance and reliability limitations.
To support streaming use cases and bulk data ingestion for high-volume workloads.
To align ArcadeDB with modern distributed and reactive application architectures where gRPC is the standard for service-to-service communication.
To improve developer experience with generated client stubs and auto-completion.

Related issues

N/A

Additional Notes

I don't yet have a set of client-side tests. These will be provided shortly.

Checklist

mvn -DskipTests install
Create a Docker image with mvn clean package -Pdocker in the package module

gemini-code-assist

Summary of Changes

Hello @oc007us, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances ArcadeDB's remote access capabilities by integrating gRPC. The primary goal is to move beyond the performance and reliability limitations of traditional REST/HTTP, offering a native, efficient, and type-safe communication layer. This enables high-volume streaming and bulk data ingestion workloads, aligning ArcadeDB with modern distributed and reactive application architectures. The changes aim to improve overall developer experience through generated client stubs and better performance characteristics.

Highlights

gRPC Core Integration: Introduces full gRPC support for ArcadeDB, providing a strongly-typed, high-performance API for remote database access.
Advanced Query Streaming: Enables streaming query execution with multiple retrieval modes: CURSOR (existing behavior), MATERIALIZE_ALL (materializes all results before batching), and PAGED (reissues queries per batch using LIMIT/SKIP for efficient paging).
Flexible Insert Operations: Adds various insert mechanisms including BulkInsert (single request for large batches), InsertStream (client-streaming with server-side batching), and InsertBidirectional (full duplex with per-batch acknowledgements and transaction control).
Comprehensive Record & Transaction Handling: Supports create, update, and lookup operations for records with schema-aware property handling, alongside robust transaction support via TransactionContext flags for begin, commit, and rollback, and enhanced type conversion utilities.
Dedicated Client and Server Components: Includes a full client-side implementation in RemoteGrpcDatabase and server-side components (ArcadeDbGrpcService, GrpcServerPlugin) with interceptors for logging, metrics, authentication, and compression.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces comprehensive gRPC support for ArcadeDB, which is a significant and valuable addition. The implementation covers unary, streaming, and bulk operations, along with a new gRPC client. My review focuses on ensuring the new code is robust, maintainable, and consistent. I've identified several areas for improvement, including the removal of redundant code and a duplicate Protobuf definition, more consistent exception handling, and making data conversions more robust. Addressing these points will help solidify this new API as a first-class citizen in the ArcadeDB ecosystem.

gemini-code-assist · 2025-08-31T17:12:11Z

@@ -0,0 +1,281 @@
+syntax = "proto3";


This Protobuf file appears to be a duplicate and outdated version of the one located at grpc/src/main/proto/arcadedb-server.proto. Maintaining multiple versions of the API definition can lead to inconsistencies and build issues. This file should be removed to ensure there is a single source of truth for the gRPC service contract.

gemini-code-assist · 2025-08-31T17:12:11Z

+	private com.google.protobuf.Value toProtoValue(Object v) {
+	    com.google.protobuf.Value.Builder vb = com.google.protobuf.Value.newBuilder();
+	    if (v == null) return vb.setNullValue(com.google.protobuf.NullValue.NULL_VALUE).build();
+	    if (v instanceof String s)  return vb.setStringValue(s).build();
+	    if (v instanceof Boolean b) return vb.setBoolValue(b).build();
+	    if (v instanceof Number n)  return vb.setNumberValue(n.doubleValue()).build();
+	    if (v instanceof java.time.Instant t)
+	        return vb.setStringValue(t.toString()).build(); // or a Struct for Timestamp if you prefer
+	    if (v instanceof java.util.Map<?,?> m) {
+	        var sb = com.google.protobuf.Struct.newBuilder();
+	        for (var e : m.entrySet())
+	            sb.putFields(String.valueOf(e.getKey()), toProtoValue(e.getValue()));
+	        return vb.setStructValue(sb.build()).build();
+	    }
+	    if (v instanceof java.util.List<?> list) {
+	        var lb = com.google.protobuf.ListValue.newBuilder();
+	        for (Object e : list) lb.addValues(toProtoValue(e));
+	        return vb.setListValue(lb.build()).build();
+	    }
+	    return vb.setStringValue(String.valueOf(v)).build();
+	}	


The toProtoValue method converts unknown object types to a string using toString(). This can lead to silent data loss or incorrect data representation for complex types that are not explicitly handled (e.g., byte arrays, custom date/time types, etc.). Consider making this conversion more robust by either throwing an IllegalArgumentException for unsupported types or by adding explicit handling for more common types like byte[].

gemini-code-assist · 2025-08-31T17:12:12Z

+	public static class RemoteGrpcDatabaseWithCompression extends RemoteGrpcDatabase {
+
+		public RemoteGrpcDatabaseWithCompression(String server, int port, String databaseName, String userName, String userPassword,
+				ContextConfiguration configuration) {
+			super(server, port, databaseName, userName, userPassword, configuration);
+		}
+
+		@Override
+		protected ManagedChannel createChannel(String server, int port) {
+
+			return ManagedChannelBuilder.forAddress(server, port).usePlaintext() // No TLS/SSL
+					.compressorRegistry(CompressorRegistry.getDefaultInstance())
+					.decompressorRegistry(DecompressorRegistry.getDefaultInstance())
+					.maxInboundMessageSize(100 * 1024 * 1024) // 100MB max message size
+					.keepAliveTime(30, TimeUnit.SECONDS) // Keep-alive configuration
+					.keepAliveTimeout(10, TimeUnit.SECONDS).keepAliveWithoutCalls(true).build();
+		}
+	}


This inner class RemoteGrpcDatabaseWithCompression seems redundant, as there is a separate top-level class grpc-client/src/main/java/com/arcadedb/remote/grpc/RemoteGrpcDatabaseWithCompression.java that serves the same purpose. The implementation here is also different from the top-level class, which can cause confusion. This inner class should likely be removed to avoid duplication and maintain a single source of truth for this functionality.

gemini-code-assist · 2025-08-31T17:12:12Z

+	private TransactionIsolation mapIsolationLevel(Database.TRANSACTION_ISOLATION_LEVEL level) {
+		switch (level) {
+//      case READ_UNCOMMITTED:
+//        return TransactionIsolation.READ_UNCOMMITTED;
+		case READ_COMMITTED:
+			return TransactionIsolation.READ_COMMITTED;
+		case REPEATABLE_READ:
+			return TransactionIsolation.REPEATABLE_READ;
+//      case SERIALIZABLE:
+//        return TransactionIsolation.SERIALIZABLE;
+		default:
+			return TransactionIsolation.READ_COMMITTED;
+		}
+	}


The mapIsolationLevel method is missing cases for READ_UNCOMMITTED and SERIALIZABLE isolation levels, which are commented out. If these levels are intentionally unsupported over gRPC, it would be clearer to throw an UnsupportedOperationException or add a comment explaining why they are omitted. Otherwise, they should be implemented.

gemini-code-assist · 2025-08-31T17:12:12Z

@@ -0,0 +1,422 @@
+# ArcadeDB gRPC Server Configuration Examples


This file appears to be a collection of notes and examples for gRPC configuration and usage. For better readability and maintainability, consider the following improvements:

Rename to README.md: This will enable Markdown rendering, making the headers, code blocks, and lists much easier to read.

Structure and Consolidate: The content has several repeated sections (e.g., Docker commands, configuration properties). Consolidating these into clearly defined sections would make the document more useful as a reference.

robfrank · 2025-09-11T13:53:08Z

Duplicated to #2499 and merged
Thanks @oc007us !

Bumps [actions/checkout](https://github.com/actions/checkout) from 6.0.3 to 7.0.0. Release notes *Sourced from [actions/checkout's releases](https://github.com/actions/checkout/releases).* > v7.0.0 > ------ > > What's Changed > -------------- > > * block checking out fork pr for pull\_request\_target and workflow\_run by [`@aiqiaoy`](https://github.com/aiqiaoy) in [actions/checkout#2454](https://redirect.github.com/actions/checkout/pull/2454) > * Bump actions/publish-immutable-action from 0.0.3 to 0.0.4 in the minor-actions-dependencies group across 1 directory by [`@dependabot`](https://github.com/dependabot)[bot] in [actions/checkout#2458](https://redirect.github.com/actions/checkout/pull/2458) > * Bump flatted from 3.3.1 to 3.4.2 by [`@dependabot`](https://github.com/dependabot)[bot] in [actions/checkout#2460](https://redirect.github.com/actions/checkout/pull/2460) > * Bump js-yaml from 4.1.0 to 4.2.0 by [`@dependabot`](https://github.com/dependabot)[bot] in [actions/checkout#2461](https://redirect.github.com/actions/checkout/pull/2461) > * Bump `@actions/core` and `@actions/tool-cache` and Remove uuid by [`@dependabot`](https://github.com/dependabot)[bot] in [actions/checkout#2459](https://redirect.github.com/actions/checkout/pull/2459) > * upgrade module to esm and update dependencies by [`@aiqiaoy`](https://github.com/aiqiaoy) in [actions/checkout#2463](https://redirect.github.com/actions/checkout/pull/2463) > * Bump the minor-npm-dependencies group across 1 directory with 3 updates by [`@dependabot`](https://github.com/dependabot)[bot] in [actions/checkout#2462](https://redirect.github.com/actions/checkout/pull/2462) > * getting ready for checkout v7 release by [`@aiqiaoy`](https://github.com/aiqiaoy) in [actions/checkout#2464](https://redirect.github.com/actions/checkout/pull/2464) > * update error wording by [`@aiqiaoy`](https://github.com/aiqiaoy) in [actions/checkout#2467](https://redirect.github.com/actions/checkout/pull/2467) > > New Contributors > ---------------- > > * [`@aiqiaoy`](https://github.com/aiqiaoy) made their first contribution in [actions/checkout#2454](https://redirect.github.com/actions/checkout/pull/2454) > > **Full Changelog**: <actions/checkout@v6.0.3...v7.0.0> Changelog *Sourced from [actions/checkout's changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md).* > Changelog > ========= > > v7.0.0 > ------ > > * Block checking out fork PR for pull\_request\_target and workflow\_run by [`@aiqiaoy`](https://github.com/aiqiaoy) in [actions/checkout#2454](https://redirect.github.com/actions/checkout/pull/2454) > * Bump actions/publish-immutable-action from 0.0.3 to 0.0.4 in the minor-actions-dependencies group across 1 directory by [`@dependabot`](https://github.com/dependabot)[bot] in [actions/checkout#2458](https://redirect.github.com/actions/checkout/pull/2458) > * Bump flatted from 3.3.1 to 3.4.2 by [`@dependabot`](https://github.com/dependabot)[bot] in [actions/checkout#2460](https://redirect.github.com/actions/checkout/pull/2460) > * Bump js-yaml from 4.1.0 to 4.2.0 by [`@dependabot`](https://github.com/dependabot)[bot] in [actions/checkout#2461](https://redirect.github.com/actions/checkout/pull/2461) > * Bump `@actions/core` and `@actions/tool-cache` and Remove uuid by [`@dependabot`](https://github.com/dependabot)[bot] in [actions/checkout#2459](https://redirect.github.com/actions/checkout/pull/2459) > * upgrade module to esm and update dependencies by [`@aiqiaoy`](https://github.com/aiqiaoy) in [actions/checkout#2463](https://redirect.github.com/actions/checkout/pull/2463) > * Bump the minor-npm-dependencies group across 1 directory with 3 updates by [`@dependabot`](https://github.com/dependabot)[bot] in [actions/checkout#2462](https://redirect.github.com/actions/checkout/pull/2462) > > v6.0.3 > ------ > > * Fix checkout init for SHA-256 repositories by [`@yaananth`](https://github.com/yaananth) in [actions/checkout#2439](https://redirect.github.com/actions/checkout/pull/2439) > * fix: expand merge commit SHA regex and add SHA-256 test cases by [`@yaananth`](https://github.com/yaananth) in [actions/checkout#2414](https://redirect.github.com/actions/checkout/pull/2414) > > v6.0.2 > ------ > > * Fix tag handling: preserve annotations and explicit fetch-tags by [`@ericsciple`](https://github.com/ericsciple) in [actions/checkout#2356](https://redirect.github.com/actions/checkout/pull/2356) > > v6.0.1 > ------ > > * Add worktree support for persist-credentials includeIf by [`@ericsciple`](https://github.com/ericsciple) in [actions/checkout#2327](https://redirect.github.com/actions/checkout/pull/2327) > > v6.0.0 > ------ > > * Persist creds to a separate file by [`@ericsciple`](https://github.com/ericsciple) in [actions/checkout#2286](https://redirect.github.com/actions/checkout/pull/2286) > * Update README to include Node.js 24 support details and requirements by [`@salmanmkc`](https://github.com/salmanmkc) in [actions/checkout#2248](https://redirect.github.com/actions/checkout/pull/2248) > > v5.0.1 > ------ > > * Port v6 cleanup to v5 by [`@ericsciple`](https://github.com/ericsciple) in [actions/checkout#2301](https://redirect.github.com/actions/checkout/pull/2301) > > v5.0.0 > ------ > > * Update actions checkout to use node 24 by [`@salmanmkc`](https://github.com/salmanmkc) in [actions/checkout#2226](https://redirect.github.com/actions/checkout/pull/2226) > > v4.3.1 > ------ > > * Port v6 cleanup to v4 by [`@ericsciple`](https://github.com/ericsciple) in [actions/checkout#2305](https://redirect.github.com/actions/checkout/pull/2305) > > v4.3.0 > ------ > > * docs: update README.md by [`@motss`](https://github.com/motss) in [actions/checkout#1971](https://redirect.github.com/actions/checkout/pull/1971) > * Add internal repos for checking out multiple repositories by [`@mouismail`](https://github.com/mouismail) in [actions/checkout#1977](https://redirect.github.com/actions/checkout/pull/1977) > * Documentation update - add recommended permissions to Readme by [`@benwells`](https://github.com/benwells) in [actions/checkout#2043](https://redirect.github.com/actions/checkout/pull/2043) > * Adjust positioning of user email note and permissions heading by [`@joshmgross`](https://github.com/joshmgross) in [actions/checkout#2044](https://redirect.github.com/actions/checkout/pull/2044) > * Update README.md by [`@nebuk89`](https://github.com/nebuk89) in [actions/checkout#2194](https://redirect.github.com/actions/checkout/pull/2194) > * Update CODEOWNERS for actions by [`@TingluoHuang`](https://github.com/TingluoHuang) in [actions/checkout#2224](https://redirect.github.com/actions/checkout/pull/2224) > * Update package dependencies by [`@salmanmkc`](https://github.com/salmanmkc) in [actions/checkout#2236](https://redirect.github.com/actions/checkout/pull/2236) > > v4.2.2 > ------ > > * `url-helper.ts` now leverages well-known environment variables by [`@jww3`](https://github.com/jww3) in [actions/checkout#1941](https://redirect.github.com/actions/checkout/pull/1941) > * Expand unit test coverage for `isGhes` by [`@jww3`](https://github.com/jww3) in [actions/checkout#1946](https://redirect.github.com/actions/checkout/pull/1946) > > v4.2.1 > ------ > > * Check out other refs/\* by commit if provided, fall back to ref by [`@orhantoy`](https://github.com/orhantoy) in [actions/checkout#1924](https://redirect.github.com/actions/checkout/pull/1924) ... (truncated) Commits * [`9c091bb`](actions/checkout@9c091bb) update error wording ([#2467](https://redirect.github.com/actions/checkout/issues/2467)) * [`1044a6d`](actions/checkout@1044a6d) getting ready for checkout v7 release ([#2464](https://redirect.github.com/actions/checkout/issues/2464)) * [`f028218`](actions/checkout@f028218) Bump the minor-npm-dependencies group across 1 directory with 3 updates ([#2462](https://redirect.github.com/actions/checkout/issues/2462)) * [`d914b26`](actions/checkout@d914b26) upgrade module to esm and update dependencies ([#2463](https://redirect.github.com/actions/checkout/issues/2463)) * [`537c7ef`](actions/checkout@537c7ef) Bump `@actions/core` and `@actions/tool-cache` and Remove uuid ([#2459](https://redirect.github.com/actions/checkout/issues/2459)) * [`130a169`](actions/checkout@130a169) Bump js-yaml from 4.1.0 to 4.2.0 ([#2461](https://redirect.github.com/actions/checkout/issues/2461)) * [`7d09575`](actions/checkout@7d09575) Bump flatted from 3.3.1 to 3.4.2 ([#2460](https://redirect.github.com/actions/checkout/issues/2460)) * [`0f9f3aa`](actions/checkout@0f9f3aa) Bump actions/publish-immutable-action ([#2458](https://redirect.github.com/actions/checkout/issues/2458)) * [`f9e715a`](actions/checkout@f9e715a) block checking out fork pr for pull\_request\_target and workflow\_run ([#2454](https://redirect.github.com/actions/checkout/issues/2454)) * See full diff in [compare view](actions/checkout@df4cb1c...9c091bb) [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility\_score?dependency-name=actions/checkout&package-manager=github\_actions&previous-version=6.0.3&new-version=7.0.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

Sync gRPC related work changes into Forked Repo

49baaa9

gemini-code-assist Bot reviewed Aug 31, 2025

View reviewed changes

oc-shakeIQ added 16 commits August 31, 2025 18:12

Checkpoint Commit

39ad394

Checkpoint Commit

b81da83

Checkpoint Commit

af1fb5e

Checkpoint Commit

ab64b7b

Checkpoint Commit

871eb97

Checkpoint Commit

4850386

provided as scope

97ae92f

Checkpoint Commit

44dcb65

Checkpoint Commit

ed6bd11

Checkpoint Commit

cbb8db8

Checkpoint Commit

e11b284

Checkpoint Commit

923e3bd

Checkpoint Commit

69183d5

Checkpoint Commit

6d4ac30

Checkpoint Commit

e267b03

Checkpoint Commit

a7ba6be

robfrank self-assigned this Sep 2, 2025

robfrank added the enhancement New feature or request label Sep 2, 2025

robfrank added this to the 25.9.1 milestone Sep 2, 2025

robfrank added the high_priority label Sep 2, 2025

oc-shakeIQ added 7 commits September 2, 2025 08:13

Revised updateRecord logic to support EMBEDDED types properly

7d9dd91

Merge remote-tracking branch 'upstream/main'

11369fb

Added Admin Service to GRPC Plugin

e242927

Introduce GrpcValue and GrpcRecord constructs

ff26b23

Merge remote-tracking branch 'upstream/main'

f2176c2

Checkpoint Commit

f661950

Checkpoint Commit

fd14b6a

oc-shakeIQ added 13 commits September 5, 2025 06:16

Revised GrpcValue structure

07a088a

Added debug logging

f7f78d4

Revised GrpcValue structure and additional logging

236d714

Adding more advanced JSON processing

3769678

Checkpoint Commit

7850fbb

Support Schema Types retrieval using gRPC

511cbd1

Added Project Serialization

99dc6d2

Checkpoint Commit

0beb1a4

Checkpoint Commit

0d9fc0a

Checkpoint Commit

2455ef7

Checkpoint Commit

efd1ac8

Checkpoint Commit

aa764ff

Checkpoint Commit

33e55f5

This was referenced Sep 11, 2025

Feat/2498 grpc protocol #2499

Merged

Add grpc protocol wrapper #2498

Closed

robfrank closed this Sep 11, 2025

lvca assigned oc007us Oct 9, 2025

lvca added the fixed / implemented label Oct 9, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

gRPC support for ArcadeDB#2462