Skip to content

feat(compiler): add flatbuffers idl support#3184

Merged
chaokunyang merged 18 commits intoapache:mainfrom
chaokunyang:add_flatbuffer_idl_support
Jan 22, 2026
Merged

feat(compiler): add flatbuffers idl support#3184
chaokunyang merged 18 commits intoapache:mainfrom
chaokunyang:add_flatbuffer_idl_support

Conversation

@chaokunyang
Copy link
Copy Markdown
Collaborator

@chaokunyang chaokunyang commented Jan 21, 2026

Why?

Currently, Fory compiler only supports its native FDL format and Protocol Buffers. Users who have existing FlatBuffers schemas or prefer FlatBuffers IDL syntax cannot leverage Fory's high-performance serialization without manually converting their schemas. This creates friction for teams migrating from FlatBuffers to Fory or those who want to use FlatBuffers IDL as their schema definition language.

What does this PR do?

This PR adds comprehensive FlatBuffers IDL support to the Fory compiler, enabling users to define schemas using .fbs files and generate serialization code through Fory's compilation pipeline.

Key Features

  1. FlatBuffers IDL Parser: Complete lexer and recursive descent parser for FlatBuffers .fbs files

    • Supports namespaces, includes, attributes, enums, structs, and tables
    • Parses field attributes like deprecated, priority, default values
    • Error reporting with line/column information
  2. IR Translator: Translates FlatBuffers AST to Fory IR

    • struct → Fory struct with evolving=false (fixed layout)
    • table → Fory struct with evolving=true (schema evolution supported)
    • Field attributes map to Fory field options
    • Auto-growing field IDs based on declaration order (for schema compatibility)
  3. Scalar Type Mapping:

    FlatBuffers Fory Type
    byte int8
    ubyte uint8
    short int16
    ushort uint16
    int varint32
    uint var_uint32
    long varint64
    ulong var_uint64
    float float32
    double float64
    bool bool
    string string
    [T] list<T>
  4. Unsigned Array Support: Added cross-language unsigned array type annotations

    • Java: @Uint8ArrayType, @Uint16ArrayType, @Uint32ArrayType, @Uint64ArrayType, @Int8ArrayType
    • Python: Uint8Array, Uint16Array, Uint32Array, Uint64Array, Int8Array type hints
    • Enables FlatBuffers [ubyte], [ushort], [uint], [ulong] to serialize correctly across languages
  5. Cross-Language Field ID Support: Extended fingerprint computation to include field IDs for stable cross-language type compatibility

Usage

# Compile FlatBuffers schema
fory compile schema.fbs --lang java,python --output ./generated

# Inspect translated FDL for debugging
fory compile schema.fbs --emit-fdl --emit-fdl-path ./translated

Example

Input (FlatBuffers):

namespace MyGame;

enum Color : byte { Red = 0, Green, Blue }

struct Vec3 {
  x: float;
  y: float;
  z: float;
}

table Monster {
  pos: Vec3;
  mana: short = 150;
  hp: short = 100;
  name: string;
  friendly: bool = false (deprecated, priority: 1);
  inventory: [ubyte];
  color: Color = Blue;
}

Generated (Java):

public class Monster {
    @ForyField(id = 0)
    private Vec3 pos;

    @ForyField(id = 1)
    private short mana;

    @ForyField(id = 2)
    private short hp;

    @ForyField(id = 3)
    private String name;

    @ForyField(id = 4)
    private boolean friendly;

    @ForyField(id = 5)
    @Uint8ArrayType
    private byte[] inventory;

    @ForyField(id = 6)
    private Color color;

    // getters/setters...
}

Related issues

Closes #3171

Related: #3099, #1197, #1017

Does this PR introduce any user-facing change?

  • New API: FlatBuffers IDL support in fory compile command
  • New Annotations: Java unsigned array type annotations (@Uint8ArrayType, etc.)
  • New Types: Python unsigned array type hints (Uint8Array, etc.)

No breaking changes - all existing FDL and Protobuf workflows continue to work unchanged.

Does this PR introduce any binary protocol compatibility change?

No. The generated code uses the same Fory serialization format. The FlatBuffers support is purely a schema definition convenience - the wire format remains Fory's binary protocol, not FlatBuffers wire format.

@chaokunyang chaokunyang changed the title feat(compiler): add flatbuffer idl support feat(compiler): add flatbuffers idl support Jan 21, 2026
@chaokunyang chaokunyang mentioned this pull request Jan 21, 2026
16 tasks
@chaokunyang chaokunyang changed the title feat(compiler): add flatbuffers idl support feat(compiler): add flatbuffers idl parse and translate support Jan 21, 2026
@chaokunyang chaokunyang changed the title feat(compiler): add flatbuffers idl parse and translate support feat(compiler): add flatbuffers idl support Jan 21, 2026
@chaokunyang chaokunyang merged commit ce0e5d3 into apache:main Jan 22, 2026
63 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Xlang] [Compiler] Flatbuffer Schema IDL support

2 participants