[Xlang][Spec] Fory XLang Union Encoding Spec

[chaokunyang]

This proposal specifies the IDL union encoding in the Fory xlang serialization format using three union type IDs:

• UNION (31): union value without embedded union schema identity (schema known from context)
• TYPED_UNION (32): union value with embedded registered numeric type id
• NAMED_UNION (33): union value with embedded registered type name / shared typedef

This design moves union schema identity into Type Meta, making union consistent with STRUCT/ENUM/EXT patterns and
natural to carry inside Any.

---

1. IDL Syntax

1.1 Union definition

union Contact [id=0] {
  string email = 1;
  int32  phone = 2;
}

Rules:

• Each union alternative MUST have a stable tag number (= 1, = 2, ...).
• Tag numbers MUST be unique within the union.
• Tag numbers SHOULD follow protobuf evolution rules: do not reuse removed tag numbers.

1.2 Union usage

message Person [id=1] {
  Contact contact = 1;
}

---

2. Mapping from Other IDLs

2.1 Protobuf oneof → Fory union

Protobuf:

message Person {
  oneof contact {
    string email = 1;
    int32  phone = 2;
  }
}

Mapping:

• The oneof group maps to a Fory union.
• Each oneof field number becomes the union alternative tag number (case_id).

2.2 FlatBuffers union → Fory union

FlatBuffers:

union Equipment { Weapon, Monster }

Mapping example:

union Equipment {
  Weapon  weapon  = 0;
  Monster monster = 1;
}

• FlatBuffers discriminator values map to union alternative tag numbers (case_id).

---

3. Type IDs

3.1 Internal Type ID Table Updates

Type ID	Name	Description
31	UNION	Union value, schema identity is NOT embedded (context required).
32	TYPED_UNION	Union value with embedded registered numeric union type id.
33	NAMED_UNION	Union value with embedded union type name / shared typedef.

3.2 Type Meta Encoding

All type IDs are written as varuint32 as per the xlang Type Meta rules.

• UNION (31): no additional type meta payload
• TYPED_UNION (32): followed by union_type_id (varuint32)
• NAMED_UNION (33): followed by named-type meta payload:
  • meta share disabled: namespace + type_name (meta strings)
  • meta share enabled: shared TypeDef marker + TypeDef body (per xlang meta share)

Notes:

• union_type_id uses the standard Full Type ID rule:
  • Full Type ID = (user_type_id << 8) | internal_type_id
• How union schemas are registered/mapped is implementation-defined, but the numeric union_type_id MUST be stable
  between producer and consumer when TYPED_UNION is used.

---

4. Union Value Payload Encoding

A union value payload is encoded as:

| case_id (varuint32) | case_value (encoded as Any-style value) |

4.1 case_id

• case_id is the union alternative tag number from FDL/protobuf/FlatBuffers mapping.
• It is encoded as varuint32.
• case_id MUST be stable for evolution and MUST NOT be reused for a different alternative.

4.2 case_value (MUST be encoded as Any-style value)

To guarantee that unknown alternatives can be skipped, case_value MUST be encoded as a full Fory value,
equivalent to encoding the value as if it were stored in Any/UNKNOWN:

| field_ref_meta | field_value_type_meta | field_value_bytes |

Where:

• field_ref_meta is the standard reference meta (NULL/REF/NOT_NULL/REF_VALUE).
• field_value_type_meta is the standard xlang Type Meta (a varuint32 type_id plus optional meta payload).
• field_value_bytes is the value bytes encoded according to field_value_type_meta.

This is required even for primitives (e.g., INT32, STRING) to ensure skipping is always possible.

---

5. Full Wire Layout Examples

5.1 UNION (schema known from context)

Used when the deserializer already knows the union schema (e.g., the field is declared as a specific union type):

| ... outer ref meta ... | type_id=UNION(31) | case_id | case_value(any-style) |

5.2 TYPED_UNION (schema embedded by numeric id)

Used when union schema is not known from context, e.g., union is stored in Any:

| ... outer ref meta ... | type_id=TYPED_UNION(32) | union_type_id | case_id | case_value(any-style) |

5.3 NAMED_UNION (schema embedded by name/typedef)

Used when union schema is resolved by name or via meta share TypeDef:

| ... outer ref meta ... | type_id=NAMED_UNION(33) | (namespace,type_name OR typedef marker) | case_id | case_value(any-style) |

---

6. Decoding Rules

6.1 High-level decoding algorithm

1. Read outer ref meta (per standard rules).
2. Read type_id as varuint32.
3. If type_id == UNION(31):
   • Union schema MUST be provided by context (declared field type / target type).
4. If type_id == TYPED_UNION(32):
   • Read union_type_id (varuint32) and resolve the union schema from the registry.
5. If type_id == NAMED_UNION(33):
   • Read named-type meta (name strings or TypeDef marker) and resolve the union schema.
6. Read case_id (varuint32).
7. Read case_value as Any-style value:
   • Read field_ref_meta
   • If non-null and not a reference:
     • Read field_value_type_meta
     • Read/construct the value using that type meta

6.2 Unknown case_id handling (forward compatibility)

If the resolved union schema does not contain case_id, the decoder MUST still consume the case value:

1. Read field_ref_meta.
2. If non-null and not a ref:
   • Read field_value_type_meta
   • Call standard skipValue(field_value_type_meta.type_id) to skip field_value_bytes.

This guarantees that adding new union alternatives is forward compatible.

---

7. When to Use UNION vs TYPED_UNION vs NAMED_UNION

• Use UNION (31) when the union schema is known from context:
  • struct fields declared as a union type
  • collections/maps with declared union element/value type
  • explicit deserialize<TUnion>()
• Use TYPED_UNION (32) when the union schema is not known from context and numeric registration is available:
  • union stored in Any
  • union stored in fully dynamic UNKNOWN fields
• Use NAMED_UNION (33) when numeric registration is not available or name-based resolution is preferred:
  • unregistered union schemas
  • cross-language name mapping
  • meta share environments using shared TypeDef

Implementations MAY choose to always write TYPED_UNION/NAMED_UNION for simplicity, but UNION is recommended where
context exists for smaller payloads.

---

8. Compatibility and Evolution Notes

• case_id MUST be treated as a stable identifier (like protobuf field number).
• Adding a new alternative is forward compatible:
  • old readers skip unknown case_id because case values carry standard type meta.
• Removing an alternative is backward compatible if:
  • removed case_id is not reused
  • readers treat unknown alternatives as “present but ignored”.

---

9. Summary

• Introduce three internal union type IDs: UNION(31), TYPED_UNION(32), NAMED_UNION(33).
• Union schema identity is carried in Type Meta for typed/named unions, consistent with other user-defined types.
• Union payload is always:
  • case_id(varuint32) + case_value encoded as Any-style value (ref meta + type meta + value).
• Unknown union alternatives can always be skipped safely.

Related Issues

#3027

[chaokunyang]

Another proposal, which is not taken by me:

This issue specifies how Fory IDL unions are represented in the Fory xlang wire format, and how protobuf oneof
and FlatBuffers union can be mapped to Fory unions.

Design goals:

• Support schema evolution / unknown union alternatives (skip unknown cases safely).
• Support dynamic Any carrying a union value and restoring the exact union type.
• Reuse existing Fory value encoding (ref meta + standard type meta + value bytes) for union case values.
• Keep a single internal type id for union (Types.UNION = 31) and encode “typed union” as a variant in union payload.

---

1. IDL Syntax

1.1 Union definition

union Contact [id=0] {
  string email = 1;
  int32  phone = 2;
}

Rules:

• Each union alternative MUST have a tag number (= 1, = 2, ...).
• Tag numbers MUST be unique within the union.
• Tag numbers SHOULD follow protobuf evolution rules: do not reuse removed tag numbers.

1.2 Union usage

message Person [id=1] {
  Contact contact = 1;
}

---

2. Mapping from Other IDLs

2.1 Protobuf oneof → Fory union

Protobuf:

message Person {
  oneof contact {
    string email = 1;
    int32  phone = 2;
  }
}

Mapping:

• The oneof group maps to a Fory union.
• Each oneof field number becomes the union alternative tag number (case field_id).
• The oneof value is encoded using the union encoding defined in this document.

2.2 FlatBuffers union → Fory union

FlatBuffers:

union Equipment { Weapon, Monster }

Mapping example:

union Equipment {
  Weapon  weapon  = 0;
  Monster monster = 1;
}

• FlatBuffers union discriminator values map to union alternative tag numbers.

---

3. Wire Format Overview

A union value is encoded as a normal Fory value:

1. Reference meta (NULL/REF/NOT_NULL/REF_VALUE flags) is handled at the outer object level as usual.
2. Type meta contains internal type id UNION (31) and optional user type id/name resolution as per standard rules.
3. The union payload (defined below) follows.

This spec defines only the union payload layout.

---

4. Union Payload Encoding

4.1 Key requirement: case value is encoded as Any

To ensure a decoder can always skip unknown union alternatives, the union case value MUST be encoded as a full Fory value
(i.e., the same encoding as if the case value were an Any):

| field_ref_meta | field_value_type_meta | field_value_bytes |

This guarantees that an implementation which does not recognize field_id can still skip the case value by reading
field_value_type_meta and invoking standard skipValue(type_id) logic.

4.2 Union header (discriminator + payload kind)

The union payload starts with a union header, encoded as varuint32:

union_header = (field_id << 2) | kind

• field_id is the union alternative tag number (from FDL/proto).
• kind occupies the low 2 bits:

kind	Name	Meaning
0	NORMAL	Union type identity is NOT embedded (requires external declared type).
1	TYPED_BY_ID	Union embeds its registered numeric type id.
2	TYPED_BY_NAME	Union embeds its registered type name / typedef reference.
3	RESERVED	Reserved for future extension.

4.3 Union payload layouts

4.3.1 NORMAL union payload (kind = 0)

| union_header | field_ref_meta | field_value_type_meta | field_value |

Use when the union schema/type is known from context, e.g.:

• A struct field declared as Contact contact
• A list/map element whose declared generic type is the union type
• Any situation where the deserializer has a declared target union type

4.3.2 Union typed by registered numeric type id (kind = 1)

| union_header | union_type_id | field_ref_meta | field_value_type_meta | field_value |

• union_type_id is encoded as varuint32, using the standard “Full Type ID” rule:
  • Full Type ID = (user_type_id << 8) | internal_type_id
  • Named types do not embed a user id; see kind=2.

Use when the union schema/type is NOT known from context, most importantly when a union value is stored inside Any.

4.3.3 Union typed by name / shared typedef (kind = 2)

| union_header | union_type_name_or_typedef | field_ref_meta | field_value_type_meta | field_value |

• union_type_name_or_typedef MUST reuse the existing xlang named type meta mechanism:
  • If meta share is disabled: write namespace + type_name as meta strings.
  • If meta share is enabled: write a shared TypeDef marker and TypeDef body as defined by the xlang meta share rules.

Use when union type identity must be carried by name (unregistered types or cross-process name-based resolution),
especially for Any payloads.

---

5. Reference Meta and Value Type Meta

5.1 field_ref_meta

field_ref_meta is encoded exactly like any other value in xlang:

• NULL FLAG (0xFD): null case value, no further bytes for the case value
• REF FLAG (0xFE + ref_id): shared reference
• NOT_NULL VALUE FLAG (0xFF): non-null, no ref tracking
• REF VALUE FLAG (0x00): first occurrence with ref tracking

5.2 field_value_type_meta

field_value_type_meta is encoded exactly like normal xlang Type Meta:

• type_id as varuint32
• Optional meta payload depending on internal type id
  • e.g. NAMED_STRUCT uses name strings or shared TypeDef marker

This is required even for primitives, because it enables safe skipping of unknown union alternatives.

---

6. Decoding Rules

6.1 Decoding algorithm (high level)

1. Read union_header (varuint32).
2. Compute:
   • kind = union_header & 0x3
   • field_id = union_header >> 2
3. If kind == 1: read union_type_id and resolve the union schema type (for Any / dynamic contexts).
4. If kind == 2: read union_type_name_or_typedef and resolve the union schema type.
5. Read field_ref_meta.
   • If null/reference completes the value, stop.
6. Read field_value_type_meta (standard xlang type meta).
7. Deserialize the case value using field_value_type_meta and populate union result.

6.2 Unknown field_id handling

If the decoder does not recognize field_id in the resolved union schema:

• It MUST still consume the case value bytes by:
  1. reading field_ref_meta
  2. if non-null and not a ref, reading field_value_type_meta
  3. invoking standard skipValue(type_id) to skip field_value

This provides forward compatibility for added union alternatives.

---

7. When to Use Each kind

• Use NORMAL (kind=0) whenever the union schema type is known from the declared field type or decoding target type.
• Use TYPED_BY_ID (kind=1) or TYPED_BY_NAME (kind=2) when union schema type is not known from context, especially:
  • union stored in Any
  • union stored in an UNKNOWN / fully polymorphic field without a declared union type

Implementations MAY always use typed forms for simplicity, but kind=0 is recommended for smaller payloads when context is available.

---

8. Compatibility and Evolution Notes

• Union alternative tags (field_id) MUST be stable identifiers.
• Adding a new alternative is forward compatible:
  • old readers can skip unknown field_id because case values are encoded as Any (with type meta).
• Removing an alternative is backward compatible if:
  • the removed field_id is not reused
  • readers treat unknown alternatives as “present but ignored”

---

9. Summary

• Keep one internal type id for union: UNION (31).
• Encode union discriminator as union_header = (field_id << 2) | kind.
• Encode union case values exactly like Any: ref_meta + type_meta + value.
• Support Any holding unions by optionally embedding union type identity in union payload (kind=1/2).
• Use FDL/protobuf tag numbers as union case ids (field_id) for stable evolution.