Skip to content

Update Forward Protocol Specification for zstd compression support #4758

Open
Open
Update Forward Protocol Specification for zstd compression support#4758
Labels
enhancementFeature request or improve operationspendingTo be done in the future

Description

@daipom
daipom
opened on Jan 6, 2025

Is your feature request related to a problem? Please describe.

The following PR supports zstd compression.

So, we need to update Forward Protocol Specification - CompressedPackedForward Mode.

Describe the solution you'd like

Update the following description and add zstd value to compressed option.

Would that be Forward Protocol Specification v1.6?

Describe alternatives you've considered

Having no idea.

Additional context

No response

Activity

  1. Imagedaipom
    added
    waiting-for-triage
    on Jan 6, 2025
  2. Imagegithub-project-automation
    added this to Fluentd Kanbanon Jan 6, 2025
  3. Imagedaipom
    added
    enhancementFeature request or improve operations
    and removed
    waiting-for-triage
    on Jan 6, 2025
  4. Imagedaipom
    moved this to Work-In-Progress in Fluentd Kanbanon Jan 6, 2025
  5. Imagedaipom
    self-assigned this
    on Jan 6, 2025

  6. daipom commented on Jan 6, 2025

    @daipom
    daipom
    on Jan 6, 2025
    ContributorAuthor

    I propose changes to the following two sections.

    CompressedPackedForward Mode

    https://github.com/fluent/fluentd/wiki/Forward-Protocol-Specification-v1.5#compressedpackedforward-mode

     ### CompressedPackedForward Mode
 
-It carries a series of events as a msgpack binary, compressed by gzip, on a single request. The supported compression algorithm is only gzip.
+It carries a series of events as a msgpack binary, compressed by gzip or zstd, on a single request.
 
-- `entries` is a gzipped binary chunk of `MessagePackEventStream`, which MAY be a concatenated binary of multiple gzip binary strings.
-- Client MUST send an option with `compressed` key with the value `gzip`.
-- Client MUST send a gzipped chunk as msgpack `bin` format.
+- `entries` is a gzip/zstd binary chunk of `MessagePackEventStream`, which MAY be a concatenated binary of multiple gzip/zstd binary strings.
+- Client MUST send an option with `compressed` key with the value `gzip` or `zstd`.
+- Client MUST send a gzip/zstd chunk as msgpack `bin` format.
 - Server MUST accept `bin` format.
 - Server MAY decompress and decode individual events on demand but MAY NOT do right after request arrival. It means it MAY costs less, compared to `Forward` mode, when decoding is not needed by any plugins.

    Option

    https://github.com/fluent/fluentd/wiki/Forward-Protocol-Specification-v1.5#option

     - Server MAY just ignore any options given.
 - `size`: Clients MAY send the `size` option to show the number of event records in an entries by an integer as a value. Server can know the number of events without unpacking entries (especially for PackedForward and CompressedPackedForward mode).
 - `chunk`: Clients MAY send the `chunk` option to confirm the server receives event records. The value is a string of Base64 representation of 128 bits `unique_id` which is an ID of a set of events.
-- `compressed`: Clients MUST send the `compressed` option with value `gzip` to tell servers that entries is `CompressedPackedForward`. Other values will be ignored.
+- `compressed`: Clients MUST send the `compressed` option with value `gzip` or `zstd` to tell servers that entries is `CompressedPackedForward`. Other values will be ignored.
 
 ```json
 {"chunk": "p8n9gmxTQVC8/nh2wlKKeQ==", "size": 4097}

  7. daipom commented on Jan 6, 2025

    @daipom
    daipom
    on Jan 6, 2025
    ContributorAuthor

    Does this revision require a voting process?

  8. Athishpranav2003 commented on Jan 6, 2025

    @Athishpranav2003
    Athishpranav2003
    on Jan 6, 2025
    Contributor

    The changes suggested here looks good to me.

  9. daipom commented on Jan 7, 2025

    @daipom
    daipom
    on Jan 7, 2025
    ContributorAuthor

    If there are no objections, I would like to set it to v1.6 with the agreement of both Fluentd and Fluent Bit maintainers.

    @cosmo0920
    Could you please check #4758 (comment) ?

  10. ashie commented on Jan 7, 2025

    @ashie
    ashie
    on Jan 7, 2025
    Member

    LGTM

  11. cosmo0920 commented on Jan 8, 2025

    @cosmo0920
    cosmo0920
    on Jan 8, 2025
    Contributor

    This would be good:

    -- entries is a gzipped binary chunk of MessagePackEventStream, which MAY be a concatenated binary of multiple gzip binary strings.
    -- Client MUST send an option with compressed key with the value gzip.
    -- Client MUST send a gzipped chunk as msgpack bin format.
    +- entries is a gzip/zstd binary chunk of MessagePackEventStream, which MAY be a concatenated binary of multiple gzip/zstd binary strings.
    +- Client MUST send an option with compressed key with the value gzip or zstd.
    +- Client MUST send a gzip/zstd chunk as msgpack bin format.

    However, I have a question for these sentences. Currently, we're able to decompress compressed chunks one-by-one. So, in paper, we are also able to take turns to decompress gzip compressed chunks and zstd compressed chunks.
    Is this specification needed to point out explicitly in the revised specification?

  12. Athishpranav2003 commented on Jan 8, 2025

    @Athishpranav2003
    Athishpranav2003
    on Jan 8, 2025
    Contributor

    @cosmo0920 are u pointing to the case where different sources send different type of compressed chunks?

    Incase of forward protocol the message itself has the compression type so the decompression happens wrt the metadata of the compressed chunk.

  13. cosmo0920 commented on Jan 8, 2025

    @cosmo0920
    cosmo0920
    on Jan 8, 2025
    Contributor

    @cosmo0920 are u pointing to the case where different sources send different type of compressed chunks?

    Incase of forward protocol the message itself has the compression type so the decompression happens wrt the metadata of the compressed chunk.

    Yes, different sources use the same in_forward case. This is surely occurred because Fluentd is able to be used as an aggregator which will collect the different Fluentd instances with pointing the same forward endpoint. So, probably we need to clarify on it even if the implementation already does.

  14. Athishpranav2003 commented on Jan 8, 2025

    @Athishpranav2003
    Athishpranav2003
    on Jan 8, 2025
    Contributor

    @cosmo0920
    https://github.com/fluent/fluentd/pull/4657/files#diff-71dd20388a1f90eaec72fb257002fc6d44a497d457821d0c105acc0510844be1R314
    In this line the compression format is also used as an argument during decompression. I guess this should answer your question.

  15. 29 remaining items

  16. daipom commented on Jan 23, 2025

    @daipom
    daipom
    on Jan 23, 2025
    ContributorAuthor

    Or, add #4758 (comment) something like the following:

    • Client SHOULD NOT send zstd to the server that doesn't support zstd format.

    Or, for now, we will make zstd available as an exprimenatal feature on the Fluentd side, and note in the documentation that it is an experimental feature that cannot be used unless the server supports it.

  17. Athishpranav2003 commented on Jan 23, 2025

    @Athishpranav2003
    Athishpranav2003
    on Jan 23, 2025
    Contributor

    Even i strongly feel that its more suitable to put this as an important note and leave it to the responsiblity of the users

  18. cosmo0920 commented on Jan 23, 2025

    @cosmo0920
    cosmo0920
    on Jan 23, 2025
    Contributor

    First, we make Fluentd handle zstd internally and revert Fluentd's out_forward zstd feature to avoid violating the protocol.
    When Fluent Bit is ready to support zstd, we will update the protocol and restore Fluentd's out_forward zstd feature.

    Reverting zstd compression support is not working for us. Just displaying as a warning log to tell: "This is experimental feature and not standardized yet" or similar message should be enough.

    I mean, adding zstd compression with upgrading mechanism could make seamless upgrading to use zstd compression in forward protocol. However, we didn't find a suitable way to prevent unsupported errors on "fluent servers".
    So, just leaving as an experimental feature should be enough on Fluent Bit side.

  19. daipom commented on Jan 23, 2025

    @daipom
    daipom
    on Jan 23, 2025
    ContributorAuthor

    @Athishpranav2003 @cosmo0920
    OK! Thanks!
    Let's note in the document and log message!

  20. Imagedaipom
    added a commit that references this issue on Apr 9, 2025

    out_forward: add warning for zstd as experimental

    64dff03
  21. Imagekenhys
    added a commit that references this issue on Apr 10, 2025

    out_forward: add warning for zstd as experimental (#4896)

    3a6826a

  22. daipom commented on Jun 27, 2025

    @daipom
    daipom
    on Jun 27, 2025
    ContributorAuthor

    Let's note in the document and log message!

    Done!

  23. daipom commented on Jun 27, 2025

    @daipom
    daipom
    on Jun 27, 2025
    ContributorAuthor

    We will wait for Fluent Bit to support zstd, and then update the protocol.

  24. Imagedaipom
    removed this from Fluentd Kanbanon Jun 27, 2025
  25. Imagedaipom
    added
    pendingTo be done in the future
    on Jun 27, 2025
  26. Imagedaipom
    removed their assignment
    on Jun 27, 2025

  27. cosmo0920 commented on Jun 27, 2025

    @cosmo0920
    cosmo0920
    on Jun 27, 2025
    Contributor

    We will wait for Fluent Bit to support zstd, and then update the protocol.

    We're also waiting for supporting observability payloads on Fluentd. :)
    It means that cmetrics binding for Ruby which is called cmetrics2 for handling incompatibilities of cmetrics msgpack payload between v0.3 and v0.5 or later.
    Plus, adding a capabilities to ingest those of msgpack payloads on Fluentd.
    Currently, we converted those of payloads as logs mode and sent them into Fluentd.

  28. daipom commented on Jun 27, 2025

    @daipom
    daipom
    on Jun 27, 2025
    ContributorAuthor

    Oh,,, thanks!
    Sorry, I don’t fully understand that part.
    You're saying that Fluentd doesn't support the v1.5 protocol yet, right?
    That's something Fluentd needs to address.
    We should start by creating an issue.

  29. cosmo0920 commented on Jun 27, 2025

    @cosmo0920
    cosmo0920
    on Jun 27, 2025
    Contributor

    You're saying that Fluentd doesn't support the v1.5 protocol yet, right?

    Yes. That's why we need to treat that observability payloads should be treated as optional stuffs in v1.5.

  30. cosmo0920 commented on Aug 6, 2025

    @cosmo0920
    cosmo0920
    on Aug 6, 2025
    Contributor

    FYI: We're starting to process this feature in Fluent Bit side. The thing is, Fluent Bit has strict rules of size of chunks. So, we need to implement streaming style of decompression methods and use it at in_forward plugin.
    This is because our assumed environments are under containerized environments with strict rule of resource limits.
    So, it could take a bit of longer than we firstly expected to implement zstd decompression and compression in Fluent Bit side.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementFeature request or improve operationspendingTo be done in the future

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions