You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Problem
Finatra Kakfa Streams is an integration between Kafka Streams and Finatra which we've
been using internally at Twitter for the last year. The library is not
currently open-source.
Solution
Open-source Finatra Kafka Streams.
Result
Finatra Kafka Streams can be used outside of Twitter! Also, since Finatra Kafka
Streams can be mixed into a Finatra HTTP or Thrift service, it is now possible to
utilize Kafka Kafka Streams queryable state using a Finatra HTTP
or Thrift endpoint.
Differential Revision: https://phabricator.twitter.biz/D248408
The `integration tests <https://github.com/twitter/finatra/blob/develop/kafka-streams/kafka-streams/src/test/scala/com/twitter/unittests/integration>`__ serve as a good collection of example Finatra Kafka Streams servers.
7
+
8
+
Word Count Server
9
+
-----------------
10
+
11
+
We can build a lightweight server which counts the unique words from an input topic, storing the results in RocksDB.
12
+
13
+
.. code:: scala
14
+
15
+
class WordCountRocksDbServer extends KafkaStreamsTwitterServer {
16
+
17
+
override val name = "wordcount"
18
+
private val countStoreName = "CountsStore"
19
+
20
+
override protected def configureKafkaStreams(builder: StreamsBuilder): Unit = {
We can then expose a Thrift endpoint enabling clients to directly query the state via `interactive queries <https://kafka.apache.org/21/documentation/streams/developer-guide/interactive-queries.html>`__.
35
+
36
+
.. code:: scala
37
+
38
+
class WordCountRocksDbServer extends KafkaStreamsTwitterServer with QueryableState {
39
+
40
+
...
41
+
42
+
final override def configureThrift(router: ThriftRouter): Unit = {
43
+
router
44
+
.add(
45
+
new WordCountQueryService(
46
+
queryableFinatraKeyValueStore[String, Long](
47
+
storeName = countStoreName,
48
+
primaryKeySerde = Serdes.String
49
+
)
50
+
)
51
+
)
52
+
}
53
+
}
54
+
55
+
In this example, ``WordCountQueryService`` is an underlying Thrift service.
Finatra has native integration with `Kafka Streams <https://kafka.apache.org/documentation/streams>`__ to easily build Kafka Streams applications on top of a `TwitterServer <https://github.com/twitter/twitter-server>`__.
7
+
8
+
Features
9
+
--------
10
+
11
+
- Intuitive `DSL <https://github.com/twitter/finatra/blob/develop/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/internal/utils/FinatraDslV2Implicits.scala>`__ for topology creation, compatible with the `Kafka Streams DSL <https://kafka.apache.org/21/documentation/streams/developer-guide/dsl-api.html>`__
12
+
- Full Kafka Streams metric integration, exposed as `TwitterServer Metrics <https://twitter.github.io/twitter-server/Features.html#metrics>`__
13
+
- `RocksDB integration <#rocksdb>`__
14
+
- `Queryable State <#queryable-state>`__
15
+
- `Rich testing functionality <testing.html>`__
16
+
17
+
Basics
18
+
------
19
+
20
+
With `KafkaStreamsTwitterServer <https://github.com/twitter/finatra/blob/develop/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/KafkaStreamsTwitterServer.scala>`__,
21
+
a fully functional service can be written by simply configuring the Kafka Streams Builder via the ``configureKafkaStreams()`` lifecycle method. See the `examples <examples.html>`__ section.
22
+
23
+
Transformers
24
+
~~~~~~~~~~~~
25
+
Implement custom `transformers <https://kafka.apache.org/21/javadoc/org/apache/kafka/streams/kstream/Transformer.html>`__ using `FinatraTransformerV2 <https://github.com/twitter/finatra/blob/develop/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/streams/transformer/FinatraTransformerV2.scala>`__.
26
+
27
+
Aggregations
28
+
^^^^^^^^^^^^
29
+
There are several included aggregating transformers, which may be used when configuring a ``StreamsBuilder``
30
+
+ ``sample``
31
+
+ ``sum``
32
+
+ ``compositeSum``
33
+
[TODO : Add as available]
34
+
- Unique counting with HyperLogLog
35
+
- TopK counting
36
+
37
+
Stores
38
+
------
39
+
- Stores
40
+
- Timers / TimerStores
41
+
42
+
RocksDB
43
+
~~~~~~~
44
+
In addition to using `state stores <https://kafka.apache.org/21/javadoc/org/apache/kafka/streams/state/Stores.html>`__, you may also use a RocksDB-backed store. This affords all of the advantages of using `RocksDB <https://rocksdb.org/>`__, including efficient range scans.
45
+
46
+
Queryable State
47
+
~~~~~~~~~~~~~~~
48
+
Finatra Kafka Streams supports directly querying state from a store. This can be useful for creating a service that serves data aggregated within a local Topology. You can use `static partitioning <https://github.com/twitter/finatra/blob/develop/kafka-streams/kafka-streams-static-partitioning/src/main/scala/com/twitter/finatra/streams/partitioning/StaticPartitioning.scala>`__ to query an instance deterministically known to hold a key.
49
+
50
+
See how queryable state is used in the following `example <examples.html#queryable-state>`__.
Finatra Kafka Streams includes tooling that simplifies the process of writing highly testable services. See `TopologyFeatureTest <https://github.com/twitter/finatra/blob/develop/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/streams/tests/TopologyFeatureTest.scala>`__, which includes a `FinatraTopologyTester <https://github.com/twitter/finatra/blob/develop/kafka-streams/kafka-streams/src/test/scala/com/twitter/finatra/streams/tests/FinatraTopologyTester.scala>`__ that integrates Kafka Streams' `TopologyTestDriver <https://kafka.apache.org/21/javadoc/org/apache/kafka/streams/TopologyTestDriver.html>`__ with a `KafkaStreamsTwitterServer <https://github.com/twitter/finatra/blob/develop/kafka-streams/kafka-streams/src/main/scala/com/twitter/finatra/kafkastreams/KafkaStreamsTwitterServer.scala>`__.
0 commit comments