Reaqtor is a framework for creating reliable, stateful, distributed, and scalable event processing based on Reactive Extensions.

The team that created Rx did not rest on their laurels: they used Rx as the basis for developing a distributed, reliable, and extremely scalable event processing service. Reactor, as it was then called, has been integral to some of Microsoft's most widely used services for many years, including Cortana and Office 365, but although Microsoft has talked in public about it a few times, it has never been available for use outside of Microsoft—until now.

On May 18th 2021, Bart de Smet, made Reaqtor (its new name) open source, as a .NET-Foundation-sponsored project.

This talk will explain what makes Reaqtor different from other high-scale event processing systems. It will also show some of the foundational components that have been released as part of this, most notably 'Bonsai', a mechanism for serializing computations based on .NET's expression tree system, and which is central to how Reaqtor works.

But what does this mean?

Let’s use the Take operator as an example.

Firstly, using normal Rx, let's say we used this operator to take the first 8 items in an observable stream as follows:

var taken = observable.Take(8)

We will assume that the observable stream acts like a queue, where items are processed once and then removed from the queue.

Using the code above, we would only take the first 8 items in the stream. However, if the process was restarted then the Take operator would be reapplied and we would again take the first 8 items.

This would mean that all of the items that had previously been processed would be lost, and that the first 8 items from part way through the stream would be processed.

When using Reaqtor on the other hand, a key feature is that operator state is persistent.

In Reaqtor

The way this works is that the reactive engine generates checkpoints regularly. These checkpoints record where in the stream you have reached, and the operator state at that point.

This means that should the engine be restarted, you can restart processing from where you left off. Not only this, but the preserving of operator state means that operations (such as take) will not be reapplied.

Let's run through a demo to highlight what this looks like. The main method of the demo looks like this:

static async Task Main(string[] args)
{
    await PayNoAttentionToTheManBehindTheCurtain();
    switch (args[0])
    {
        case "1":
            await Part1();
            break;
        case "2":
            await Part2();
            break;
    }
    ps.Dispose();
}

Where we can either run Part1 or Part2 each time we start up the process.

Part 1

When we start up Reaqtor's query engine we run the following code:

Uri queryEngineIdentifier = new Uri("demo://test/engine1");

var engine = new QueryEngine(queryEngineIdentifier, scheduler, store, ingressEgressManager);

await engine.RecoverAsync(store.GetReader());

var ctx = new ReactorContext(engine);

First, we recover the query engine from any existing state (at this point no subscriptions have been started in the engine, so when it starts nothing will happen).

Once we have instantiated our query engine, we then get a reference to definitions of the timer observable , and the consoleOutput observer which will allow us to produce output to the console:

var timer = ctx.GetObservable<TimeSpan, DateTimeOffset>(new Uri("demo://observables/timer"));
var consoleOutput = ctx.GetObserver<double>(new Uri("demo://observers/cout"));

We are able to retrieve these from there definitions becase they have been registered on the reactive context:

await ctx.DefineObserverAsync(new Uri("demo://observers/cout"), 
                              ctx.Provider.CreateQbserver<T>(Expression.New(typeof(ConsoleObserver<T>))), 
                              null, 
                              CancellationToken.None);

await ctx.DefineObservableAsync<TimeSpan, DateTimeOffset>(new Uri("demo://observables/timer"), 
                                                          t => new TimerObservable(t).AsAsyncQbservable(), 
                                                          null, 
                                                          CancellationTokenNone);

In this case the observer/observable definitions are a part of the demo project, e.g.:

internal sealed class ConsoleObserver<T> : IObserver<T>
{
    public void OnCompleted() => Console.WriteLine("OnCompleted()");
    public void OnError(Exception error) => Console.WriteLine($"OnError({error})");
    public void OnNext(T value) => Console.WriteLine($"OnNext({value})");
}

However there are many of these types defined as part of Reaqtor, with more being added as we speak!

Once we have retrieved our timer and consoleOutput definitions, we can then use them to create a subscription:

var currentSpeedObservable =
    timer(TimeSpan.FromSeconds(1))
    .Select(_ => GetSpeed())
    .Take(8);

This takes the timer observable definition, tells it to call the GetSpeed method every second, and take the first 8 of those values. This is an asynchronous process definition, and no work will start until something subscribes.

The report speed method here is just reporting a random number:

private static double GetSpeed()
{
    return Math.Round((30 + (new Random().NextDouble() * 5)), 0);
}

But this is where the real data that you are processing using Reaqtor would be produced. This data could be anything - current speed, temperature, failure events, location data, etc. Reaqtor has the potential to be used in a huge variety of different scenarios - many of which we are yet to think of!

So, we will now subscribe our consoleOutput observer:

await currentSpeedObservable.SubscribeAsync(consoleOutput, 
                                            new Uri("demo://subscriptions/sub1"), 
                                            null, 
                                            CancellationToken.None);

And wait 4 seconds before checkpointing and then unloading the engine:

await Task.Delay(TimeSpan.FromSeconds(4.3));
await engine.CheckpointAsync(store.GetWriter());
await engine.UnloadAsync();

When we run the code, we will see the following output:

OnNext(34)
OnNext(33)
OnNext(33)
OnNext(32)
OnNext(30)
Engine was unloaded, saving state...
State saved.

We can see that five items are passed to the OnNext method of our output observer. The engine is then unloaded and the state is saved.

Back in the code we can see that we are writing this state out to a file on disk:

Console.WriteLine("Engine was unloaded, saving state...");

string json = store.AsJson();

File.WriteAllText("State.json", json);
Console.WriteLine("State saved.");

After saving out this state, we reach the end of the Part1 method and the program exits.

Part 2

If we now restart the program, this time running Part2, we again start by reloading the engine from the stored state:

string json = File.ReadAllText("State.json");
store = InMemoryKeyValueStore.FromJson(json);
var engine = new QueryEngine(QueryEngineIdentifier, scheduler, store, ingressEgressManager);
await engine.RecoverAsync(store.GetReader());

As soon as the engine recovers, it finds that there are unprocessed items which need to be passed to our consoleOutput observer, and the following output will be produced:

Loading state...
Recovering from stored state
Recovery complete
OnNext(31)
OnNext(30)
OnNext(33)

We can see that the remaining items from the observable are now passed to our observer. Unlike in regular Rx, the state of the Take operator has been saved and instead of restarting and taking the next 8 items, it remembers that it has already taken 5 and just produces the remaining 3.

Therefore, processing continues as if the restart had never happened.

The fact that queries can continue to run throughout engine restarts means that you have the ability to support extremely long running operations. These queries can move between separate processes and machines, with the current state of the process being written out to persistent storage.

Reaqtor also gives assurances about reliability that mean that you can be sure that no state will be lost throughout these restarts, and that you can run Rx-like queries but with predictable and repeatable results.

The upshot is that you can create queries which can run for hours, days, months, or even years! Without worrying about engine restarts losing state, or producing unpredictable and incorrect results. And, it this ability to execute these incredibly long running queries that enables support for services like Bing, Cortana and M365.

This exciting technology brings the world of live, reactive, Rx-based queries together with the long-term persistence of information!

Here's a link to a video recording of this demo.

And if you want to know more about Reaqtor, head over to reaqtive.net for more detail about how it works, the kind of scenarios it can support, and the history of the project!

It shows a live stream of shipping location data (AIS) initially handled with classic Rx, and then shows how Reaqtor can host this same processing model as a long-running subscription in the cloud.

See the talk Durability and Reliability in Reaqtor Subscriptions for the second part of this demo.

Note: this demo uses reaqtor.cloud, endjin's proprietary Azure Service Fabric based host for the Reaqtor primitives.

Just as when you store data in a database, you can rely on that data to survive across reboots and even hardware failures, the same is true of a Reaqtor subscription.

See the talk Rx as a Service for the first part of this demo.

Note: this demo uses reaqtor.cloud, endjin's proprietary Azure Service Fabric based host for the Reaqtor primitives.

"What single action could you take which would deliver the biggest positive impact on customer satisfaction?"

The question popped up in a Brain Trust session with Ilario Corna, Head of Infrastructure, Content & Operations at Talk Talk (a UK Telco). The organisation had just rolled out an objective to become the UK's "most recommended provider". As part of this strategic initiative, we had designed and implemented an Azure based solution capable of ingesting and analysing over 200 million network telemetry events per day for anomalous behaviour. Feeling buoyed by success we were wondering "which hard problem could we tackle next?"

Ilario had the answer immediately. "Oh, that's easy" he said, "I would send a field engineer out to each customer's house, run diagnostics on their broadband connection and hardware, upgrade their firmware and implement any other tweaks required. But with over 1 million customers that's not commercially feasible to implement."

After taking a sip of tea and pondering for a few moments I responded "So we have a solution, but we need to find a way to implement and scale it several orders of magnitude more cheaply. If I were in your shoes and could wave a magic wand, I'd want to be able to say something like 'Cortana, monitor the Quality of Service for each of my customer's connections, and if it's below our SLA, then run a trouble-shooter to diagnose why and then apply a number of automatic remediation strategies'. Wouldn't that be a great way to manage the network?"

He responded with a chuckle, "Sure would! When can I have it?!?" We both laughed then, because it sounded like a far-off sci-fi fantasy.

On the train home, that part of the conversation kept coming back to me. Why couldn't we create some type of digital agent that could monitor not only the customer's device, but every part of the network infrastructure that led to the customers home? The nuance was that unlike traditional "big data" problems, whereby you need to reduce vast amounts of data into a format a human can comprehend, this type of problem is more akin to signals analysis: you need to process the full fidelity raw data in a stateful way.

If you have 1 million customers, you need demultiplex those 200 million messages into 1 million queries, each query will need to look at the message and ask "does this apply to me?" if so it would process the event and update its quality of service metric, evaluate if that had met a threshold and if so, trigger a new "threshold exceeded" event. Articulating the problem this way sounded a bit like the Actor Model, or perhaps an Rx query.

A few years previously we had worked on an DevOps project for a major UK online retailer. We used Rx to aggregate and disaggregate semantic logging events from all the servers in the data centre. The Rx query we implemented was simple, elegant, and incredibly powerful; a testament to the design of Rx, and a reason we're huge fans of it at endjin.

I wondered if it would be possible to run 1 million Rx queries concurrently, and remembered Bart De Smet has presented a session at NDC 2015 about Cloud Event Processing the previous year, so I found & watched the recording. In that talk he mentioned that he was working in the Bing organisation on an evolution of Rx which powered Cortana experiences, and this involved stateful, durable, high density Rx queries. This sounded very promising indeed!

While researching to see if this technology had been open sourced, I came across a Microsoft case study about "Bing Cortana" (unfortunately no longer public) that mentioned a technology called Reactor, which was used to evaluate 500 million queries per second. If this was the same technology, we only needed 1/500th of that capacity in order to solve our broadband Quality-of-Service monitoring scenario.

I reached out to David Goon, who was our Partner Manager at Microsoft, explained the situation and asked "do you think it would within the realms of possibility for us to get access to this technology?" He responded "It's a really compelling use case, and I can see there being lots more across other sectors. Let me reach out to Bart De Smet and see what he says. We can only try." Within a couple of weeks, we had an hour long call with Bart where we took him through the scenario we had elaborated, and asked if he thought Reactor would be a viable solution. He said that it seemed to be a good fit, and we agreed to start collaborating on a proof of concept.

It's hard to convey what an absolute honour and privilege it has been to work with Bart, and I will be forever grateful for the amount of his incredibly valuable time he dedicated to this endeavour. It goes without saying that without him, absolutely none of what follows would have been possible. Reaqtor is Bart's baby; we've just had the privilege of delivering it into the world.

Once Bart provided us with access to some sample Reactor code, Mike Larah and I paired on a "simple scenario"; an entirely in-memory version of Reactor hosted in a Console App, based on the NDC talk code sample Bart shared with us. We wanted to see if we could solve the broadband anomaly detection problem by processing a stream of RADIUS network telemetry and detecting the specific events that indicate there were problems in the broadband network. In this prototype we modelled the network as a graph, which allowed us to determine if an issue was localised to the customer's home or was occurring at a local (an outage for a street) or regional node level (an outage for a town). It worked, and we were fizzing with excitement.

One of the key points you need to understand is that Reactor is orders of magnitude bigger than Reactive Extensions with over 166 projects, and 90 NuGet packages, and yet it is still a framework, not a platform. While state, durability and reliability are defined in the design of the framework, the implementation is missing, as this is tied to the hosting environment.

Reactor was born into Bing and adopted by M365 and other product teams. Bart had done a fantastic job of ensuring a clean separation of Reactor from the hosting environment, but it meant that we had to tackle the thorny problem of building our own reliable hosting platform, state store, ingress & egress adapters, management & control plane, and workbench for talking to Reactor. We had a mountain to climb.

When it comes to solving hard problems, Ian Griffiths has always been our first port of call; he's been an endjin associate since we founded the company. I knew he was very passionate about Rx as he'd included a chapter about the subject in his Programming C# series of books for O'Reilly, and we'd had many conversations about it. Over a few months we implemented an initial spike of a reliable hosting platform, and had an absolute blast while doing it. Shortly afterwards Ian joined endjin full time as our first Technical Fellow, so that he could continue to work on the endeavour.

Over the next few years, we built several technical spikes exploring different facets of the Reactor technology; we embedded ML.NET models into Reaqtor queries, we created a proof-of-concept anomaly detection temporal query (using virtual time) that could correctly identify broadband quality of service events that indicate outages; processing a days' worth of telemetry (around 40GB) in under 90 seconds. The ability to do "what-if" simulations over historic datasets, which we could then flip a switch and turn into live queries running against realtime data blew us away. I want to express my thanks to Ben Dyer and Alex KeySmith for providing us with the anonymised dataset for this proof of concept - they were early believers.

In May 2018 Matthew Adams, my co-founder at endjin, & I flew over to Seattle for the Microsoft Build conference and we managed to spend our last day of the trip with Bart. In the morning Bart told us the story of how Reactor came to be (this is covered in detail in the "A Little History of Reaqtor" ebook you can download for free on the homepage). Over a fantastic lunch we waxed lyrical about the future of data processing, and Bart talked about his vision for IComputationProcessing (more about this in the ebook!) and then in the afternoon we talked about whether we could open source Reactor and what that process would entail.

Carmel Eve, who had interned with us the previous year, re-joined endjin on our Apprenticeship Programme. Ian and Carmel started exploring Reactor with other proof of concepts covering different customer scenarios we had identified. Throughout this time Carmel documented her learnings about Rx and writing high performance C# in popular blog post series. The more we investigated Reactor, the more impressed we became.

In March 2019 I attended the MVP Summit in Redmond, and I bumped into Jon Galloway, who at the time was the Executive Director of the .NET Foundation. I gave him as brief a Reactor elevator pitch as I could, and said that endjin would love to become .NET Foundation Corporate Sponsors, as we wanted to demonstrate our commitment to the .NET ecosystem, while we also collaborated with Bart to create whitepapers and a business case in order to justify the investment from Microsoft to carry out the legal processes required to open source Reactor to the .NET Foundation under The MIT License.

By the end of the year we'd filled out the paperwork, paid our sponsorship fee, and were featured on the .NET Foundation homepage alongside: AWS, DevExpress, Microsoft, Octopus Deploy, Telerik, Uno Platform and VMWare.

At this point, we had to face the problem that we had been avoiding for the past 3 years; what do we call it? "Reactor" as a name is too overused. Microsoft Reactor is the name of the Microsoft Community meet up venues around the world. Project Reactor is the name of a JVM project from VMWare, inspired by Rx, and based on Reactive Streams. React is the hugely popular reactive user interface library from Facebook. One of the key concepts that powers Reactor is IQbservable. We've always loved this weirdly named interface (partly due to how Bart pronounces it - I-Cub-servable). Carmel came up with the suggestion that we include the "Q" in the name and as soon as Reaqtor was written down, it received unanimous agreement.

This also neatly solved two other naming problems we faced. Reaqtor consists of three conceptual layers; at the bottom a set of reusable components (that can be used independently of Reaqtor) which provide compiler, JSON and LINQ, and high performance / low memory extensions and utilities, which we named Nuqleon. The middle layer contains reactive primitives that evolve many of the concepts found in Rx to enable reliable, stateful and distributed reactive processing, which we named Reaqtive. And finally the top level consists of platform level services for building reliable, stateful, distributed reactive solutions, called Reaqtor.

We started to collaborate with the .NET Foundation to onboard the project. Special thanks to Claire Novotny who helped us set up the DevOps processes on the .NET Foundation infrastructure and all the other bits of administrivia required to prepare the project to be released to the public.

We also worked with Rodney Littles and the .NET Technical Steering Group to elaborate an RFC to modernise the Expression Tree subsystem, as it has not been invested in for many years and does not have parity with the latest language features. These improvements are fundamental to the future of Reaqtor. Fortunately there seems to be some progress in this area as the Entity Framework team would also like this modernisation to occur.

Part of the problem with Expression Trees is that they are one of the most complex and least well documented parts of .NET; they are also perceived only to exist to power LINQ (and LINQ Providers). As you'll see with Reaqtor (an in particular the Nuqleon layer, which is where the Expression Tree & Bonsai subsystem lives), Expression Trees really are one of .NET super powers and enable mind-bending meta-programming scenarios.

One of the final puzzle pieces was how we could create a great documentation experience for the community. Ian has a long history in teaching programming; DevelopMentor, Pluralsight and his O'Reilly books. We've had many conversations about what a good learning environment looks like: the ability to tell a cohesive narrative including text, images, videos and code samples that you can execute in context (to eliminate task switching). Microsoft Docs have done an excellent job in this respect, but how can you offer a similar experience when you are a small open source project?

We initially investigated Try .NET which lead us to talking to Jon Sequeira, Dr Diego Colombo, Maria Naggaga and Brett Forsgren. Initial conversations quickly turned to their current project .NET Interactive, which seemed perfect for our needs, not only for interactive documentation but also as an IDE for writing, testing and running Reaqtor queries.

We soon realised that .NET Interactive was designed around a request/response (REPL) model, but we'd really need to support long running in-process and out of process reactive queries. We collaborated with the team on a design for a client-side command pattern that would allow us to define custom commands and send them between the kernels. Ian implemented the feature and the PR was merged into the .NET Interactive codebase, allowing us to create great documentation and interesting demos, that you can use to understand Reaqtor, Reaqtive & Nuqleon.

The final significant contributor to the project is Felix Corke who designed the branding and creative assets which we have made available under a Creative Commons Attribution Share Alike 4.0 International license, so that the community can use them in blog posts, videos and presentations about Reaqtor. We have also created a Community Presentations repository containing branded, pre-canned presentations and demos using the existing interactive notebooks, so that you can easily do a presentation or lightning talk at your local .NET User Group.

Our journey with Reaqtor only covers a short period in its overall history. In his (free) ebook, A Little History of Reaqtor Bart De Smet tells the full story... starting in 2005. It is an absolutely fascinating account, and I hope by the time you finish reading it, you'll understand why we believe Reaqtor is the most exciting technology in the .NET ecosystem, that it's a game changer, and why we're so excited that you can get your hands on the framework, run the demos, read through the conceptual and API documentation, watch our growing collection of talks, read our blog (and don't forget to subscribe to our RSS feed) or chat to us via Slack. Please ⭐the GitHub Repos, as we'd love to gauge community interest in the project.

There's currently a significant feature gap between the Reaqtor components that have been open sourced and what's required to implement a large scale production Reaqtor platform. That's why we've spent 5 years building reaqtor.cloud a full, commercial cloud-native platform implementation of Reaqtor. We hope you'll join the waiting list for the upcoming private preview.

As an example of what you might do with this, consider an application that wants to detect when a customer broadband router is experiencing a higher error rate than the average for the exchange to which the router is attached. A typical broadband provider will have millions of customers across hundreds of exchanges. These systems generate huge volumes of monitoring data, and it is necessary to identify and understand problems as they arise to pre-empt customer problems in order to reduce support costs and improve the customer's experience.

You could arrange for network health events be made available inside Reaqtor as event sources that can be subscribed to. You could then feed these events through the GroupBy operator to separate the data out by exchange, and then through a Buffer operator to calculate a rolling average error rate for each exchange as a whole. The results of this could then be made available within Reaqtor as an event source that other subscriptions can use.

You could then define subscriptions for each individual customer's router. (Remember, there will be millions of these routers, but Reaqtor is designed to handle that kind of scale easily.) Again we could use Buffer to calculate a rolling error rate, and then feed these through a Where clause that compares the router-specific error rate with the average for that router's exchange, only allowing through events that are above the average.

This way, you can get notifications from individual devices if they start to experience high failure rates while avoiding getting swamped by such notifications if it turns out that a large number of devices attached to the same exchange are all reporting similar failures due to an exchange-level problem. By combining a highly localized point of view (events from an individual router in a particular customer's house) with broader context (aggregated performance at the level of a whole telephone exchange) you can better understand the information you're seeing, e.g., you can distinguish between problems with individual pieces of equipment and broader failures.

This sort of logic can be expressed as LINQ queries using standard operators. These queries then get transferred into the 'Query Engine' inside Reaqtor (possibly via a 'Query Coordinator', which can optimize queries, and work out how to distribute them across multiple Query Engines to improve scale).

These then become long running queries which (unlike with ordinary Rx subscriptions) can continue to run long after the code that initially set them up has stopped running. Reaqtor has failover support enabling queries to migrate to new servers if the server they are running on needs to reboot, or just fails.

The basic abstraction underpinning this is the same one we saw in the first article in this series—a sequence of items. Processing and combination is enabled by the LINQ features. We're able to do this reactively—in response to live events as they happen—thanks to Rx, which has been augmented by Reaqtive to add with the additional concepts required for persistent, long-running subscriptions. Remotable expression trees enable both the remote instantiation of such subscriptions, but also their persistence. And Reaqtor then provides the runtime services required to make all of that real, offering a host environment for Rx-like subscriptions, with persistence, checkpointing, and failover, and it has a proven track record of scaling to tens of millions of customers.

One of the most important features added by Reaqtor, and its supporting libraries Nuqleon and Reaqtive, is the ability to serialize expression trees. We've already seen how with IQueryable<T>, it's possible to translate a LINQ query into something like SQL and execute it remotely. But with the ability to serialize expression trees, it becomes possible to enable remote execution of queries without needing to translate them to anything else: you can just ship an entire LINQ query to another process and run it there.

That capability is at the heart of Reaqtor.

First, there's the obvious fact that if you can send expressions over a network, this opens the door to being able to offer "Rx as a service". Whereas with classic Rx, everything is inherently in-process, remotable expressions provide a way to send a description of a subscription to an external process. Slightly more subtly, serializable expressions also help enable persistence—if you have a serialized description of a subscription, you can store that somewhere persistent, and then reload it into a new service instance if the one it was previously running in stops running for any reason. (In practice, persistent subcriptions require more than just the ability to serialize their defining expressions: many Rx operators maintain state, which must also be persisted, but the serialization of expressions is still a crucial element.)

It isn't strictly necessary to understand the exact details of how expressions are serialized—in fact the Reaqtor codebase offers a couple of different formats. Normally, a JSON-based format is used, but there is also code in there for serializing to a binary representation. The fundamental capabilities are more interesting than the exact format details. So it's worth knowing that serialized expression trees ('Bonsai trees', as they are typically known) are able to represent more or less any expression that a .NET expression tree can, but they are not necessarily tied to .NET.

It's important to understand that Bonsai trees are not compiled code—we are not shipping IL (.NET's equivalent to Java bytecode) over the wire. They are a description of the original structure of the expression. Also, they are typically self-contained: there is normally no need to distribute any particular extra compiled code to the receiving service to enable it to understand the tree. (Contrast this with some systems that enable remote invocation of user-defined functions implemented in Java: Spark supports user-defined functions, but to be able to run inside the processing nodes of a Spark cluster, the compiled version of a function needs to be accessible to those nodes. So you typically end up distributing .jar files across the cluster. This is unnecessary with Bonsai trees because they can be self-contained. After a service converts one back to a .NET expression tree (applying any necessary security checks, and whatever transformations it requires), it can compile it into runnable code at runtime.)

Bonsai does support embedding very .NET-specific information if you want to. (E.g., a Bonsai tree could indicate that an expression invokes the Where operator defined by the Queryable class in the System.Linq namespace in a specific version of the .NET System.Linq assembly). But it doesn't have to work this way.

When used with Reaqtor, Bonsai trees often replace such specific references with more abstract ones. It is common to rewrite expression trees so that particular methods defined by certain types are replaced with a placeholder identifying the relevant method by a URI. So although your code might refer to Queryable.Where, the Reaqtor client library would arragne for the Bonsai representation to replace this with an unbound parameter named, say, rx://operators/filter. This ability to substitute URI-based names for particular members of specific .NET types has the useful effect of decoupling Bonsai trees from any particular versions of library components, or from any particular version of .NET.

In fact, expression trees don't necessarily have to originate from .NET. In principle, any language can generate a Bonsai tree, although it's likely to be a lot more convenient if your language has support for something similar to .NET's Expression<T>, in which compilers can be induced to produce code that builds a description of an expression instead of simply compiling an expression into runnable code.

In the next and final part of this series, we'll see how the various Nuqleon, Reaqtive and .NET features described so far come together to provide a reliable, persistent high-performance, event-based service called Reaqtor.

Rx has been around for some time. Early versions were released directly by Microsoft. It was open sourced in 2012, and then moved under the .NET Foundation's control a couple of years later. After a few slightly moribund years, community activity started to increase in 2017, and it continues to be actively maintained. However, Rx also lived on in parallel inside Microsoft in a form that was not public for many years. Part of what Microsoft has made available in the Reaqtor libraries is in effect the new version of Rx, now in various namespaces under Nuqleon, Reaqtive & Reaqtor.

We've already seen a couple of 'dimensions' with IEnumerable<T> and Rx. We can think of the expression tree/callback choice (to Q or not to Q) as one particular dimension, and you can choose to stand in either the callback side or the expression tree side. This is independent of the pull/push choice, which is another dimension. We can choose between two options in that dimension too, leading to four possibilities: IEnumerable<T> (pull, callbacks), IQueryable<T> (pull, expressions), IObservable<T> (push, callbacks) or IQbservable<T> (push, expressions).

Reaqtive adds various extra dimensions. One is the nature of subscriptions. With classic Rx, when you subscribe to an IObservable<T> (or IQbservable<T>) it returns an IDisposable, and you can unsubscribe (i.e., decide you no longer want to receive events—the logical equivalent of executing a break statement in a foreach over an IEnumerable<T>) by calling Dispose. But Reaqtive elevates subscriptions to a more sophisticated level, making them entities in their own right, with some capabilities for walking over the participants of a subscription (e.g., enabling all the operators in a subscription to share a context). This is reflected in the ISubscribable<T> and IQubscribable<T> interfaces. (It does not extend the pull world in this way, so there's no IReactiveEnumerable<T>, for example. Not every conceivable position across all the dimensions is populated.) Another dimension concerns whether entities such as observable sources or subscriptions are identified purely by .NET object identity, or whether they have associated URIs acting as persistent names. In the latter case we have various interfaces with Reactive in their names such as IReactiveObservable<T> and IReactiveQbservable<T>. (Again, not every possible combination exists; these Reactive forms all have the same strong subscription model as ISubscribable<T>, and there's no Reactive flavour without that feature.)

Yet another dimension is async. Reaqtive defines IAsyncReactiveObservable<T> and IAsyncReactiveQbservable<T>. (Once more, not every conceivable corner of this hyperspace is populated: there is no IAsyncObservable<T> for example. So if you want async, you are also required to be on the 'first class subscriptions' side of the 'reaqtive' dimension.) These are useful in distributed scenarios, where establishing a subscription might entail communicating with an external service, an inherently asynchronous operation.

Each of these dimensions has a role to play in how Reaqtor builds on the capabilities of Rx, as we'll see in the final two parts.

In the fifth part of this series, we'll examine remotable expressions.

A basic assumption made by both IEnumerable<T> and IQueryable<T> is that the information source (whether it's a data structure, an algorithm, or some sort of database) is able to provide information on demand. Code using these types asks for results either using foreach, or a LINQ operator that evaluates an entire query such as ToList or ToArray.

However, there is another way to model sequences of things: the source might produce items when it is good and ready to. This is sometimes referred to as 'push' and it's the model Rx uses. It models this with an interface called IObservable<T>.

The basic idea underpinning IObservable<T> is identical to that of IEnumerable<T>: both represent a sequence of items—one damn thing after another. The only difference is that the source gets to decide when the next item is produced.

The IObservable<T> interface is a sort of mirror image of IEnumerable<T>. Just as the latter has a corresponding IEnumerator<T> interface used when iterating through the items, IObservable<T> has a corresponding IObserver<T>. But whereas with IEnumerable<T> you ask for an IEnumerator<T> when you're ready to start working through the sequence, with IObservable<T>, you must supply an IObserver<T> when you're ready for items, because the IObservable<T> is going to invoke methods on you for each item.

IObserver<T> has three methods: OnNext(T), which the source invokes for each item it produces, OnCompleted, which it invokes to tell you the sequence has finished (the logical equivalent of IEnumerator<T>.MoveNext() returning false), and OnError, which it invokes to report an error (the logical equivalent of the MoveNext() method throwing an exception).

So enumerables and observables are logically equivalent (to the extent that the Rx libraries provide adapters that can convert from one to the other). It is just that sometimes, one will be a more natural way to represent some sequence than the other. IObservable<T> is the more natural way to represent events.

Rx includes a complete LINQ provider. More than complete in fact: in addition to all the operators available in other LINQ providers, Rx adds several new ones. (The team also released Ix, the "Interactive Extensions for .NET" which essentially provides versions of all these additional operators for IEnumerable<T> and IQueryable<T>.) So given some source of events, you can filter them with a Where clause, or do any of the other things LINQ supports.

Rx's 'push' world offers a particular distinction that we have also seen in the 'pull' world with IEnumerable<T> vs IQueryable<T>: in addition to IObservable<T>, Rx also defines IQbservable<T> (note the 'Q'). This is essentially a version of IObservable<T> but one where all the LINQ operators take lambdas in the expression tree form (whereas for a regular IObservable<T> the standard operators all take ordinary delegates—references to methods).

In the fourth part of this series, we'll take a look at Reaqtive.

Version 3.5 of the .NET Framework added a set of features which, amongst other things, made it significantly easier to work with sets of items. The features were known, collectively, as LINQ: Language Integrated Query. The headline capability of LINQ didn't obviously seem to have anything to do with IEnumerable<T>: its most touted feature was the ability to write database queries directly in C#, e.g.:

var openOrderIds =
    from order in dbContext.Orders
    where order.Status == OrderStatus.Open
    select order.Id;

Such queries could be translated at runtime into SQL. So although this code uses C#'s == operator in a way that appears to be inspecting the Status field of every order, in practice that will be converted into a SQL WHERE clause, meaning that the filtering will in fact be performed by the database server (which is generally what you want—downloading an entire copy of a table and then picking out the items you want is almost always a mistake).

However, while the marketing at the time put the spotlight on relational database support, the real power of LINQ came from the fact that none of the features added to support it had anything to do with databases. The new language features (new in 2008, that is) at work in that code snippet are:

• A query syntax
• A set of standard 'operators' for performing operations over sets of data
• Lambdas, i.e. the ability to write an expression whose value is a function
• The option to have the compiler translate lambdas into data structures describing those expressions instead of runnable code

It's the first of these that makes examples like the one above most resemble database queries. It's arguably also the least interesting, because you could remove it from the language without losing any of the functionality LINQ enables. Moreover, it obscures things a little—it's not obvious that the other features in that list are in use because of the way query syntax hides certain details. (Some people insist on not using query syntax for this reason. I think that's overkill because there are some situations in which query syntax enables simpler code that is easier to read, particularly in scenarios where let clauses are useful.)

The C# compiler translates queries into a series of method calls at compile time, with each call being invoked on the return value of the preceding one, leading to a chain of method calls. The query shown above is equivalent to this:

var openOrders = dbContext
    .Orders
    .Where(order => order.Status == OrderStatus.Open)
    .Select(order => order.Id);

If you were to inspect the output of the compiler for this and the previous snippet you would not be able to tell which of the results came from which input—the compiler simply translates the first form into the second form before going on to compile the code. This is why the query syntax isn't hugely interesting—it just provides a different look for your code.

In this second form, we can now see the operators more clearly: the Where and Select methods are examples of LINQ operators. A LINQ operator is not a language feature—it's more of a convention: for each operator there is a standard name for the method, some expected behaviour, and a particular method signature. (In some cases, there might be multiple standard signatures, in which case the methods will have overloads for each form).

For example, take the Where operator. The expected behaviour is that this will filter the items: the operator takes a filter function which must accept a single input—an item of whatever type the data set contains—and return a bool. If this function returns true for any particular item, that item will be included, and if it returns false, that item will be filtered out.

You might be wondering at this point "How is this a 'convention'? Isn't this just a method?" The answer is that there are many different implementations of the Where operator. The one you get depends entirely on the type of data set that you invoke it on. In the snippets above we are invoking Where on dbContext.Orders, and since I've not yet made it clear what that actually is, it's not yet possible to tell exactly which implementation of LINQ we're going to get.

So let's get clear.

Imagine dbContext is an Entity Framework context object, and its Orders property represents a particular table in the database. In that case, Orders would implement the IQueryable<Order> interface, an instance of the generic IQueryable<T> interface. This represents something slightly more abstract than IEnumerable<T>: it is some set of items of type T. One important distinction is that unlike with IEnumerable<T> there is no presumption of ordering. (This matters if the data set is ultimately a table in a database, because queries may well be executed in parallel, or in some cunning highly optimized way that means the ordering of elements is essentially unknowable while the query is in progress.) And the other important distinction is that an IQueryable<T> gets to decide exactly how queries are ultimately processed.

To expand on that last point, it's important to understand that there are several different LINQ implementations. Microsoft seems not to use these terms so much any more, but when LINQ was first introduced, they talked about "LINQ to Objects", "LINQ to SQL", and "LINQ to XML" and LINQ was designed explicitly to be open, so that anyone could implement their own "LINQ to Something".

The particular implementation of LINQ you get depends first of all on the static type of the object on which you invoke the operator. For example, if you invoke Where on something that implements IEnumerable<T>, this ends up invoking the Enumerable.Where extension method, and you are now using LINQ to Objects. If on the other hand you invoke it on something that implements IQueryable<T> then you end up invoking the Queryable.Where extension method, at which point the target object can decide at runtime exactly how to process the query. (And if you are writing your own "LINQ to Whatever" then you get to implement the Where method yourself, as either a member of, or an extension method for the target type.)

IQueryable<T> is a special case, and a slightly odd one on first inspection: if it ends up letting the target object decide exactly what to do, then what's the point of it? Why not just have the target object implement Where (and any other operators it wishes to support) directly? Well part of the answer is that by having a standard interface, it becomes possible to write code that does things with types that support LINQ, without needing to know what those types are. But in that case, why doesn't absolutely everything use it? What's the point of IEnumerable<T> if we have the more general purpose IQueryable<T>?

To understand why, it's worth looking at a difference between how standard operators look on these two types. Here's the method signature of the LINQ to Objects version of Where, supplied by the Enumerable class:

public static IEnumerable<T> Where<T>(
    this IEnumerable<T> source,
    Func<T, bool> predicate)

The this keyword on the first argument makes this an extension method, meaning the C# compiler will let you invoke this method directly on anything that implements IEnumerable<T> as though it were a member of that interface, even though it isn't really. And then the interesting part is the second argument: the predicate that determines whether each item should be let through. This is of type Func<T, bool> and that's an ordinary delegate type—this argument should essentially be a reference to a function.

So if I write this:

var openOrders =
    someIEnumerableOfOrders
    .Where(order => order.Status == OrderStatus.Open);

the compiler will turn that into something like this:

var openOrders =
    someIEnumerableOfOrders
    .Where(FilterMethod);

...

private static bool FilterMethod(Order order)
{
    return order.Status == OrderStatus.Open;
}

So it takes that lambda expression I wrote as the argument to Where, turns it into a method, and then passes a reference to that method. So this is ordinary executable code. Now let's look at the IQueryable<T> version of Where:

public static IQueryable<T> Where<T>(
    this IQueryable<T> source,
    Expression<Func<T, bool>> predicate)

Aside from the obvious difference that this deals with IQueryable<T> and not IEnumerable<T>, there's a more subtle difference: the predicate argument is now of type Expression<Func<T, bool>>. The significance of this is that it tells the compiler to do something completely different. Instead of wanting a reference to a method, this form of Where operator says that it wants a data structure describing the expression. So the compiler turns this:

var openOrders = 
    someIQueryableOfOrders
    .Where(order => order.Status == OrderStatus.Open);

into something more like this:

var orderParam = Expression.Parameter(typeof(Order), "order");
var predicate = Expression.Lambda<Func<Order, bool>>(
    Expression.Equal(
        Expression.Property(orderParam, "Status"),
        Expression.Constant(OrderStatus.Open)),
    orderParam);
var openOrders =
    someIQueryableOfOrders.Where(predicate);

In fact it's slightly more complex than that due to the way enum comparisons get handled, but this illustrates the basic idea: the compiler generates code which, when executed at runtime, creates a tree of objects representing the structure of the expression you originally wrote.

So in this case, our order => order.Status == OrderStatus.Open gets turned into an object representing a lambda expression with a single parameter called order of type Order, and whose body is an object representing an equality comparison that compares the result of fetching the order parameter's Status property with the constant value OrderStatus.Open.

Anything implementing IQueryable<T> will end up with code like this when you use LINQ operators that take lambdas—they all compile into the expression tree form. And then the source object implementing IQueryable<T> gets to decide how to process that expression tree.

If the source represents some table in a relational database as in our original query example, it will convert this expression into a SQL query which it will then send to the database.

Objects representing other sources could do something similar—for example, something representing a collection in a CosmosDB could convert an expression into a query that CosmosDB can process.

The upshot of all this is that LINQ defines a set of operators (which goes well beyond the simple filtering of the Where operator shown in this example—you can also do grouping, ordering, existential predicates, and numerous other jobs). Any type that supports LINQ can then offer some or all of these operators, and it can then implement them in whatever fashion it sees fit—it might simply run code on collections of objects (which is what LINQ to Objects does) but it might also translate queries into a different language to be executed remotely.

When Execution Occurs

In general, LINQ operators don't do anything immediately. When using either IEnumerable<T> or IQueryable<T> they only do work when you ask them for objects. For IEnumerable<T> this is important because it makes it possible to work with infinite sequences. If a Where clause attempted to perform filtering the moment you call the Where method, it wouldn't work on an infinite sequence because it would never finish. But in fact LINQ to Objects works on demand—when you chain together a series of operators, you end up with an IEnumerable<T> that only starts work when you obtain an enumerator and then call MoveNext().

In general, IQueryable<T> implementations also defer their work although they have an additional motivation: in cases where they translate the query into something else (e.g. SQL) they want to wait until they know you've finished building the query. Until you actually start asking for results, a LINQ provider has no way of knowing if you're planning to append any more operators. Take this example from earlier:

var openOrders =
    dbContext.Orders
    .Where(order => order.Status == OrderStatus.Open)
    .Select(order => order.Id);

We could also have written this thus:

var orders = dbContext.Orders;
var whereQuery = orders.Where(order => order.Status == OrderStatus.Open);
var openOrders = whereQuery.Select(order => order.Id);

This has exactly the same effect, we've just put the result of each operator into a named variable rather than going on to invoke a method on it directly. If operators ran immediately, the Where operator would have run before the Select, and that's bad: what we would actually want this to do is run as a single SQL query that combines the Where and Select, and only returns the ids.

So when you use a LINQ operator, you don't normally get actual results back. You usually get a new instance of the same sort of data set you applied the operator to. For example, applying an operator to an IEnumerable<T> produces an IEnumerable<R> (where R may or may not be the same as T because some operators change the type—Select does that, for example).

This is called deferred execution: LINQ queries (for all widely used LINQ providers) don't do any work until you start asking them for results, which you typically do either by using a foreach loop on them, or by using LINQ's ToList operator, which evaluates the query and returns the results in a List<T>. (ToList is one of the few standard LINQ operators that does in fact execute immediately.)

In short, with the LINQ providers we've seen so far, operators do work when you pull information out of them. However, there is another way.

In the third part of this series, we'll delve into Rx.

Sequences and IEnumerable<T>

At Reaqtor's heart lies an incredibly simple abstraction: a sequence of items. It's just one damn thing after another, so to speak.

C# has had integrated language support for sequences of items from day one. Perhaps the most obvious example is the array, but that's not the most important: there are many different ways in which one might want to make a sequence of items, and while arrays matter because they are an intrinsic feature of the .NET runtime, there are many reasons you might not always want to use one. For example, .NET arrays have a fixed size. For this reason, we might often instead choose something like List<T>, a generic type that provides something strongly resembling a resizable array.

But there are often good reasons to use something that does not much resemble a dynamic array. Perhaps you need to support very long sequences with the ability to insert new items in the middle. Array-like structures can't do this efficiently, because if you want to insert an item in the middle (or at the start) of an array, you need to move all the items after the insertion point along by one position to make room for the new item. A linked list, on the other hand, is able to support insertion extremely cheaply. But it is absolutely terrible at finding the nth item, something arrays are extremely good at. This is why we have multiple data structures—there are many ways of representing things, each with its own strengths and weaknesses.

However, it is useful to be able to write code that doesn't need to concern itself with the details of the data structure. For example, if you want to calculate the sum of all the integers in a sequence, it's handy to be able to write code that works independently of the exact representation. And you can in C#—you can write this sort of thing:

int total = 0;
foreach (int i in sequence)
{
    total += i;
}

This code can work across all sorts of different representations—it will work with native .NET arrays, or with List<T>, but also with collections that use radically different data structures such as LinkedList<T>, HashSet<T>, or ImmutableList<T>.

This is possible because the foreach statement recognizes a pattern embodied by the IEnumerable<T> interface. All of the aforementioned collection types implement IEnumerable<T> and can therefore be consumed by a foreach loop. (Language pedants might point out that IEnumerable<T> is not strictly a requirement, because the compiler will in fact accept any type that offers the necessary members. I would respond by suggesting said pedants reread the carefully worded first sentence of this paragraph, which accommodates this fact without letting it get in the way of things. Experienced writers will then note that I've blown it by drawing attention to the issue in this parenthetical; I have no defence on that point. In practice, IEnumerable<T> matters because as we will see, even though the C# foreach keyword is pretty flexible, many language features insist on this particular interface. And in practice, it's more or less ubiquitous—anything you might want to iterate over with foreach is likely to implement IEnumerable<T>.)

An important aspect of IEnumerable<T> is that it does not offer random access—it doesn't have an indexer enabling you to retrieve, say, the 1000th item. The way it works is that you call its GetEnumerator() method, which returns an IEnumerator<T>, which you then ask for items by repeatedly calling MoveNext(). If MoveNext() returns false, that means you've reached the end of the sequence. (Sometimes sequences are empty, in which case the very first call to MoveNext() will return false.) If it returns true, that means a new item is available, which you can then retrieve from the enumerator's Result property. Once you're done with an enumerator (either because you reached the end, or because you've decided not to fetch all the elements) you call Dispose() on it. The foreach keyword generates all this code for you.

This one-at-a-time behaviour can sometimes be a little frustrating for the consumer of an IEnumerable<T> but it enables considerable variety of implementation. It makes it easy for a linked list to implement, for example.

A more subtle upshot of the nature of IEnumerable<T> is that it means the sequence of objects doesn't necessarily have to exist in advance. For example, you could write an IEnumerable<int> that returns numbers produced by the Random class. Sequences that produce their items on demand like this are sometimes known as 'lazy' because they don't do work until they absolutely have to. Another interesting characteristic of this hypothetical random number generating IEnumerable<T> is that it never needs to end—you could make its MoveNext() return false after some number of calls, but you don't have to—you could just hardwire MoveNext() to return true, because you can carry on producing random numbers indefinitely. This could be described as an 'infinite' sequence. Infinite sequences are necessarily lazy (because any attempt to produce all the items up front would run out of memory), although not all lazy sequences are infinite.

So in summary, IEnumerable<T> is the normal way in .NET to represent sequences of items. Collection types (List<T>, LinkedList<T> etc.) almost invariably implement it, but it's also possible to write lazy implementations that produce items on demand, possibly without end.

In the second part of this series, we'll investigate LINQ.

In this talk, Ian Griffiths shows why Rx should be a critical part of any .NET developer's toolbox, and shows a real-world application that illustrates how the powerful processing and orchestration facilities Rx provides can be applied to collect and process data in edge devices and then stream the high-value events to an Azure IoT hub for further analysis in the cloud.

In this session Bart shows what it took to bring the concepts of Reactive Extensions (Rx) to the cloud to deal with latency, scale, reliability, and other concerns.

You'll also learn about the core API abstractions that are used to represent event processing systems of any size across our stack, including the cloud as well as devices and sensors.