Email plays a major role in our online collaboration and communication efforts. For most denizens of the Internet, email is the one true universal medium of communication. It connects us with our friends, coworkers, businesses, government and institutions. Simply put, for many of us email is a way of life.

For all the efficiency and freedom that email promises, it is not without flaws. The sheer volume and variety of emails we receive on a daily basis take up a lot of time to analyze, remember, and respond in a meaningful manner. Then there’s also the swarm of promotional and spam emails that nobody can seem to escape, which requires a significant amount of time and cognitive overhead to screen out.

These problems are exacerbated by the fact that typical email clients are not making it easy to triage emails. Consequently, many people are left with no choice but to treat their inbox as an ever-growing, unstructured pile of correspondences. For a lot of us it’s been the only way to ensure that relevant emails are easily reachable. Some email clients provide labels as a means for organizing this mess. However, without good automation, labels require extra effort to keep labeling new incoming emails, or juggle an unruly number of email filters.

Since early 2022 I’ve been working at Shortwave, a startup that is looking to build a better and smarter inbox experience on top of Gmail. Shortwave is available for web and iOS, with Android support available in beta. Our primary objective is to help people save time, and enable them to manage their online conversations better. In this post I detail some of the ways we achieve that.

Email triaging made easy

Shortwave treats the email inbox as a ToDo list. And like with any ToDo list, we want people to complete all tasks in the list as efficiently as possible. Note that this is polar opposite to the “inbox as an ever-growing stack” school of thought. Instead we believe that your inbox should be empty or close to empty whenever possible. This is sometimes known as the inbox zero method for managing emails. The idea is that you triage new incoming emails quickly, get to inbox zero state, and get on with your other important work.

So how does Shortwave enable that? For each email thread, Shortwave offers four possible triage actions:

1. Done ✅: If a thread is not important or otherwise doesn’t require any action on your part, just mark it as Done. This moves the thread out of the inbox, and archives it. Most emails we receive are one-way notifications (e.g. GitHub updates, newsletters, billing alerts), and do not require an explicit response. Done is the perfect triage response for those threads. You can always recall the thread from the Done-page or via search.
2. Pin 📌: If it’s an important and immediately actionable thread, pin it. This will keep the thread at the top of your inbox page. So no more rummaging around for important conversations you wish to frequently lookup. Once a pinned thread has run its course, and starts to become obsolete/un-actionable, you can mark it as Done to get it out of the inbox too.
3. Snooze ⏰: Some email threads don’t warrant an immediate action, but may require some work in the future. Perhaps you have an important meeting coming up next week that you need to prep for, or your car registration is up for renewal in a couple of months. You can snooze threads like these until a specific time. Snoozed threads stay out of your inbox, and pops back in when the specified time is up. At that point you can decide whether to Pin it, mark it as Done, or Snooze for a bit longer.
4. Delete 🗑: We can think of this as a more heavy-handed Done operation. I’m personally not in the habit of deleting emails regularly. But for those who do, Delete is indeed another available option.

You can read more about the triage actions, and the overall Shortwave method on our website. The goal is to apply the above triage actions to each email thread in the inbox, and burn down the ToDo list towards inbox zero.

We want people to reach this state quickly and frequently. To that end, Shortwave enables triage actions on bundles of threads.

Get things done faster with bundles

Shortwave groups related threads into bundles 📦. Threads from automated senders are always bundled. You can further enable bundling for individual senders, mailing lists, and labels.

Bundling is akin to pre-sorting your emails into groups. Imagine going through a large collection of traditional (snail mail) letters. Your goal is to find the handful of letters in the collection that you are required to respond to. This is quite difficult if you’re given just an unsorted pile of letters. You will have to go through each letter, one at a time, until you find the ones that are important. Unfortunately, this is the experience that most email clients offer today, and it is a huge time drain.

But suppose somebody already made a pass through the letters, and organized them by sender and category. Your job immediately becomes a lot easier. You can just discard the bundles that are obviously spam and/or irrelevant, and jump into the important bundles that are likely to have useful letters. This is the experience Shortwave attempts to provide. With bundling you don’t have to go through your inbox one thread at a time. You can quickly scan through an entire bundle of threads, and take a triage action on the whole bundle in one click or keyboard shortcut. This is great for getting through large piles of automated notifications, promotions and other noisy emails.

In addition to automated bundling, you can create your own custom bundles as well. Just drag-and-drop a few threads together to create a bundle. Then add a note for clarity. I use this feature a lot whenever I’m traveling. I create a custom bundle with all my travel reservations and other related threads. These threads are usually from a variety of senders (hotels, airlines, car rentals etc). Then I give the custom bundle a clear name like “Trip to San Diego”, and pin the whole bundle at the top of my inbox. This way I know that my trip details are always at the top of my inbox. No more frantically searching on my phone for a key piece of information while on the road. I can also arrange the threads within the bundle in the exact order I need them.

Get organized with smart labels

Shortwave also supports labels 🏷. Your existing Gmail labels will continue to work in Shortwave, and any new labels you create in Shortwave will sync back to Gmail. But additionally, Shortwave provides a collection of built-in labels. These serve as high-level email categories that you can easily configure bundling and push notifications for. Labels can also be added to favorites for easy access, and can be used as a search vector.

What’s interesting about the labels support in Shortwave is that it drastically reduces the overhead of manually applying labels or having to create filters for every single sender. Instead, whenever you apply a label to a thread, Shortwave will remember that action, and automatically apply the label to future threads from the same sender. This is what makes them smart labels — they apply themselves without you having to take any manual action repeatedly, or having to create an explicit filter in Gmail.

There are of course situations where you don’t want this. For those cases you can configure Shortwave to opt-out of the auto-apply behavior based on a specific sender or a label. There are also probably a handful of other cases where you might have to manually create a Gmail filter for a label. But I’ve personally only ever needed the default smart label behavior of Shortwave.

Labels also support batched delivery 📬. You can configure threads with a given label to be delivered to the inbox on a schedule that you determined. I use this for built-in labels like Promotions and Forums, which normally contain a lot of noisy, low-priority threads. Instead of receiving a steady stream of those emails into my inbox throughout the day, I’ve configured them for once-a-day batched delivery. Consequently, I receive my Promotions bundle every day sharp at 6PM, which I most of the time just mark as Done in a single click after a quick scan.

This saves me time by enabling me to make a single triaging decision for a whole day’s worth of promotional emails.

Get the gist with smart summaries

So far we discussed how Shortwave helps to organize your inbox, and triage incoming emails efficiently. But what if the threads you care about happen to be long and text-heavy? This is where smart summaries 🗣 come in. Shortwave uses AI (LLMs) to summarize lengthy threads into a few short sentences, so you can quickly get the gist of a given thread.

This is an excellent way to catch up on a long thread that you haven’t participated in a while, or as a triage-time tool to determine the importance of an email. Instead of spending 10 minutes reading a long thread only to realize that it’s not that important after all, you can read the smart summary produced by Shortwave, and then decide whether it’s worth your time. You can also attach smart summaries to outgoing emails whenever you forward an email to another party, and help save their time too.

If the original email thread is in a different language, Shortwave AI produces a translated summary in English. For English speakers this is a good way to analyze and understand emails that are written in other languages. We are planning to add more language support around this feature in the future.

Conclusion

Email plays an important role in our personal and professional lives. But without specialized tools email can become a huge burden and a waste of time. We designed Shortwave out of this necessity to make emails both productive and enjoyable.

Shortwave is packed full of features that are designed to save time and effort. I wanted to keep this post at least somewhat short, so I really had to cut down on the number of features I wanted to highlight. I will follow up with more Shortwave features and tips in a future post. In the meantime, if you have a Gmail account, please give Shortwave a try, and let us know what you think.

New user FAQs

If this is the first time you’re hearing about Shortwave, you most likely have many questions before wanting to give Shortwave a try. Take a look at Shortwave docs that should answer a lot of them. I’ve also included some personal notes below for easy reference — these are questions that I personally would ask, if I were just discovering Shortwave. Please let us know (support@shortwave.com) if something is not explained clearly in our docs.

Q1: Why does Shortwave only support Gmail?

With around 1.5 billion users, Gmail enables us to showcase our product to a broad spectrum of potential users, gather feedback and iterate. We do want to support other email service providers in the future. But it may be a while before we get there.

Q2: Is Shortwave free?

Shortwave offers a free tier that contains all the features of the paid tier (see pricing). The only limitation is that your search history is capped at 90 days. I use the free tier on my personal email account. So far the 90 day restriction has not been an issue for me, but YMMV. You can always go to Gmail when you have to search for something older than 90 days.

Q3: Is Shortwave secure?

We take security extremely seriously. We wouldn’t feel comfortable using it for our own work and personal use otherwise. Check this page for more information on the steps we take to ensure your account and data is secured.

Q4: What information in my Google account does Shortwave access?

Shortwave requires access to your emails in order to provide the typical functionalities of an email client (view, triage, send, apply labels etc). Additionally it requires access to your Google contacts in order to support contact auto-completion, and display contact names and photos.

Q5: What if I don’t end up liking it?

You can delete your Shortwave account, and leave anytime. This has no impact on your Gmail account. All your emails, labels and contacts will continue to exist in Gmail as usual. Although, if possible, we would love to hear why Shortwave didn’t work out for you.

Security

Firebase recently introduced App Check, a security feature that protects backend resources from abuse. It is a way to ensure that the requests to your app’s backend resources (e.g. Realtime Database or Cloud Storage) originate from your authentic app, and not from some unknown or malicious source.

Why App Check?

With public BaaS providers like Firebase, it is possible for an attacker to obtain the public credentials of an app, and access certain backend resources of the app. Your users and their data are typically protected from such attacks as long as you have correctly implemented user authentication, and Firebase security rules for your app. But that alone does not prevent an attacker from spamming the backend resources with spurious requests in an attempt to hog your backend resources, or incur unwarranted billing costs for you.

App Check is designed to protect the app developer

This is the problem App Check aims to solve. While Firebase Authentication and security rules protect app users by providing user authentication, App Check is designed to protect the app developer by providing app authentication.

How does it work?

An App Check enabled app first interacts with a platform-specific attestation provider to obtain an attestation of the app’s authenticity. Firebase currently supports the following attestation providers out of the box:

• SafetyNet for Android
• DeviceCheck and App Attest for iOS
• reCAPTCHA v3 for Web

Once an attestation has been obtained, the app sends it to the Firebase App Check service, which further verifies the attestation against some app-specific parameters configured by the developer. If the attestation is recognized as authentic, the App Check service will issue a special App Check token for the app. The Firebase SDKs cache this token, and attach it to every backend request made by the app. Then you can activate App Check access control enforcement in the Firebase backend services like Realtime Database, Firebase Storage and Callable Functions so that they will only accept requests with valid App Check tokens attached to them.

This sounds like a handful, but as a developer you don’t really have to do much to start using Firebase App Check. A lot of the heavy lifting is handled by the Firebase SDKs, and the backend services. In most cases all you have to do is configure a few things in the Firebase console, and add a bit of new code to your app. Listing 1 shows all the extra code you have to write to add App Check support with reCAPTCHA v3 to a web app.

You also have the option to enable App Check in you app without activating App Check access control enforcement in the Firebase backend services (i.e. backend services will continue to accept all requests as usual). This is useful since App Check access control enforcement has the potential to break older versions of your app. The Firebase console provides detailed metrics about the sources of your backend traffic: it shows you how many requests came from authentic clients, and how many came from outdated, unknown or potentially malicious clients. You can choose to activate access control enforcement based on this information. In most cases you would at least want to wait until a large majority of your users have upgraded to the latest App Check enabled version of your app before starting enforcement.

Note that the level of security provided by App Check depends on the attestation provider. Providers like reCAPTCHA v3 can only attest that a given request came from an authentic instance of your app. But providers like App Attest and SafetyNet can go further, and also attest to whether a given request originated from an untampered device.

Protecting your own backend services with App Check

Firebase App Check helps protect your Realtime Database, Firebase Storage and Cloud Functions resources. However, if your app depends on any custom backend services of your own, you can protect those services with App Check too. For example, suppose you have implemented and deployed the backend service shown in listing 2 at api.myserver.com.

Here, callExternalService() might be an API call to a third party that you get billed for. You can call the above service from your web application as shown in listing 3.

Like most services accessible over the Internet, your service at api.myserver.com is also vulnerable to a wide range of phishing and billing attacks. An attacker can flood your endpoint with unnecessary traffic, which will slow your service, and incur a large bill due to all the additional invocations of callExternalService(). You can protect your service against this type of attacks with Firebase App Check. App Check can help you make sure that all incoming requests to your service are coming from your authentic web application, and nowhere else. Here’s how.

You can protect your own service against attacks with Firebase App Check

First, you need to change the client application so that it sends an App Check token along with the requests made to your backend endpoint. Listing 4 shows how to obtain an App Check token from the Firebase Web SDK, and include it in the outgoing request as a header.

Here we have chosen to send the App Check token in a custom header named X-Firebase-AppCheck, but you can choose some other way to convey that information if you like. Just make sure to encrypt all communications with TLS, and don’t expose the tokens anywhere during transit.

Next, we modify the backend service implementation to validate the App Check token using the Firebase Admin SDK. Listing 5 illustrates how you can do that.

The appCheckVerification middleware gets executed before our actual business logic. This middleware extracts the App Check token from the request header, and verifies it using the Admin SDK. If the token is recognized as valid, we continue to process the request. If the token is missing or otherwise invalid, we halt processing the request, and send back a 401 Unauthorized response.

This simple mechanism can be used in any backend service that your Firebase apps depend on. The net result is that your backend service will become restricted to your authentic apps, and nobody else. App Check token verification is a rather efficient operation, and can be included in most backend request processing flows. The JWKS objects required to perform the verification are downloaded once, and cached by the Admin SDK for several hours at a time. Therefore after the initial cold start of the SDK, the App Check token verification can proceed entirely as a local operation without making any additional RPCs.

Why does this look familiar?

If you have used Firebase Auth in the past, and have implemented backend services that require ID token verification, you will see some parallels here. Conceptually, ID token verification and App Check token verification are quite similar. But they serve very different purposes.

Firebase Auth ID token verification is a way to establish user identity, and restrict a service to a specific group of authenticated users. App Check token verification is a way to establish app identity, and restrict a service to a specific set of apps. Depending on your use case you may need either one or both in your backend services. Here are some example use cases:

• Only the users with the moderator role should be able to access my service. I don’t care which apps they use to interact with my service: This is a case for user authentication.
• Only the users on my XYZ app should be able to access my service. I don’t care what roles and permissions they have on the app: This is a case for app authentication.
• Only the users with the moderator role coming from my XYZ app should be able to access my service. Non-moderators on XYZ, and moderators logged into apps other than XYZ are not allowed: You need both user and app authentication here.

Conclusion

In other words, App Check is not a replacement for user authentication and security rules. Rather, it is a new dimension of application security that complements the already existing Firebase security tools and practices. As the threat of cyber attacks continues to intensify, developers should be more vigilant than ever, and implement security mechanisms that protect both the app users, and the backend resources that are critical for the continued operation of the apps.

Firebase App Check: A New Dimension in App Security was originally published in Firebase Developers on Medium, where people are continuing the conversation by highlighting and responding to this story.

As developers we often come across situations where we’d like to see all the remote API calls made by a certain application. This requirement often comes up during development and testing, but there are times when you might want to see this information for production applications too.

In this article we explore a couple of simple tricks for logging the HTTP requests made by applications written in Node.js or Python. These techniques do not require any specific logging framework, and can be managed entirely within the application code without too much effort. I’m going to use the Firebase Admin SDK as an example library in my code samples, but the techniques described here should work with most, if not all Node.js and Python libraries. Let’s start with Node.js.

Node.js

One of the easiest ways to log all the HTTP requests made by any Node.js application is to wrap the core HTTP module of the Node.js runtime. To be more specific, we replace the http.request() (or https.request()) method of Node.js with a custom function that logs the outgoing request, and delegates the request back to the original method. Listing 1 shows what this looks like in practice.

We enable HTTP request logging by calling enableRequestLogger() in our code. This function handles the actual wrapping of the https.request() method. Note how it logs some of the request options to the console, before calling the original method. Executing the above program produces an output similar to the following:

2020-10-25T00:03:54.629Z POST https://accounts.google.com/o/oauth2/token
2020-10-25T00:03:54.789Z POST https://fcm.googleapis.com/v1/projects/my-project/messages:send

This output indicates that the Admin SDK has made two requests during execution of the above code — one to the OAuth2 token server, and the other to the Firebase Cloud Messaging (FCM) service. The implementation in listing 1 is pretty basic, and logs only the request method and URL for each outgoing request. It is certainly possible to come up with a more sophisticated implementation that logs both requests and responses, including the message headers and payloads if you like. There are even third party libraries like http-debug and global-request-logger that provide these capabilities out of the box, so you don’t have to implement custom wrappers for http.request() yourself.

Note: As of October, 2020 global-request-logger doesn’t work for HTTPS interactions. But their implementation is pretty straightforward, and serves as a good reference if you ever decide to implement a custom request logger for Node.js.

Python

Many Python applications use the requests library to make remote API calls, which in turns uses Python’s http.client. Therefore you can simply turn on the debug logs at the http.client level to log request headers, payload and response headers (but not the response body) for each HTTP interaction. Listing 2 demonstrates this approach.

We enable debug logging by setting the HTTPConnection.debuglevel property to a value greater than 0. Executing the above program results in the following logs:

send: b'POST /token HTTP/1.1\r\nHost: oauth2.googleapis.com\r\nUser-Agent: python-requests/2.24.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\ncontent-type: application/x-www-form-urlencoded\r\nContent-Length: 1127\r\n\r\n'
send: b'assertion=eyJ....'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: application/json; charset=UTF-8
header: Vary: Origin
header: Vary: X-Origin
header: Vary: Referer
header: Content-Encoding: gzip
header: Date: Sun, 25 Oct 2020 20:46:43 GMT
header: Server: scaffolding on HTTPServer2
header: Cache-Control: private
header: X-XSS-Protection: 0
header: X-Frame-Options: SAMEORIGIN
header: X-Content-Type-Options: nosniff
header: Alt-Svc: h3-Q050=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
header: Transfer-Encoding: chunked

send: b'POST /v1/projects/my-project/messages:send HTTP/1.1\r\nHost: fcm.googleapis.com\r\nUser-Agent: python-requests/2.24.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nX-GOOG-API-FORMAT-VERSION: 2\r\nX-FIREBASE-CLIENT: fire-admin-python/4.4.0\r\nauthorization: Bearer ya29.c.KqYB4gesy_EQO8hL3r_RFC3tSSA_6of8JuN8cvgFy78FwI1HFpSlJmsrHHivx1Dd_lDzwY8tbrfpK4MLmTVNO-e0RUm7H4y9siwGUPjyOmTr4TTQzDLDhANiy4oJaKJt77OOgIp3gCiyxNcsl9YvQskVca1iHlXParHjsduJYgzvWCC-CsYEFe4O4Jz6H9ti8f3smfuoN8lhJrL4fxrg713hZ_lM5Fj7bQ\r\nContent-Length: 73\r\nContent-Type: application/json\r\n\r\n'
send: b'{"message": {"notification": {"title": "Hello world!"}, "topic": "test"}}'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: application/json; charset=UTF-8
header: Vary: Origin
header: Vary: X-Origin
header: Vary: Referer
header: Content-Encoding: gzip
header: Date: Sun, 25 Oct 2020 20:46:44 GMT
header: Server: ESF
header: Cache-Control: private
header: X-XSS-Protection: 0
header: X-Frame-Options: SAMEORIGIN
header: X-Content-Type-Options: nosniff
header: Alt-Svc: h3-Q050=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
header: Transfer-Encoding: chunked

We are seeing the same two HTTP requests as before, but this time logged with more details. For a slightly less verbose output, you can try enabling debug logs at the urllib3 package level. This package serves as an intermediary between requests and http.client. Listing 3 shows how this can be achieved.

Notice the use of Python’s built-in logging package. This is essential to see the log output from urllib3. The output of listing 3 looks something like this:

DEBUG:google.auth.transport.requests:Making request: POST https://oauth2.googleapis.com/token
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): oauth2.googleapis.com:443
DEBUG:urllib3.connectionpool:https://oauth2.googleapis.com:443 "POST /token HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): fcm.googleapis.com:443
DEBUG:urllib3.connectionpool:https://fcm.googleapis.com:443 "POST /v1/projects/my-project/messages:send HTTP/1.1" 200 None

Refer to the Python documentation for more details about the logging package, and the various configuration options it provides. You can also use the techniques shown in listing 2 and listing 3 together in the same application for a highly verbose log output.

Conclusion

We looked at some simple techniques for logging the HTTP requests made by Node.js and Python applications. We used the Firebase Admin SDK as an example in our code snippets, but our approach generalizes well to a wide range of Node.js and Python libraries. Developers can use these techniques to simplify testing applications and debugging complex issues. We can also use them in production deployments to gain more visibility into the internal workings of an application. However, keep in mind that extra logging almost always comes with a performance penalty, so only use logging when necessary, and in the right amount.

Firebase Admin SDK contains APIs for fetching and deleting individual user accounts. These APIs enable developers to fetch Firebase Auth user accounts by their uid, email or phone number, or delete a selected user account by its uid. Developers have been using these APIs to implement tools, automation scripts and even Cloud Functions that perform complex user management tasks. Listing 1 shows a Node.js script that uses the Admin SDK to fetch a user account by email address, and delete it.

While these APIs are useful in a wide range of use cases, sometimes we need to implement even more powerful backend applications that deal with multiple user accounts at once. As a developer you may wish to fetch several user accounts as a batch, and perform some administrative operations on them. Or you may have been running some experiments with Firebase that created a bunch of test accounts, and now you wish to delete all of them.

Implementing such bulk operations using the existing getUser() and deleteUser() APIs is rather cumbersome. The end result is also quite slow to execute since each API call results in a new RPC call being made. You can make things a little better by using the listUsers() API in your code, but that’s not a great solution either as it doesn’t allow you to filter the user accounts that are returned. As a result many developers have been asking for a simpler API to fetch and delete Firebase Auth user accounts in bulk. If you are one of those developers, I have some really good news for you!

Starting from v8.12.0 release of the Firebase Admin Node.js SDK, developers have two new APIs — getUsers() and deleteUsers() — available to them. As evident by their names, these APIs can be used to fetch and delete more than one user account at a time. The getUsers() API accepts up to a hundred user identifiers, and returns the matching user accounts. The user identifier could be a Firebase uid, email address, phone number or an IdP-assigned uid. Similarly the deleteUsers() API accepts up to a thousand Firebase uids, and deletes those user accounts from the project. Both APIs only make a single RPC to the backend, and therefore are more efficient than calling the old getUser() or deleteUser() APIs in a loop. Listing 2 shows an example of how these APIs can be used in practice.

This example program looks up several user accounts, and goes over the results to mark the already disabled accounts for deletion. Then it uses the deleteUsers() API to delete all the marked user accounts. This example makes at most 2 RPC calls to the Firebase Auth backend despite the number of user accounts fetched or deleted.

One caveat to note is that the deleteUsers() API is rate limited to 1 call per second. This might change in the future, but at least for now this means you can only delete at most 1000 accounts per second using this API.

I hope this addition simplifies the development of many advanced user management features. Please try it out and provide feedback via GitHub. More code samples and language support (Java, Python, Go and C#) are expected soon.

Firebase: Fetching and deleting user accounts in bulk was originally published in Firebase Developers on Medium, where people are continuing the conversation by highlighting and responding to this story.

Cloud Firestore is Google’s highly scalable, NoSQL database for mobile and web applications. On top of a document-oriented data model, Firestore offers an array of powerful features including realtime event listeners, atomic transactions, and offline support.

Just because it is another NoSQL database, many developers assume that you cannot enforce any integrity constraints on Firestore data. But that is not entirely correct. In this article we look at Firestore’s data model, the constraints we can apply on Firestore data, and the programming semantics that result from such constraints. We also explore how to change the Firestore data constraints over time without disrupting the applications that depend on the data.

Schemaless? Doesn’t have to be.

Firestore and other similar NoSQL databases are typically dubbed schemaless databases. But as Martin Kleppmann points out in the book Designing Data-Intensive Applications, this term is bit of a misnomer. Any application that queries data usually assumes some kind of a structure. Without a structure it’s impossible to implement any meaningful queries or application logic to interact with the data. Consider the code fragment in listing 1 as an example.

Listing 1 queries Firestore for all cities with a population over one million. Results are then unmarshalled into the custom City class. The City class outlines the implicit structure developer expects to be present in the documents read from Firestore. The query itself assumes there’s a numeric population field present in each document. This demonstrates how there’s always some notion of a schema in play when writing code to interact with data. The term schemaless is often just an indication of whether the underlying database enforces it or not.

Kleppmann uses the terms schema-on-write and schema-on-read to disambiguate this idea. Traditional relational databases are schema-on-write systems, in which every write operation is verified against a predefined schema. Firestore is usually a schema-on-read system where there’s an implicit structure present in the data, which is only interpreted when the data is queried by an application.

Schema-on-read systems are generally more flexible and adaptable to the variety in real-world data than schema-on-write systems. Suppose you defined your city name column to be VARCHAR(32) in a relational database, only to come across a city name 58 characters long! Your only options are to either store a truncated city name or perform a potentially expensive schema alteration. Schema-on-read systems handle such edge cases much more gracefully, often without any effort on the developer’s part. However, app development is usually about guarantees, and schema-on-write systems undeniably give stronger guarantees about the structure, correctness and completeness of the data.

Firestore is typically a schema-on-read database. Security rules allow adding schema-on-write semantics to Firestore.

Thankfully, Firestore doesn’t lock you into the realm of schema-on-read. Security rules enable you to implement schema-on-write semantics in your Firestore databases. This prevents applications from writing malformed or incomplete data to Firestore, which in turns gives applications stronger guarantees about the data when querying. Listing 2 shows an example rules configuration that we can use to validate the city documents. This configuration requires every city document being written to have a population field with a positive numeric value, and a name field with a non-empty string value.

It is highly advisable to implement data validation rules similar to listing 2 in every production Firestore database instance. You do sacrifice some of the flexibility in the process, but allowing applications to write arbitrary-structured data is almost never a good idea. More variety across your documents make querying your database that much harder. Validation rules can also help you detect subtle data mutation bugs in the application code, and prevent malicious clients from corrupting the data.

Flexible typing

Firestore allows you to specify validation rules that are flexible regarding the types of your document fields. Take listing 3 for example, which effectively assigns the Integer type to the population field.

Since our rules make use of the int() function, clients may also write float or string values to the population field without violating the schema constraint. That means all of the following values are accepted in the population field:

• Integer: 1000
• Float: 1000.0
• String: '1000'

This is in stark contrast to schema-on-write systems like SQL where an INT column is strictly constrained to integer values. This flexibility around typing can be quite useful when your data is written by a wide range of clients, on different programming languages and platforms. Firestore provides similar type conversion functions for other primitive types — string(), float(), bool().

However, note that whenever you allow the type of a field to be flexible, any application logic that queries the documents should be prepared to cope with the resulting variety of values. Listing 4 shows some JavaScript code that reads and displays a city document on a web page.

The display() function is type-agnostic, and simply adds the values to the HTML output. In contrast, the closeToMillion() function performs some arithmetic on the population values. Since our type-flexible schema allows string values like '1000' in the database, we need to perform an explicit type conversion before applying the + operator. This is a good example of the trade-off between flexibility and guarantees. Increased flexibility leads to weaker guarantees regarding the form of the data, which in turns requires us to manage it in the application code.

Data constraints

Schema-on-write systems typically enforce four main types of integrity constraints on data. The following list outlines these constraint types along with how they are supported in SQL.

1. Domain integrity: Types (INT, VARCHAR etc), Not-null constraints, check constraints
2. Entity integrity: Primary key constraints
3. Referential integrity: Foreign key constraints
4. Key integrity: Unique key constraints

Following SQL statement demonstrates how these different integrity constraints are used in practice.

CREATE TABLE PERSON (
  UID VARCHAR(128) NOT NULL PRIMARY KEY,
  AGE INTEGER CHECK(AGE >= 18),
  EMAIL VARCHAR(256) CHECK(EMAIL LIKE '%___@___%'),
  SCREEN_NAME VARCHAR(64) NOT NULL UNIQUE,
  COMPANY_ID INT,
  FOREIGN KEY(COMPANY_ID) REFERENCES COMPANY(COMPANY_ID)
);

Firestore security rules naturally lend themselves to expressing domain integrity constraints. Document ID is the closest thing to an entity integrity constraint in Firestore, as it is guaranteed to be unique within a collection, and acts as a natural collection-wide primary key. Referential and key integrity constraints are not as obvious to implement in Firestore, but certainly not impossible. Listing 5 shows an attempt at implementing all the semantics of the above SQL statement with Firestore security rules.

Notice that we are using the DocumentReference data type for the company field, which manifests as a document path during rules evaluation. We can simply lookup this reference using either exists() to enforce the referential integrity constraint. The unique key constraint is enforced by mapping each screen name to a document in a separate screenNames collection. Each of these documents contain a uid field which can be cross referenced against the document IDs in the users collection.

You should consider the cost of implementing security rules of this form. Due to our use of get() and exists() functions, each write requires 2 additional reads for rules enforcement. These reads do count towards your Firestore API quota, and are billed accordingly. Depending on how often the users collection is updated you may or may not be willing to bear the extra cost. But the technology does support it.

Also keep in mind that what’s good in the relational+SQL world may not be idiomatic in the Firestore world. Therefore consider your application requirements carefully, and map them to a document-oriented data model that is both natural and cost-effective to your workload. Your requirements may cover a broad range of topics including the volume of reads and writes, read latency, data access patterns, realtime listeners vs one-off queries, battery life of clients and more. Think about the flexibility and guarantee trade-offs as well. You need some constraints to prevent writes that can render your data meaningless, but not so many that they become a hindrance.

There are many helpful references on the subject of Firestore data modeling from the Firebase team. Here’s a talk from Todd Kerpelman to get you started.

Schema evolvability without data migration

As your application changes over time, your schema will also have to adapt and evolve. An interesting feature of Firestore security rules is their ability to change without forcing an expensive data migration. To demonstrate this, notice that listing 2 limits the city names to 32 characters (similar to VARCHAR(32) in a relational database). If you later decide to increase this limit to 64 characters, you can do so by simply updating your rules configuration. Your existing data will remain intact, and none of the application code needs to be updated.

You can always update your Firestore data validation rules without touching the data itself. However, note that if your schema change is not backward compatible, it can leave your old data in an inconsistent state. For example suppose we have a description field in the city documents that is 128 characters long. If we later decide to lower this to only 100 characters, the existing documents with longer descriptions go out of compliance with our latest rules. The applications can continue to read these documents as long as they don’t make any assumptions regarding the field length. But any attempts to update the field will fail if the updated value is longer than 100 characters.

In general you should keep schema changes to a minimum, regardless of their potential impact. Carefully evaluate your schema choices before you implement them, and once implemented plan to stick with them. But if you ever have to implement a schema change, make sure it is at least backward compatible. That is any newly written code should not encounter problems while reading or writing old documents. However, this is also only a necessary requirement. Depending on the nature of the schema change, backward compatibility may not be sufficient to ensure continued operation of your apps.

Backward compatibility may not be enough

Making sure that a planned schema change is backward compatible is usually easy enough. You can always write new application code to gracefully handle any documents that were written before the last schema change. What is complex is planning for forward compatibility — i.e. making sure the old application code can read and write new documents that were validated against a newer schema.

This is a particularly pressing concern for mobile apps, since the app developer has no control over when the users upgrade their apps. For instance you may have just released v2 of your app along with a related set of schema changes. But if you were not careful, the users who are still on v1 may fail to read or write the data generated by the new version of the app. Hence, if you are a mobile app developer you need to make sure that each schema change you roll out is both backward and forward compatible.

Adding and removing fields

In time you may wish to add new fields to your documents or remove existing fields that are no longer of use. As long as the fields being added or removed are optional, such changes are both backward and forward compatible. When we say optional, that means both the Firestore security rules and the application code should treat the fields as optional. Provided this condition is satisfied, the old versions of the app will simply ignore any new fields, and will not break when a familiar field is not present in the newly written documents. Similarly, new versions of the app will not break when they cannot find a new field in the old documents. Once you can gracefully handle addition and removal of individual fields, you can also easily deal with renaming of fields.

It is interesting to consider what would happen if a new required field is added, or an existing required field is removed. These changes disrupt application usage in slightly different ways. Adding a new required field can prevent old versions of the app from writing documents. As your old code is not aware of the new field, it fails to compose documents that satisfy the latest data validation rules. On the other hand removing a required field can prevent old versions of the app from reading the documents written by the new versions of the app. Recall that in both cases we can manage to maintain backward compatibility with a bit of clever programming. It is the forward compatibility that usually suffers in these situations.

As an example, suppose we renamed the required field surname to familyName in a collection. This is same as removing the surname field, and adding the familyName field. Listing 6 shows how a new, backward compatible version of the app can gracefully handle this change.

Now consider listing 7, which is from an old version of the app. This code fails to create new documents as it is not aware of the now required familyName field. Furthermore, it cannot read the documents created by new code (listing 6), as they do not contain the surname field.

We can mitigate the issue of writes by relaxing our validation rules to allow either familyName or surname fields in the documents. This is shown in listing 8.

But resolving the issue of failing reads is tricky if we hadn’t planned for such changes from the beginning. In case of listing 7 we should have at least programmed our app more defensively, using sensible default values for each field wherever possible (data classes with default field values is a good way to implement this). This of course doesn’t solve all of our forward compatibility woes. But it at least prevents total failures in the old versions of the app, as the new rules and the corresponding code roll out.

To retain full forward compatibility through our field rename we can keep adding both familyName and surname fields to each document created by new versions of the app. We can do this in listing 6 itself, and we will have to keep this duplication up until we are confident that all users have migrated off the old version of the app.

Retaining forward compatibility with Remote Config

In this final section we look at a more elaborate solution for implementing forward compatible schema changes. This is not a retroactive fix. Rather you had to have implemented this from the beginning in anticipation of future schema changes. Most developers won’t need something like this. Nevertheless it is an interesting technique, and a powerful application of Firestore and Firebase Remote Config.

To make this work we should add a schema version field to all of our documents. In the following example this field is just called v. Suppose that each version of our app knows how to read documents of a specific schema version. We can also assume that the application code is backward compatible— i.e. it can also read documents of older schema versions. If the application encounters a document with a newer schema version, it consults Remote Config for instructions on how to interpret the document. Consider the following two documents for example.

people
  |
  +---- person1
  |        +-- v: 1
  |        +-- firstName: Peter
  |        `-- surname: Parker
  |
  `---- person2
           +-- v: 2
           +-- firstName: Carol
           `-- familyName: Denvers

Notice that the two documents have schema versions 1 and 2 respectively. The surname field has been renamed to familyName in the new version. We specify this schema change as a Remote Config parameter named v1_to_v2 with the JSON value {"surname":"familyName"}. Now whenever our V1 application code encounters a V2 document, it can consult Remote Config to figure out how to interpret the data. Listing 9 illustrates this implementation.

This correctly displays the values from both documents, even though the surname field is not present in one of them. The getField() method correctly resolves surname field to familyName when reading the V2 document.

D/MainActivity: Field resolved: surname --> familyName

This solution can transparently deal with any renamed field. It is also quite efficient since the schema mappings can be fetched from Remote Config once, and cached in memory. If necessary this solution can be easily extended to support other more complex schema changes by incorporating more mapping instructions to the JSON parameter value stored in Remote Config. For example, here’s a hypothetical schema change where a fullName field has been split into two firstName and lastName fields:

{
  "fullName": {
    "changeType": "split",
    "fields": ["firstName", "lastName"],
    "separator": " "
  }
}

This tells the reader to fetch the firstName and lastName fields from the new schema, and concatenate them together to obtain the fullName value of the old schema.

Conclusion

Cloud Firestore is a NoSQL schema-on-read database. But security rules enable us to implement powerful schema-on-write semantics on top of it. With the right combination of rules, Firestore supports implementing all major types of data integrity constraints, resulting in stronger guarantees for the application developers. These strengthened guarantees usually come at the price of reduced flexibility, and increased data validation costs, and therefore should only ever be implemented in order to meet specific application goals. Implementing an overly rigid structure can make testing and long-term maintenance harder for developers.

With integrity constraints also comes the need to revise those constraints to meet changing application needs — i.e. evolvability. Developers must keep backward compatibility in mind when adding, removing or otherwise moving document fields around. But if you are a mobile app developer who maintains several versions of the same app, then you might also have to think about forward compatibility. That is your old versions of the app should be able to read and write the documents that adhere to a newer schema. We discussed a few simple guidelines to follow that help ensure this property, as well as a somewhat advanced solution based on Remote Config that allows developers program forward compatibility into their applications.

Cloud Firestore: On data constraints and evolvability was originally published in Firebase Developers on Medium, where people are continuing the conversation by highlighting and responding to this story.

Google Cloud Identity Platform (GCIP) allows you to add Google-grade identity and access management support to your own applications. Enterprises can use GCIP to manage the identities of their employees, customers, partners and IoT devices at scale. In November 2019 Google further extended GCIP with support for tenant management. This new feature enables enterprises to define multiple tenants within a single instance of GCIP. Each user account can be assigned to a tenant, and authenticated using a collection of tenant-specific identity providers.

As an example consider an automobile manufacturer who’s partnered with at least two distributors. The manufacturer offers an array of online inventory management, order processing and payment applications to the distributors. However, as often is the case, the distributors use different SAML identity providers to manage the identities of their own employees and systems. So how do we go about securely exposing the manufacturer’s applications to the distributors?

GCIP multi-tenancy support is ideal for implementing this type of business-to-business (B2B) integrations. The auto manufacturer can use GCIP, and define separate tenants for each distributor. Within each tenant the manufacturer can specify different SAML configurations that would enable employees of the distributor companies to authenticate with manufacturer’s applications. Furthermore, if the manufacturer wishes to enable additional authentication methods (e.g. Google or Facebook sign-in) for selected tenants, GCIP supports that too.

Developers that use GCIP to implement multi-tenant applications can use the Firebase Admin SDKs to create and manage their tenants programatically. This is useful for developing backend services, automation scripts and customer on-boarding flows that need to define and access tenants in an automated fashion. So far the required APIs have been added to the Node.js and Golang variants of the Admin SDK, with more language support expected in the future. Listing 1 shows how to create a new tenant with a tenant-specific email sign-in configuration in Go.

Once a tenant has been created, you can create, update, retrieve and delete users within that tenant. Listing 2 shows some of the corresponding Go APIs.

You can also define SAML and OIDC identity providers for each tenant using the Admin SDK.

Listing 4 shows how to list or iterate over all the tenants defined in a single GCIP instance. This is useful when developing high-level administrative tools for managing all the B2B integrations from a single central point.

But what about the client-side development interface? How would our auto manufacturer add authentication support to their apps, so the distributors can sign-in as separate tenants? You can use the Firebase Web SDK to implement tenant-aware user authentication flows in your client-facing web apps. As shown in listing 5, you just need to specify the target tenant ID before invoking the usual Firebase authentication logic.

Multi-tenancy is a critical feature for many large enterprises who wish to expose their apps to different customer and/or partner organizations. GCIP and Firebase make developing multi-tenant applications easier by providing a wide range of server-side and client-side APIs. Refer to the documentation and the code samples for more details, and see if GCIP can help simplify your B2B integration story.

Multi-tenant applications with Firebase and Google Cloud was originally published in Firebase Developers on Medium, where people are continuing the conversation by highlighting and responding to this story.

Using the Firebase Admin SDK for .NET in Google Cloud Run

Recently I was reading Patrick Martin’s article on deploying a C# backend in Google Cloud Run, and I figured it would be fun to write a follow up piece featuring the Firebase Admin SDK for .NET. Hence, in this post I show how to implement a custom authentication service in C#, and deploy it as a serverless API using Cloud Run. This service validates user credentials, and uses the Admin SDK to generate signed custom tokens that clients can use to sign into Firebase.

Google Cloud Run is one of the latest additions to Google’s serverless portfolio. It enables developers to deploy any Docker container as a serverless application in the cloud. This means you no longer have to code your serverless apps using a specific set of programming languages or APIs. Any application can be deployed in Cloud Run as long as it is:

1. stateless,
2. has an HTTP interface, and
3. can be packaged as a Docker container.

Patrick’s article clearly outlines how to deploy any generic ASP.NET Core application using Cloud Run. The basic idea is to develop the app using the .NET Core and the dotnet command-line utility, and then package it into a Docker image. Microsoft provides official .NET Core Docker images that can be used as the basis for this. Finally, the Dockerized application is deployed to Cloud Run using the gcloud tool. You can use the exact same steps to package and deploy an application that uses the Firebase Admin SDK for .NET. Therefore we will mainly focus on the additional steps you need to follow when using the Admin SDK.

First, you need to add the Admin SDK dependency to your project. Having initialized your .NET web project, you can add the Admin .NET SDK to it by running the following command.

$ dotnet add package FirebaseAdmin --version 1.6.0

Version 1.6.0 is the latest available at the time of this writing. Check the Admin SDK documentation to see if there’s a more recent version available.

Next, we need to initialize the Admin SDK. We can do this in the Startup class of the web application as shown in listing 1.

Note that we are not passing any arguments to the FirebaseApp.Create() method. When initialized this way, the Admin SDK will automatically discover the required authorization credentials from the runtime environment. This mechanism works out of the box on all application runtimes managed by the Google Cloud Platform including Cloud Run.

Our Startup class also engages the MVC support in ASP.NET, which means we will be implementing our request processing logic in a controller class. Listing 2 shows what this looks like.

Our controller is programmed to respond to POST requests on the /login URL. It accepts a JSON payload containing custom user credentials. We then query our custom user store to validate the credentials sent by the client. The UserStore class is just a stand-in for the actual database or the API where your app’s (or organization’s) user information is stored. In our example it just contains two fake user accounts — alice and bob — with hard-coded credentials.

If the credentials are valid, we use the FirebaseAuth API of the Admin SDK to create a signed custom token. The client (usually a mobile or web app) can then use this custom token to complete the Firebase authentication flow.

Full source code of this web application is available in GitHub. To test it locally, download a service account JSON file from your Firebase project, and set it to your environment.

$ export GOOGLE_APPLICATION_CREDENTIALS=path/to/service-account.json

Now run the following command to build and launch the app.

$ dotnet run

This starts a web server on local port 5000. You can send some requests to the service to verify its behavior. If all goes well, the application responds back with a signed token when valid credentials are presented.

$ curl -v -X POST -d '{"username": "alice", "password": "password"}' -H "Content-type: application/json" localhost:5000/login
> POST /login HTTP/1.1
> Host: localhost:5000
> User-Agent: curl/7.47.0
> Accept: */*
> Content-type: application/json
> Content-Length: 45

< HTTP/1.1 200 OK
< Date: Sat, 08 Jun 2019 21:56:52 GMT
< Content-Type: application/json; charset=utf-8
< Server: Kestrel
< Transfer-Encoding: chunked
< 
{"customToken":"eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9......

All other requests fail with HTTP 401 Unauthorized responses.

$ curl -v -X POST -d '{"username": "alice", "password": "incorrect"}' -H "Content-type: application/json" localhost:5000/login
> POST /login HTTP/1.1
> Host: localhost:5000
> User-Agent: curl/7.47.0
> Accept: */*
> Content-type: application/json
> Content-Length: 46

< HTTP/1.1 401 Unauthorized
< Date: Sat, 08 Jun 2019 21:58:51 GMT
< Content-Type: application/json; charset=utf-8
< Server: Kestrel
< Transfer-Encoding: chunked
< 
{"error":"Incorrect username or password"}

At this point we can go ahead and deploy the application to Google Cloud Run. Again I refer you to Patrick’s article for the relevant instructions. The deployment process mainly involves creating a Dockerfile that describes how to build and package our code as a Docker image. Then we kick off a new build in Google Cloud Build which takes our code and Dockerizes it. The resulting image is saved to the Google Container Registry. Finally, we deploy this Docker image to Cloud Run to be executed as a serverless application.

$ export PROJECT_ID=my-gcp-project-id
$ gcloud config set project $PROJECT_ID
$ gcloud builds submit --tag gcr.io/${PROJECT_ID}/custom-auth-service 
$ gcloud beta run deploy --image gcr.io/${PROJECT_ID}/custom-auth-service

Now our application is deployed and ready to serve traffic. The final deployment command displays the Cloud Run endpoint that accepts incoming requests.

Please specify a region:
 [1] us-central1
 [2] cancel
Please enter your numeric choice:  1

Service name (custom-auth-service):  
Allow unauthenticated invocations to [custom-auth-service] (y/N)?  Y

Deploying container to Cloud Run service [custom-auth-service] in project [solarflares-f4bee] region [us-central1]
✓ Deploying new service... Done.                                                                                                                         
  ✓ Creating Revision...                                                                                                                                 
  ✓ Routing traffic...                                                                                                                                   
  ✓ Setting IAM Policy...                                                                                                                                
Done.                                                                                                                                                    
Service [custom-auth-service] revision [custom-auth-service-00001] has been deployed and is serving traffic at https://custom-auth-service-example-uc.a.run.app

However, there are couple more things we need to do before we can run a successful trial. Remember that we have set up the Admin SDK to automatically discover credentials from the Cloud Run environment. It will also use the Google Cloud IAM service to sign custom tokens. Therefore we need to complete the following one-time configuration steps to ensure successful operation of our code.

1. Enable the Cloud IAM API for your project. This can be done by visiting the APIs & services page in the Google Cloud console.
2. Authorize the Cloud Run credentials to sign tokens. This can be done by visiting the IAM & admin page in the Google Cloud console, and granting the Service Account Token Creator role to the Default compute service account.

Admin SDK documentation provides more details on accomplishing these tasks. Once that’s done, we are ready to try our shiny new serverless app.

$ curl -v -X POST -d '{"username": "alice", "password": "password"}' -H "Content-type: application/json" https://custom-auth-service-example-uc.a.run.app/login
> POST /login HTTP/1.1
> Host: custom-auth-service-example-uc.a.run.app
> User-Agent: curl/7.47.0
> Accept: */*
> Content-type: application/json
> Content-Length: 45

< HTTP/1.1 200 OK
< content-type: application/json; charset=utf-8
< X-Cloud-Trace-Context: d35d7da733bd98697940d035c2de9dbd;o=1
< Date: Sat, 08 Jun 2019 21:15:06 GMT
< Server: Google Frontend
< Content-Length: 752
< Alt-Svc: quic=":443"; ma=2592000; v="46,44,43,39"
< 
{"customToken":"eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9....

In this post we looked at how to use the Firebase Admin SDK for .NET in Google Cloud Run. We showed how to automatically discover Firebase credentials from the runtime, and use them to mint Firebase custom authentication tokens that are signed by the Google Cloud IAM service.

Most serverless platforms restrict developers to specific programming languages and/or APIs. Google Cloud Run on the other hand enables developers to bring all their favorite programming languages, libraries and frameworks to the cloud. Developers can continue to implement their backend components in languages they know and love, and use Cloud Run to deploy them as serverless apps at scale. Moreover, Cloud Run is based on open standards and opensource technologies like Docker and Knative. The end result is a very powerful, convenient and cost-effective platform that developers can use without the fear of vendor lock-in.

We did use a couple of additional Google Cloud services in our development flow — Cloud Build and Google Container Registry. But those are optional. We could just as well build the image ourselves using Docker tools, and upload it to a container registry of our choice. Cloud Run is agnostic of how and where the Docker images are stored, as long as they are accessible.

The Firebase Admin SDK for .NET is still somewhat new, and therefore doesn’t support all the APIs that you will find in the other Admin SDK implementations. But thanks to an enthusiastic open source community, many of the key Firebase Authentication APIs and Cloud Messaging APIs are now available in it. If you’re using Firebase Unity and a fan of .NET, this package (and Cloud Run) might unlock a whole new way of implementing backend components for your Firebase games. Check out the example app as well as the Admin SDK for .NET, and let me know what you think.

Using the Firebase Admin SDK for .NET in Google Cloud Run was originally published in Firebase Developers on Medium, where people are continuing the conversation by highlighting and responding to this story.

Generating email action links using the Firebase Admin SDK

Does this look familiar?

Your new account has been created.

To confirm your email and activate your account, please click the below link:

https://www.fancy.new.service/0bscureC0nf1rmat10nL1nk

If the above link does not work, please copy and paste the link on your web browser.

We’ve all received emails of this nature from various online services. It is a simple yet effective way to interact with the users, and get them to take certain actions. If you are the developer of such an online service, you might also be looking for ways to send emails when new users sign up for your service, or when existing users attempt to change their passwords.

If your application is built using Firebase, specifically Firebase Authentication, then you can easily send emails from your application prompting users to complete certain authentication-related tasks. For example, Listing 1 below shows how a web application can send out an email verification link for a newly signed up user.

It is worth noting that listing 1 is comprised of client-side code that runs in the user’s web browser. The sendEmailVerification() API comes from the Firebase JS SDK. This API, and others like it, make RPC calls to the Firebase Auth service, which then sends out the desired emails using an email service built into Firebase. Similar client-side APIs are available in Firebase SDKs for Android and IOS. The emails sent using these APIs contain auto-generated action links that work out of the box. What’s nice about this approach is that Firebase handles everything for you — generating email action links, placing them in a predefined email template, and sending them out.

However, there might be times when you want more control over this process. You may want to fully customize the content and layout of the emails by including your company logo and other brand information. You might even want to customize the email from user to user based on their country or language. Or you might want to use a different email service like SendGrid, where you have more visibility into the email delivery process. In these types of situations, you just need a way to obtain the email action links from Firebase. Then you can place those links in an email template of your choosing, and send them out using your preferred email service.

If this sounds like what you want, then you should look into the Firebase Admin SDK. A recent series of releases have added several new APIs to the Admin SDK that make it possible to generate Firebase email action links in server-side environments. Specifically, you will find the following new APIs in the Admin SDK:

• generateEmailVerificationLink()
• generatePasswordResetLink()
• generateSignInWithEmailLink()

Listing 2 shows a serverless function that uses the Firebase Admin Node.js SDK to generate email verification links, and sends them out using a third-party library. The function itself is implemented using the Firebase Functions SDK, and it is set to execute on every new user sign up event.

The email action link APIs are now available in the Node.js, Java and Python flavors of the Firebase Admin SDK. These new APIs also allow configuring additional information that control what happens when a user clicks on a link. For example, in Listing 2 above, we indicate that the users should be redirected to the homepage of our web app once they verify their email address. Similarly you can specify whether to open the link in the web browser or in a mobile app. Check out the documentation for more examples, and see how you can incorporate these new APIs to further expand the user management capabilities of your Firebase apps.

If you wish to learn more about Firebase Admin SDKs and the new email action link APIs, check out the following presentation from the recently concluded Google Cloud Next 2019.

In this talk, Lauren Long and I walk the audience through a series of use cases that involve Firebase Auth, Cloud Firestore, Cloud Functions and Admin SDK. In the process we also demonstrate a more elaborate version of the example in listing 2 (jump to the timestamp 28:10 of the video if you’re in a hurry). Enjoy the talk, and hit me up with any questions.

Generating email action links with the Firebase Admin SDK was originally published in Firebase Developers on Medium, where people are continuing the conversation by highlighting and responding to this story.

Since the Firebase Cloud Messaging (FCM) support became available across all flavors of the Admin SDK, many developers have been requesting for an API that enables sending messages to more than one recipient. The original FCM API, which is demonstrated in listing 1, facilitates sending messages to individual devices by addressing each message to a single device registration token.

But developers often come across scenarios where a certain message needs to be sent to multiple devices. I myself ran into this use case when I was working on my cryptocurrency price monitor app. My server-side code that watches for cryptocurrency price updates has to notify users of the price changes, and often it has to notify multiple users. Sending one message at a time is quite inefficient when you have hundreds if not thousands of users to notify.

One way to implement this use case is to employ the pub-sub messaging style. You can subscribe an arbitrary number of devices to an FCM topic, and then address your messages to the topic using the Admin SDK. FCM topics are optimized for throughput, and you can efficiently send messages to millions of devices this way. However, there are still many scenarios where pub-sub may not be the answer. For example in my cryptocurrency app, the set of devices to be notified is different every time. For each price change, my application executes a Firestore query to determine the set of devices that should be notified. FCM topics are not very useful in situations like this where the set of target recipients is dynamic. Moreover, with the pub-sub messaging paradigm you should also think about how to handle and maintain the topic subscriptions over time. This is extra work when all you want is to run a database query, and notify the returned devices.

In order to better support sending messages to multiple FCM recipients, Firebase Admin SDK recently introduced a couple of new APIs. First of these is the sendMulticast() method, which supports sending a message to a list of devices. As shown in listing 2, you can use this API to send a message up to 100 devices using their device registration tokens.

The other new API added to the Admin SDK is the sendAll() method. This method accepts a list of up to 100 arbitrary messages, and sends them all as a single batch. The payload of each message can be different, and they can also be addressed to any combination of device tokens, topics and condition statements. Listing 3 shows how this API can be used to send a customized set of messages, each addressed to a different recipient.

The sendAll() method makes only one RPC call to deliver the entire batch of messages. It’s worth noting that the sendMulticast() API described earlier is also implemented on top of the sendAll() API. The SDK creates a list of messages from the given MulticastMessage object, and calls the sendAll() method under the hood. This is why the upper limit of messages is 100 in both cases (this limit may be increased in the future). For the same reason, both sendMulticast() and sendAll() methods have the same return type — BatchResponse. You can inspect this object to check the success/failure status of each message in a batch. Or you can query the successCount and failureCount attributes of the return value to get a quick summary of the result.

The new FCM APIs are great when you wish to send messages to hundreds or even thousands of users. Listing 4 shows a simple test that uses the send() method and the new sendAll() method to deliver a list of 1000 messages. In each test, we start a set of concurrent tasks, and wait for them to finish. Therefore in case of the send() method, we are not actually sending one message after another. That will obviously be much slower than the sendAll() method, and won’t be a fair comparison. Instead, we measure the time it takes to finish 1000 concurrent send operations.

In my local test environment, the implementation that uses the send() API takes around 2750 ms on average, while the implementation that uses the sendAll() API takes only 830 ms. I also repeated the experiment with the message count increased to 10000. Still the sendAll() API manages to hand off all 10000 messages to FCM in about 1400 ms. But the send() API fails with I/O errors in this case due to the high volume of concurrent RPC calls.

If we keep increasing the number of messages, we will eventually run into I/O errors with the new FCM APIs as well. Therefore, if your app needs to regularly send out messages to millions of users, topic messaging should be the preferred option. Alternatively, you can consider using the new APIs in combination with some local throttling to make sure your runtime doesn’t start more concurrent I/O operations than it can handle. Having said that, these new FCM APIs should still be a huge step up for many applications that only need to send messages to several thousand users at a time.

The new FCM APIs are now available in the Node.js and Java flavors of the Firebase Admin SDK. Check the documentation for more details and examples. If you have any feedback or suggestions related to this feature, please feel encouraged to share them on the Firebase GitHub repos. Happy coding with Firebase!

The Go runtime for Google Cloud Functions was released into beta this month. I’m a huge fan of statically typed languages, and of Golang in particular. Therefore I’m excited about the possibility of developing serverless functions in Go, and deploying them on the Google Cloud Platform (GCP). In this post we look at how to implement a Cloud Function in Go, and set it up to be called from a Cloud Firestore trigger. Then we are going to make things more interesting by throwing the Firebase Realtime Database also into the mix.

Our use case is pretty simple. Suppose we have a movie reviews app. Users post reviews, which are stored in a Firestore collection named movie_reviews. Whenever a new review is posted, we want to analyze it, score the content, and award the author of the review some points. Total scores of users are kept up-to-date in the Firebase Realtime Database. We can certainly implement this use case with just Firestore. But I’m also using the Realtime Database to demonstrate integrating multiple services in the Firebase ecosystem via Cloud Functions. Also I want to try the Firebase Admin SDK in the new Cloud Functions runtime.

Setting up

We need a Firebase project with Firestore and Cloud Functions enabled. If you don’t have one already, just go ahead and create a new Firebase project. Note that this actually creates a GCP project under the hood. Check out the getting started guides for Firestore and Cloud Functions to learn how to enable those services in a new project.

Next, install the Google Cloud SDK in your local development environment so that you have the gcloud command-line tool at your disposal. Also make sure you have Golang 1.11 installed, and that you are able to locally build and run Go programs. Finally, install the Firebase Admin SDK by running the following command:

$ go get -u firebase.google.com/go

This installs the Admin SDK to your GOPATH along with its required dependencies.

Coding the Cloud Function

Start by creating a new directory named scorer somewhere in your local GOPATH. Then create a file named scorer.go in that directory. This file is going to contain all the code we implement for this example. You can find the complete scorer.go file in GitHub.

There are two noteworthy components in our Cloud Function implementation:

1. An init() function that contains package-level initialization logic.
2. A ScoreReview() function that contains the main body of our serverless function.

An init() function in Go is a standard way to implement some one-time initialization logic for a package. Listing 1 shows what the init() function for our scorer package should look like.

Here we initialize the Firebase Admin SDK, and create a new db.Client for later use. Be sure to change the DatabaseURL setting (line 17 in listing 1) to point to your own Firebase Realtime Database instance. As a best practice, you should always reuse the instances of firebase.App and database.Client. Therefore by putting the above code in an init() function, we ensure that it runs only once. Multiple executions of the same Cloud Function instance will use the same db.Client. However, do note that this does not implement a global singleton. There can be multiple Cloud Function instances active at the same time, and instances are started and stopped based on the load.

The exported ScoreReview() function contains the main business logic of our serverless function. Listing 2 shows how it is implemented.

This function receives a Context and a FirestoreEvent as arguments. FirestoreEvent is comprised of all the data passed in by the Cloud Firestore trigger. Specifically, it contains the contents of the Firestore document that triggered the Cloud Function. We run a mock scoring function on the movie review text, and use the previously initialized db.Client instance to update the total score of the corresponding author. We use Transaction() instead of Set() in order to prevent the lost update problem when writing to the Realtime Database.

Declaring dependencies

Before we can deploy our function to GCP, we need to create a Go module file (go.mod) containing the dependencies required by our code. Execute the following commands in the scorer directory to auto-generate the required manifests.

$ go mod init
$ go mod tidy

The go mod command inspects the imports in the *.go files to determine which dependencies are needed to build the code. It generates two new files based on the findings. At this point our project directory looks like this.

scorer/
├── go.mod
├── go.sum
└── scorer.go

Feel free to open and explore the contents of the auto-generated files. The go.mod file lists the Firebase Admin SDK, and the required dependencies of the Admin SDK. The go.sum file contains the versions and checksums of all dependencies, so the dependency tree can be reliably reproduced later.

Go modules are an experimental new feature in Golang 1.11 for facilitating module versioning and dependency management. Based on the go.mod file, Google Cloud Functions will fetch the required dependencies, and build our code in the cloud.

Deploying to the Cloud

Execute the following command from the scorer directory to deploy our function to the cloud. Make sure to replace <PROJECT_ID> placeholder with your own GCP project ID.

$ gcloud functions deploy ScoreReview --runtime go111 \
--trigger-event providers/cloud.firestore/eventTypes/document.create \
--trigger-resource "projects/<PROJECT_ID>/databases/(default)/documents/movie_reviews/{pushId}"

The trigger-event flag indicates that our function should be invoked when new documents are created in Cloud Firestore. The trigger-resource flag specifies the Firestore path that will be watched for document creation events. The way we have set it up, our function will get triggered every time a new document is added to the movie_reviews top-level collection. This is indicated by the wildcard {pushId} at the end of the trigger resource path.

The deployment can take a few minutes. Your code is uploaded to the cloud, where it is built and deployed as a serverless function. If all goes well you should see an output similar to the following.

Deploying function (may take a while - up to 2 minutes)...done.                                                                                          
availableMemoryMb: 256
entryPoint: ScoreReview
eventTrigger:
  eventType: providers/cloud.firestore/eventTypes/document.create
  failurePolicy: {}
  resource: projects/.../databases/(default)/documents/movie_reviews/{pushId}
  service: firestore.googleapis.com
labels:
  deployment-tool: cli-gcloud
name: projects/.../locations/us-central1/functions/ScoreReview
runtime: go111
serviceAccountEmail: ...
status: ACTIVE
timeout: 60s
updateTime: '2019-01-26T23:02:15Z'
versionId: '1'

Trying it out

Use the Firebase Console to create a new Firestore collection named movie_reviews, and add a few child documents to it. Each document should contain at least two fields — author and text. Figure 1 shows what this should look like.

Each addition of a document triggers an execution of our serverless function several seconds later. From the Firebase Console you can observe the user scores being written to the Realtime Database as shown in figure 2.

You can also check the Cloud Functions logs in the GCP console for further confirmation. Figure 3 shows what to expect.

Conclusion

In this article we looked at consuming Cloud Firestore events from a serverless function implemented in Go. We also used the Firebase Admin SDK to interact with the Firebase Realtime Database. The techniques and APIs described in this post can be used to integrate with a wide range of services in GCP and Firebase. Google Cloud Functions facilitate receiving events from Cloud Storage, Cloud PubSub, Firebase Realtime Database and more. The Firebase Admin SDK enables accessing services like Firebase Auth, Firebase Cloud Messaging and more. The availability of a Go runtime for Cloud Functions means developers can now implement serverless functions that use any combination of the above products while enjoying the simplicity, performance and type safety of Go.

Firebase: Developing serverless functions in Go was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.