Background

The original internet architecture, built atop Internet Protocol Version 4 (IPv4), allocated less than 4.3 billion addresses. Every internet connected device needs an IP address, but in the early days of the internet there were no tablets or smartphones and computers were luxuries, not commodities.

Today, there are almost as many smartphones as there are IPv4 addresses (~3.5 billion smartphones at the time of writing in 2020). Numerous measures have been put in place to ration IPv4 addresses, including sharing of addresses across multiple devices within a local network. Since the late 1990’s, IPv6 has made great strides at becoming a long-term replacement for IPv4, offering nearly 3.4×10³⁸ addresses.

IPv6 deployment is steadily underway across many parts of the world, with many telecom companies in emerging markets leading the way. Reliance Jio in India, for example, is well ahead of many American companies at 90% IPv6 deployment. This should not come as a surprise given that India has the second largest total population at more than 1.35 billion. IPv4 simply cannot scale to support nations of that size.

IPv6 and Podcasting

Podcasts are essentially RSS feeds and audio files distributed via podcast hosts on servers around the world. There are two components that need to support IPv6: the client operating system and the podcast host server. Most client operating systems, including Apple iOS and Google Android 5.0+, support IPv6 (although Android still has IPv6-related limitations in 2020).

It is not enough for client operating systems to support IPv6; podcast hosts must also provide support. At minimum, this includes three key components.

• AAAA DNS records
• IPv6-compliant web server
• IPv6 DNS server

IPv6 support usually coexists with IPv4 support in what is called dual-stack delivery, so it is not a situation of IPv6 or IPv4 bur rather IPv6 and IPv4. Developers do not have to choose between the two. For cloud-native hosts, IPv6 support is often as simple as checking a box, and usually comes at no additional cost.

For podcasters, every redirect also needs to support IPv6. That means tracking and measurement services like Podtrac and Chartable need to support IPv6, in addition to the podcast host. Otherwise IPv6-only clients will still fail to resolve a DNS entry and ultimately be unable to listen.

Who supports IPv6?

Few podcast hosts provide public documentation regarding their support, or lack thereof, of IPv6 support. Doing some basic testing on a limited example set, these were my results among popular hosts:

IPv6 Support: Feedburner, Podbean, Buzzsprout, Acast, Omny, AudioBoom, Patreon, Transistor.fm, Apple, Pinecast, and PodcastOne

No IPv6 Support: Libsyn, SoundCloud, Anchor.fm, Podomatic, BlogTalk, Spreaker, Simplecast, Ivoox, Megaphone, Art19, Feedpress, Blubrry, Podtrac, and Chartable

These results may not be fully conclusive, as it is possible for sites to partially support IPv6 only on specific subdomains or within specific paths. However, it seems systematic that many podcast hosts simply do not offer support for distributing content over IPv6.

Some of the largest podcast hosts still do not support IPv6 in 2020, which means many of the most popular podcasts from around the world are only accessible to listeners connecting via IPv4.

Why not support IPv6 support?

The only common case we heard about [for disabling IPv6 support] was internal IP address processing. — AWS FAQs

While the IAB Measurement Guidelines require IP addresses be used as part of a solution to identify unique listens, for reasons of data privacy and security, it is better for businesses to convert IPv4 or IPv6 into one-way hashes, both to anonymize user data and also standardize formatting. i.e. IPv4 and IPv6 addresses are formatted differently, but a hashing function like MD5 processes variable-length inputs into fixed-length outputs.

One notable example of IP address processing is GeoIP look-ups that map IP addresses to approximate physical locations. This is how localized podcasts like NPR’s ‘Consider This’ are able to adapt programming by geography. IP Geolocation is still possible on IPv6, but it is a bit more complicated. Supporting IPv6 would require hosts and analytics services make code modifications to handle GeoIP look-ups for both IPv4 and IPv6 addresses.

Potential Benefits of IPv6

IAB Measurement Guidelines note that there are potential issues with, “shared locations such as corporate offices, dorms etc., that have a large number of people sharing the external IP Address.” With IPv6, every device can have its own, unique IP address. That means IPv6 can improve analytics accuracy by reducing or eliminating the use of shared IP addresses, as well as improve performance and security for listeners.

In IPv6, addresses could be static since there are many more IPv6 addresses than connecting devices globally. This would mean an IPv6 address is basically a UUID. Due to the security and privacy concerns this raises, IP addresses rotate on a regular basis on most residential and cellular networks. This in turn creates a challenge for accurate analytics. In many settings, session cookies or similar tracking technologies can be used to overcome some of these challenges.

Podcasting has largely been focused on developed markets like the US and EU, where IPv6 support now lags behind much of the developing world. Without business incentive, there is no compelling case for podcast hosts to offer IPv6 support. Worse, since IPv6-only users fail to establish a connection to an IPv4-only host, podcast hosts and podcasters have no visibility into how many listeners they may be missing out on.

Some podcasters may not see this as a problem, but OEMs and telecom companies like Reliance Jio often make IPv6-only the default configuration for devices like the JioPhone. In a 2019 presentation, Leveraging IPv6 for Explosive Growth, Ramesh Chandra of Reliance Jio writes:

• 93.7% requests served are on IPv6
• Jio’s exponential growth enabler is IPv6 (soon to be IPv6-only)
• Jio no longer encouraging IPv4 addresses

By not supporting IPv6, you are unnecessarily disregarding access to some of the largest and fastest-growing Anglophone markets like India.

Final thoughts

As the developer of PodLP, listeners from around the world have reported issues accessing their favorite content on KaiOS mobile devices due to network connectivity issues related to a lack of IPv6 support. For podcasters and podcast hosts, the bottom line is this: if you do not support IPv6, you are losing listeners, especially in emerging markets.

IPv6 deployment is inevitable, and with it come gains in efficiency and security beyond the scope of in this article. There are entire podcasts like IPv6 Buzz dedicated to IPv6 networking concerns. Given it a listen, and consider how IPv6 could benefit you and your listeners.

More than 3.1 billion people (~40%) are not connected to the internet, the majority of whom live in Africa and Asia. For companies, this presents an opportunity to build hardware and software that connects the “next billion,” particularly in the mobile space with platforms like Android Go and KaiOS. The JioPhone is one such success story, selling more than 70 million phones in India at the equivalent of $10 USD (or less!).

With around 500 apps, the KaiStore is a platform where app developers can publish apps available to millions without being crowded out. However, this requires software that is designed, developed, and optimized to run on low-end hardware. That often means simplified interfaces designed for d-pad navigation, low-bandwidth networks, and really small screens¹.

¹ 1st-Gen iPhone: 320x480, KaiOS: 240x320

When I looked through the app store on my Alcatel GoFlip 3 running KaiOS, one noticeable omission for me was the lack of a podcast app. Android and iOS each have dozens of podcast apps, including default clients provided by Google and Apple. For my new phone, with the navigation and screen size constraints, web-based podcast apps accessed through the browser had a poor user experience and no offline capabilities. Additionally, if I were going to make the switch from smartphone to feature phone, I wanted a podcast app to listen to all my own favorite podcasts. So, I decided to build PodLP, a podcast app designed for KaiOS.

Design

I started the process by looking through the podcast apps I currently use to understand which features would be the most important to include, and then started to think about how I could minimize and fit those features into both a smaller screen space and non-touch screen interaction.

Visual design began with the UI Kit for Sketch.app that KaiOS provides to create standard user experiences that leverage familiar UI elements like software keys and options menus. For podcast clients, nearly all apps offer some combination of popular podcasts, search, subscribe, and download. These formed the set of minimum feature requirements that would enable users to discover, listen to, and follow podcasts from around the world.

After walking through the simplest version based on existing KaiOS components, subtle design improvements like including the number of items in a list were added. This affords users a sense of scale on a device where only 3–4 items can fit on screen at once. Visual progress indicators were also added both on the play and episode screens for users who might start and then come back to an episode. “Invisible” UX additions included automatically deleting episodes after listening to free up space on devices with minimal storage.

Finally, I began prototyping the app on my Nokia 8110 (aka the “banana phone”). Especially when dealing with T9 typing and d-pad navigation, there is no substitute for a real device when validating designs — and having others trying to use and navigate through the app revealed information hierarchy and findability issues as well as where error/success states needed to be designed. It also helps proof details like color, sizing, and spacing while also considering the “feel” of interactive components (modal, dialog, scrubber).

Development

Although there are very few apps on the KaiStore compared to the Apple App Store or Google Play, there are a handful of resources of software developers including the KaiOS Developer Portal, KaiUI library, and r/KaiOS Discord server. The experience can be frustrating, especially when navigating undocumented or under-documented APIs. Again, this is where a real device is helpful in feature discovery and validation. Even still, you may find yourself misguided by the existing, limited documentation.

For example, don’t be surprised if you notice inconsistencies in functionality with on-device testing. Originally PodLP was built to use the DeviceStorage API to save podcasts to files locally. However, I soon discovered certain parts of the API had never actually been implemented. This creates for a confusing (and frustrating) development experience. If you intend to rely on any KaiOS/ Firefox OS-specific APIs, smoke test them in the developer console first!

It’s also worth mentioning that most KaiOS devices have either 256MB or 512MB of RAM, which is often the most limiting resource. This means using techniques like virtualization to limit the number of elements actually rendered on screen, as well as limiting file size when downloading remote assets using XHR or Fetch. For PodLP, this was especially challenging since 1hr+ shows can easily reach 50–100MB in size on a device that doesn’t have close to that much memory freely available.

Publishing

The last step in developing PodLP was KaiAds integration, both as a monetization strategy and because it’s required for distribution on the KaiStore. Based on what I’ve observed in the KaiAds console, I expect very low effective cost per mile (eCPM) around $0.01 — $0.10 USD, which means it will be difficult to hit the $500 distribution threshold.

However, it is still the early days for KaiOS and for me, the main motivation is the prospect of becoming the de facto podcast app for millions, many of whom may be coming online and listening to podcasts for the first time! I also hope to build a useful service for many like myself interested in a “digital detox,” but who do not want to give up podcasts.

Finally, after months of designing, developing, testing, debugging, and localizing, I published PodLP on the KaiStore. The app is freely available in English, Spanish, and French (more languages coming soon) on all devices running KaiOS 2.0+.

Note: you can find up-to-date developer guides on KaiOS.dev and showcase your app with landing pages on KaiOS.app.

A Tale of Two Zeroes

Since the early versions of JavaScript there have always been two ways of performing equality comparison:

• Abstract Equality Comparison using == aka “double equals”
• Strict Equality Comparison using === aka “triple equals”

ES6 delivered a third option in the form of the Object.is method. It has some slight differences that hint at problems you did not even know you had. For instance, how would you tell JavaScript’s two zeroes apart?

Perhaps a better question is, JavaScript has two zeroes? Why does JavaScript have two zeroes? Why would any language have two zeroes?

The name for this is Signed Zero. It turns out the reason that JavaScript has -0 is because it is a part of the IEEE 754 floating-point specification. Two zeroes are also present in other languages like Ruby as well.

The two zeroes in question are:

• Positive zero +0
• Negative zero -0

As one might expect, they are treated as “the same” by both equality comparisons methods above. After all, zero is zero.

-0 == +0   // true
-0 === +0  // true

That is where Object.is comes in. It treats the two zeroes as unequal.

Object.is(+0, -0)  // false

Pre-ES6 it is still possible to tell the two zeroes apart. Looking at the results from other operators offers a hint.

-0 > +0  // false
+0 > -0  // false
-0 + -0  // -0
-0 + +0  // +0
+0 + +0  // +0
+1 / -0  // -Infinity
+1 / +0  // +Infinity

The last two operations in particular are useful. Unlike say Ruby or Python where division by zero yields a ZeroDivisionError, JavaScript returns Infinity. Now take a look at the polyfill for Object.is offered on MDN.

Since division by signed zero in JavaScript returns signed infinity, those results can be compared to tell the two zeroes apart.

Resources

• Negative Zero
• What Every Computer Scientist Should Know About Floating-Point Arithmetic
• Much Ado About Nothing’s Sign Bit

TLDR

• JavaScript (and many languages) have two, signed zeroes
• They can be distinguished pre-ES6 and using Object.is
• The difference is only evident when doing signed arithmetic

Why does JavaScript have -0? was originally published in JavaScript in Plain English on Medium, where people are continuing the conversation by highlighting and responding to this story.

Supporting Scale

As software company offerings expand, so do their teams. When startups get acquired or go public, the engineering team grows into a full engineering organization. That engineering organization then needs to scale to support more customers and in the process, monolithic architectures make way to service-oriented designs.

Like the code they write, engineering organizations migrate from a single entity into a distributed collection of two-pizza teams. As with their micro-services, these teams need to talk to one another.

Engineers have lots of ways to communicate: email, Slack, JIRA, and passive-aggressive post-it notes. So, there are many options for connectivity in AWS including Internet Gateways, NAT Gateways, Direct Connect Gateways, Egress-Only Internet Gateways, VPC Peering, and PrivateLink.

Connectivity Options

Each option has a set of trade-offs associated with it, but the main ones used for service-to-service communication are Internet Gateways, VPC Peering, PrivateLink, and Transit Gateway. Here is a vastly oversimplified use case for each of these resources:

• Internet Gateways: use only for public services. Traffic is routed over the internet and DNS entries are configured in a public Route53 Hosted Zone, making your service discoverable and accessible to the world.
• PrivateLink: use for secure, single service-to-service integration with one of the supported services. Not ideal for shared services as each consumer network requires a VPC Endpoint.
• VPC Peering: use for single, VPC-to-VPC service sharing. Not ideal for mesh networks because peering connections are not transitive and Edge-to-Edge Gateway/ VPN Connections cannot be shared.
• Transit Gateway: use for general purpose, secure, centrally-managed shared services and internal, on-premise applications. Combine with private Hosted Zones and a Route53 Resolver to support private DNS.

Transit Gateway vs. VPC Peering — Limitations

As a network-level utility that enables the sharing of multiple resources, VPC Peering is the closest analog to Transit Gateway. As such, the two also share similar limitations including the inability to have overlapping IP address ranges (CIDR blocks).

However, there are several significant unsupported VPC Peering configurations that are fully supported by Transit Gateway.

VPC Peering does not support transitive peering

This restriction is so important, it is actually the reason behind the name Transit Gateway. Transitive networks greatly simplify full, multi-VPC mesh networks where every node is connected to every other node in the network.

If you have a VPC Peering connection between VPC A and VPC B, and one between VPC A and VPC C, there is no VPC Peering connection between VPC B and VPC C. You cannot route packets directly from VPC B to VPC C through VPC A.

While it is possible to connect up to 125 VPCs through peering connections, the issue is one of scale. For a full mesh network, there need to be n(n-1)/2 total connections. For 125 VPCs, that is more than 7,000 total peering connections! On the other hand, Transit Gateway supports up to 5,000 VPCs and only requires one Attachment per VPC.

VPC Peering does not allow sharing of Edge-to-Edge Gateways

Additionally, VPC Peering does not support the sharing of Edge-to-Edge Gateways or Private Connections including Internet Gateways, Direct Connect Gateways, or VPC Endpoints. That means each of these resources needs to be recreated across every network that needs access.

With Transit Gateway, only a single Site-to-Site VPN or Direct Connection is needed to host internal services that span multiple accounts. This configuration is much easier to monitor and scale.

Edge to edge routing is not supported; you cannot use VPC A to extend the peering relationship to exist between VPC B and the corporate network.

Internal & Shared Services

Although Transit Gateway enables many advanced network topologies, at a high-level, it is especially useful in provisioning internal & shared services.

Internal Services

Cloud-hosted internal services are valuable because they combine the scalability and cost-effectiveness of cloud infrastructure with the privacy, security, and control of on-premise systems.

With the proliferation of Software as a Service (SaaS), we have become accustomed with the notion of distributing information to third-party providers and creating services that are connected to the public internet. However, restricting access to such services at the network-level improves security by reducing the surface area for a potential attack and eliminates unnecessary traffic from bots.

Before Transit Gateway, in order to launch internal, cloud-hosted services each isolated VPC needed its own connection to a corporate network. This is difficult to monitor and maintain and is not cost-effective since many of these resources are priced by the hour. With Transit Gateway, it is much easier to monitor and restrict traffic across just a single connection.

Shared Services

Monolithic architecture is by its nature centralized. This means that communication between components and sharing dependencies is trivial. Yet, monoliths are an example of tightly-coupled software that has well-known issues related to scalability, security, and stability.

In direct contrast are distributed systems and service-oriented architectures (SAO). These inherently decoupled applications now come with new issues, like the ability to share cross-cutting concerns like logging, alerting, and analytics. That is where Transit Gateway can help.

Rather than share a Site-to-Site VPN that connects to a corporate network, it is instead possible to share routes to a set of VPCs that together make up a suite of shared services. This ensures traffic to these services is not routed over the internet and both ingress and egress can be monitored in a single location. It also means that as new services get created, they can become instantly and automatically available to other applications without changes in configuration or the need to create a new pairing connection.

Complexity in the cloud only grows and that is especially true of cloud networking. As recently as 2006, Amazon Web Services (AWS) had only three service offerings: S3, SQS, and EC2. Today there are hundreds of services. Some like Transit Gateway comes with an initial investment in time, but once set up, they greatly reduce the cost and difficulty of connectivity moving forward.

Scale Securely with Transit Gateway was originally published in The Startup on Medium, where people are continuing the conversation by highlighting and responding to this story.

There are a lot of ideas about software and how to drive its development. Some approaches are promising, while others can be pretty problematic. Here are just a few strategies you might have seen used by other engineers for crafting digital products.

Beta Driven Development 🐤

MVP is Life. Beta Driven Development (AKA the other BDD) exists because the client initially requested a Proof of Concept (POC), but now they want to take that into production. The supporting code is not modular, does not adhere to SOLID principles, and does not have a single unit or integration test written. In fact, there is no need for integration tests because all of the “integrations” are actually just mocks anyway. If you are lucky, you might find a single comment.

Profit Driven Development 💵

Capital optimization drives the economy, and, this process. All infrastructure decisions, no matter how small, must be run by management and finance to ensure minimal impact to margins. Separation of concerns is not important here. Instead, developers are forced to find clever ways to squeeze near-complete utilization out of every single EC2 instance. That might include running scheduled jobs on another client’s idling machine so your Account Manager can sleep soundly at night.

Note: this approach is not to be confused with Prophet Driven Development, where architects are handed diagrams and code in the form of stone tablets from the divine (aka the design team). Also known as God Given Development, this practice makes heavy use of magic characters, hand-wavy interactions, and tends to focus more on commandments than unit tests. Pre-push hooks are the medium with which the heavenly enforces code cleanliness and purity.

Defense Driven Development 🛡️

Don’t Trust Anyone is the mantra at DDD shops. Even in dynamically-typed, interpreted languages like Javascript, every function starts with a dozen types and validity assertions. No system is a safe system, and every function is an interface to the outside world. InfoSec does not grant approval for third-party dependencies, so these codebases are often large and littered with copious exception handling logic. No left-pad imports here.

Rest Driven Development 💤

Many followers began their journey experimenting with the Pomodoro Technique, pausing every 25 minutes for a quick break. Developers know that working long hours is not conducive to productivity or quality. That is why RDD shuns the eight-hour workday in favor of rest, with its memory formation and consolidation benefits. As such, practitioners sleep 17–21 hours per day, waking only at the end of a REM cycle to write just a few lines of the absolute most readable, efficient, and concise code.

Instagram Driven Development 📷

Do it for the gram. Just find a heavily-indented file with colorful syntax highlighting, apply a tilt shift filter so you can’t quite make out the details, place a cup of trendy coffee or a cute animal next to your workstation, and snap a pic! Competition is steep, especially when you are up against Digital Nomads traveling the globe and writing code when inspiration strikes. However, if you stick to IGDD you can trade-in your day job as a #webdev for one as a #influencer. Just imagine all income from JetBrains sponsorships, Google endorsements, and Apple product placements.

#developer #programming #code #vscode #codeislife

Software is a great combination between artistry and engineering.

– Bill Gates

Perhaps some (or all) of these strategies have resonated at a point in time. Whatever methodology you subscribe to, be sure to write great software.

Totally legit software methodologies was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.

NASA’s earliest attempts at launching rockets were part of the Bumper program, comprised of unmanned rockets in the late ‘40s and early ‘50s. The next decade saw many rocket launches — some successful, while others burst into flames.

Then, in 1961, the Soviet Union successfully sent cosmonaut Yuri Gagari on the first manned rocket, making him the first human in history to journey to outer space.

Contrast that with Wan Hu, a legendary Chinese figure described as the “first astronaut.” For his ascent, he was seated in a chair attached to several dozen rockets. Most versions of this story are apocryphal, with Hu not surviving to make another attempt at spaceflight.

Wan Hu’s tale is a cautionary one: we should first test our ideas in a constrained environment before risking important resources (like our lives).

It is now common to see software in aviation, automotive, pharmacy, banking, and education. This makes software quality more important than ever, with software testing a vital part of improving quality.

Why Test Software

Unit tests are the front-line defense against bugs and errors. Without automated tests, production deployments are more like crossing your fingers and hoping for the best. Ultimately, the end-users become your test subjects.

Automated testing catches bugs early in the software development life cycle (SDLC), leading to measurable cost savings and overall improvements in software quality.

Yet, even when requirements are crystal clear and stakeholders are in alignment, engineers make mistakes translating their understanding of the requirements into executable code.

Bugs can take many forms, including errors in logic, arithmetic, and connectivity, as well as the result of obsolete dependencies, assumptions about data, poor documentation, or lack of proper version control. Errors are also commonly the result of unforeseen side effects.

Unit tests force us to think critically and verify that our code exhibits certain behaviors. In this way, tests serve the same purpose as an editor and can guard against subtle typos that might result in syntax or type errors.

For interpreted languages like Ruby, this is especially helpful because many of these exceptions are run-time errors rather than compile-time errors.

When unit tests are combined and run against real data between multiple, production-like systems, they become integration tests. This is where Behavior Driven Development (BDD) frameworks, like RSpec, can be both exceptionally powerful and frustrating.

Much like Rails, RSpec, in combination with libraries like factory_bot, DatabaseCleaner, Capybara, Timecop, and VCR, provides functionality that can easily seem like Ruby Magic™. But once you are familiar with these tools, it becomes possible to test each system in concert and in isolation.

After spending time writing (and rewriting) many RSpec tests, I want to point out the subtle and not-so-subtle issues I have writing and debugging tests.

How RSpec Tests Fail

Tests fail consistently or intermittently.

Reproducibility in tests is important because without it, developers question the value of the tests and eventually testing itself (think, Boy Who Cried Wolf Syndrome).

Consistently failing tests are easier to catch and commonly the result of mental fatigue.

Tests that fail intermittently are called flaky tests and are more difficult to analyze. They are usually caused by non-deterministic behavior, unavailable external services, non-transactional changes, assumptions about data, side-effects, and edge cases related to dates and time.

Here’s why RSpec tests fail and how to fix them.

Forgetfulness

Stubbing too much or unintentionally stubbing.

Stubs provide canned answers to calls made during the test, usually not responding at all to anything outside what’s programmed in for the test. — Martin Fowler, Mocks Aren’t Stubs

RSpec does not explicitly require the use of test spies to make message expectations. Instead, it offers the receive matcher, which can be used on any object to make expectations about branching behavior.

While this approach offers more concise and convenient syntax, it can also be easy to forget that the receive matcher stubs the target method by default.

Real estate 101 tells us that home prices only ever go up.

Now, we might want to test that assumption in RSpec:

When we run this test, to our surprise, it fails with the following error:

Failure/Error: expect(house.price).to be > 649000
       expected: > 649000
       got:        649000

As it turns out, the receive matcher stubs the target method unless we use and_call_original to un-stub. This no-oped appreciate_a_lot, and the result is that the house price remains unchanged.

One strategy is to separate branch testing from results tests. This keeps test cases concise and avoids issues with inadvertently stubbing methods.

A more general approach is to stub as little as necessary to avoid discrepancies between test and production code paths. Most importantly, unlike the example above, don’t stub the system under test.

TL/DR: Remember, stubbing is the default behavior of the receive matcher; use and_call_original to un-stub but retain message expectations.

In-Memory and In-Database Discrepancies

There are many ways to find and update active record models, but by default, active record will use an integer column named id as the table’s primary key.

Eventually, with Rails 4.2, came GlobalID, an app-wide uniform resource identifier (URI). As a result, Rails features include active job support, passing models directly.

This made it easier to pass models as arguments to jobs. However, it also masks what is happening behind the scenes which can cause confusion during testing.

Active job uses GlobalID::Locator.locate internally to deserialize the model GlobalID. It then performs a database query using the primary key.

Now, it is tempting to write a test like this one, as on the surface it looks like we are passing the user object directly.

The first test fails with an error like this one:

Failure/Error: expect(user).to receive(:perform_send_welcome_email!)
     
   (#<User:#..>).perform_send_welcome_email!(*(any args))
           expected: 1 time with any arguments
           received: 0 times with any arguments

The second test also fails, but with a different error:

Failure/Error: expect(user.welcome_email_sent?).to eq(true)
     
       expected: true
            got: false
     
       (compared using ==)

Both of these tests fail because the subject under test, a specific User, is the same in-database but not in-memory. Database changes are not automatically propagated to models in memory.

The first test could instead make use of a matcher like expect_any_instance_of(User) while the second test can be done easily with #reload, which re-syncs active record with the database.

TL/DR: Be cognizant of in-memory and in-database discrepancies; use #reload as necessary to re-sync a record with the database.

Non-Determinism

Sometimes it passes, sometimes it fails.

Randomness

Can a computer generate a truly random number? Not really, at least not without special hardware. Instead, computers simulate randomness by drawing from a pool of entropy like cursor movements, keystrokes, hard-disk access, etc.

These “random” sources are then combined in a single location (like /dev/random) for applications that require pseudo-random behavior.

Regardless of whether the behavior is truly-random or pseudo-random, testing randomness can be quite difficult. Unsurprisingly, it is also an obvious source of non-deterministic behavior. Consider a Dice class:

Calling roll should return a random number between one and six, just like a real dice.

Next, we write tests to verify this behavior. Perhaps we also include a test that verifies that the dice is not somehow weighted (or maybe this code is for a gambling service and we want skewed results).

We run our tests locally, and they pass. They pass again in CI. A few days later, we notice an intermittent failure on the second test. The easiest solution would be to avoid randomness altogether. When that is not possible, we can instead provide a “seed” to the Random class.

Note: This is the same process we use when debugging randomly-ordered RSpec tests by manually setting the --seed option.

Pseudo-random number generators (PRNGs) are just functions that map one input to a seemingly-random, but deterministic sequence. Using a fixed seed provides us with a fixed sequence, and thus deterministic results.

Now, total will always yield 191 and the test will pass consistently! Although we have also introduced an inconsistency between test and production conditions. Fortunately, this scenario is not that common.

Unique constraints

Randomness will more likely show up as collisions between randomly generated data in a model factory with unique constraints.

The odds of collision with the ObjectId or similar UUIDs are exceptionally low, but the probability of duplication with the description are 1:1000. When randomly generated properties are necessary, the easiest solution is to pick from a larger pool of random.

Note: Faker is another great tool that can be used in test factories to create unique, structured data without the risk of collisions.

TL/DR: Avoid testing randomness. If you can’t, specify a seed and/or use more random random like SecureRandom.

State Preservation Across Tests

Arne Hartherz’s article summarizes this well: Using before(:all) in RSpec will cause you lots of trouble unless you know what you are doing.

before(:all) creates data outside of transaction blocks, persisting changes across tests.

Best practice (which Rspec supports using the --order rand option) is to run tests in random order to reveal implicit dependencies between tests. Data that persists between randomly ordered tests naturally results in inconsistencies.

Here is one example of how not to use before(:all):

If the tests are run in the order they are defined, the second test will fail:

Failure/Error: raise AlreadyVotedError

# ./voter.rb:26:in `vote'

This is why RSpec offers lazy-evaluated let, and eager-evaluated let! helper methods. When to use RSpec let offers more details, but in short:

Always prefer let to an instance variable.

TL/DR: Always use let, do not use before(:all).

Network Calls

Most applications rely on third-party services like Stripe or Square for payment processing, Splunk for logging, or New Relic for monitoring.

That means that applications generate lots of network requests. While we can trust that the engineers for these service providers have thoroughly tested their client libraries, it is important to test the use of these libraries within our applications as well.

Unfortunately, not all client libraries will support an environment for integration testing. Even if they did, networking issues could lead to transient failures in tests like this one. It is best not to test beyond the remote boundary when writing contract tests.

# --- Caused by: ---
# Net::HTTPServerException:
#   503 "Service Unavailable"

That is where tools like VCR are useful!

VCR lets developers “record and replay” HTTP interactions, including those generated by third-party tools. This makes tests fast, deterministic, accurate, and not reliant on networking availability or external services.

Tests are wrapped in a use_cassette block so that on their first execution, a YAML file gets created to save the response.

For more in-depth information, RubyGuides has a good tutorial on using and configuring VCR.

TL/DR: Use pre-recorded HTTP responses in tests to improve overall speed, reliability, and accuracy.

Time

Not freezing time.

There are 28–31 days in a month, 365 days in a year, and 2,027 days in which Nixon was president. Actually, there are about 365.25 days in a Julian year (365.2422 to be more precise).

Alternatively, every fourth year can be called a leap year where we just tack on an extra day. February always gets that day, because it is the shortest month.

Some countries like the USA (and territories) span 11 timezones, while others like China observe just a single timezone (UTC+8). Then, for some crazy reason, twice a year a handful of countries increment or decrement their clocks in an attempt to coordinate working hours with sunlight.

The tests work reliably, between 01:00–23:00.

Needless to say, time is very complicated and testing time at or across boundaries is incredibly difficult. I have heard these kinds of tests referred to as Cinderella Tests because they turn into pumpkins at midnight.

To make these tests more reliable, it is best to become a Time Lord… or at least learn how to freeze and travel time in RSpec.

Here is an example with a Venue model that has many Events. If we want a list of upcoming concerts, we can compare start times to the current day.

Then, during testing, we want to make sure this scope returns concerts that happened today or will happen in the future.

This test looks fine. It even derives start times from a single source to guarantee consistent deltas. A problem arises if this test runs at a day boundary, for example, just before midnight (11:59:59pm).

If enough time elapses during concert creation, upcoming_concerts might get called the next day, causing the first expectation to fail.

The easiest way to resolve this with Timecop is to freeze time before :each and return after :each. This way, time elapses between but not during tests.

TL/DR: Use Timecop to freeze and travel time predictably in RSpec; test time boundaries and edge cases defensively.

Side Effects

Active record callbacks

Active record offers convenient life cycle callbacks before and after state alteration that are easy to “set and forget”. Thorough testing should include these callbacks, but sometimes it is necessary to sidestep them.

That is where skip_callback comes in. It can be used both in a spec factory as well as individual tests. Consider a User model:

While it is possible to stub send_welcome_email, in a real-world application this method might be buried deep in a stack trace and turn out to be only one of many callbacks.

Or, perhaps the callbacks were meant to run before a job asynchronously as above, but all jobs are being run inline for a specific test. Whatever the case, it is possible to instruct active record to skip callbacks for a specific model.

It may be that a User model is required for an Address to be considered residential.

However, creating and associating a User with the Address inside of a perform_enqueued_jobs block triggers an after-create callback, and ultimately the SendWelcomeEmail job.

That said, skipping callbacks should be avoided in favor of more explicit factories, recorded network calls, and environment-specific configurations.

For example, in the “test” environment, sending an email could be handled globally within ActionMailer. If you do choose to skip_callback, remember to call set_callback to restore callbacks for other tests.

TL/DR: skip_callback can avoid active record callbacks, but use cautiously.

Active Job QueueAdapters

A common use case for active record callbacks is to enqueue an active job. Under the hood, active job is configured to use a specific QueueAdapter. This adapter determines the queue order (like FIFO, LIFO, etc).

A common adapter for RSpec is the TestAdapter, which can be used to verifying that a specific job was enqueued successfully.

However, TestAdapter does not actually perform the job by default!

Depending on what you are testing, there are other adapters like the InlineAdapter that execute jobs immediately by treating perform_later calls like perform_now.

Alternatively, TestAdapter has a method, perform_enqueued_jobs, that, as its name suggests, actually performs the enqueued jobs synchronously.

As with callbacks, there is value to testing both with and without actually performing the job. RSpec provides helpful active job matchers like the have_enqueued_job matcher.

These helper methods allow for separation of concerns, making it possible to test the logic of a job in one spec, and the logic that triggers the job in another.

TL/DR: Use TestAdapter to track and perform enqueued active jobs.

Too Specific

Inconsistently ordered data.

Collections like Hashes and Arrays are used to store related data and are both enumerate in order of insertion.

During testing, this leads to comparisons that are implicitly order specific, often when creating and comparing collections of active record models. For instance, a Cat can have many toys.

A simple test is then written to confirm that a Cat can in fact have many toys.

factory_bot’s create_list helper creates three toys associated with our cat, Lovie.

The issue here is that while create_list will return three unique toys ordered consistently by creation, the association will return based on the scope ordering. If no order is provided, it defaults to order by ID.

For sequential IDs, this should not pose an issue because ordering by creation or sequence should be identical. However, the first four digits of an ObjectID represents seconds since the Unix epoch.

Instead of using eq to compare two arrays, we can use the redundantly-named match_array matcher, which is independent of order.

TL/DR: Use hash_including, include, and match_array when comparing collections independent of ordering.

Negative Test Expectations

Negative tests are a special case. Unlike positive tests, it is very easy to write overly-specific tests that ultimately test nothing. Consider a code path within a method that is not expected to raise a custom error.

A test for price_per_sq_ft might look something like this:

The first test expects an error to be raised, while the second test expects an error not to be raised. The issue with the second test is subtle, but thankfully is such a common occurrence that RSpec actually warns developers by default.

Warning:

Using expect { }.not_to raise_error(SpecificErrorClass) risks false positives, as literally any other error would cause the expectation to pass, including those raised by Ruby (e.g. NoMethodError, NameError, and ArgumentError), meaning the code you are intending to test may not even get reached.

Instead, consider using expect { }.not_to raise_error` or `expect { }.to raise_error(DifferentSpecificErrorClass).

The RSpec warning clearly explains the issue with this test.

In the example above, the second test actually raises TypeError: nil can’t be coerced into Fixnum as nowhere have we defined @sq_ft! That is the problem with overly-specific negative tests, they can miss real problems like this one.

TL/DR: Favor positive tests over negative tests; write negative tests more broadly, especially when it comes to error handling.

Resources

There are lots of great articles on testing with Ruby, Rails, active record, RSpec, factory_bot, and automated testing in general.

Here are just a few I have found especially informative.

• Better Specs.
• RSpec Style Guide.
• Tests that sometimes fail.
• It’s about time (zones).
• How should I test randomness?
• Flakey tests — The war that never ends.
• RSpec: Thank you for running my tests in random order
• How to set up Rails with RSpec, Capybara, and Database_cleaner.
• Testing is good. Pyramids are bad. Ice cream cones are the worst.

Final Thoughts

The bitterness of poor quality remains long after the sweetness of meeting the schedule has been forgotten.

Test failures can be frustrating, especially when it is not clear why they are failing. However, try not to lose sight of the overall goal behind testing: to validate behavior and alert engineers of the unintended consequence behind potential changes.

As frustrating as failing tests can be, having no tests leads to unpredictable and unreliable deployments. As software systems grow in size and complexity, so too do the risks of going straight into production. The alternative is to push code and just cross your fingers.

Of course, if after reviewing these examples your tests are still failing, there is always the possibility that it is because there’s a bug!

After all, that is what tests are designed to do: identify bugs earlier in the SDLC. It is better (and cheaper) to catch bugs during testing than in production.

Happy bug squashing!

Why RSpec Tests Fail (and How To Fix Them) was originally published in Better Programming on Medium, where people are continuing the conversation by highlighting and responding to this story.

With Proxy, you get a tiger object disguised as a cat object. Here are about half a dozen examples that I hope will convince you that Proxy provides powerful metaprogramming in Javascript.

Note: examples may be dated, and similar functionality can be achieved natively in newer versions of ECMAScript or syntactical supersets like TypeScript. This article shows what’s possible natively in ES2015 using Proxy.

Introduction to Proxy

With Proxy, you get a tiger object disguised as a cat object. Here are about half a dozen examples that I hope will convince you that Proxy provides powerful metaprogramming in Javascript.

Although it is not as well known as other ES2015 features, Proxy has many uses including operator overloading, object mocking, concise-yet-flexible API creation, Object on-change events, and even powers the internal reactivity system behind Vue.js 3.

The Proxy object is used to define custom behavior for fundamental operations (e.g. property lookup, assignment, enumeration, function invocation, etc).–MDN

A Proxy is a placeholder object that “traps” invocations and operations made to its target object which it can then passthrough, no-op, or handle more elegantly. It creates an undetectable barrier around the target object that redirects all operations to the handler object.

A Proxy is created using the new Proxy constructor which accepts two, required arguments: the target object and handler object.

Tip: Use Bit (open source) to seamlessly turn your JS code into reusable components you can easily share and manage across projects- to build faster.

Bit. Composable software.

The simplest example of a functioning Proxy is one with a single trap, in this case, a get trap that always returns “42”.

The result is an object that will return “42” for any property access operation. That means target.x, target['x'], Reflect.get(target, 'x'), etc.

However, Proxy traps are certainly not limited to get operations. It is only one of more than a dozen different traps:

• handler.get
• handler.set
• handler.has
• handler.apply
• handler.construct
• handler.ownKeys
• handler.deleteProperty
• handler.defineProperty
• handler.isExtensible
• handler.preventExtensions
• handler.getPrototypeOf
• handler.setPrototypeOf
• handler.getOwnPropertyDescriptor

Proxy Use Cases

It may not be immediately apparent how such a simple pattern could be so widely used, but hopefully, after a few examples, it will become more clear.

Default/ “Zero Values”

In GoLang, there is the concept of zero values which are type-specific, implicit default struct values. The idea is to provide type-safe default primitives values, or in Gopher-speak, “give your structs a useful zero value!”

Though different creation patterns enable similar functionality, Javascript had no way to wrap an object with implicit initial values. The default value for an unset property in Javascript is undefined. That is, until Proxy.

This three-line function wraps a target object. If the property is set, it returns the property value (pass-through). Otherwise, it returns a default “zero value.” Technically, this approach isn’t implicit either but it could be if we extended withZeroValue to support type-specific (rather than parameterized) zero values for Boolean (false), Number (0), String (“”), Object ({}), Array ([]), etc.

One place where this functionality might be useful is a coordinate system. Plotting libraries may automatically support 2D and 3D rendering based on the shape of the data. Rather than create two separate models, it might make sense to always include z defaulted to zero rather than undefined.

Negative Array Indices

Getting the last element in an Array in Javascript is verbose, repetitive, and prone to off-by-one errors. That’s why there is a TC39 proposal that defines a convenience property, Array.lastItem, to get and set the last element.

Other languages like Python and Ruby make access to terminal elements easier with negative array indices. For example, the last element can be accessed simply with arr[-1] instead of arr[arr.length-1].

With Proxy, negative indices can also be used in Javascript.

One important note is that traps including handler.get stringify all properties. For array access, we need to coerce property names into Numbers which can be done concisely with the unary plus operator.

Now [-1] accesses the last element, [-2] the second to last, and so on.

There is even an npm package, negative-array, that encapsulates this functionality more completely.

Hiding Properties

Javascript has notoriously lacked private properties. Symbol was originally introduced to enable private properties, but later watered down with reflective methods like Object.getOwnPropertySymbols that made them publicly discoverable.

The longstanding convention has been to name private properties with a leading underscore, effectively marking them “do not touch.” Proxy offers a slightly better approach to masking such properties.

The hide function wraps a target object and makes properties that are prefixed with an underscore inaccessible from the in operator and from methods like Object.getOwnPropertyNames.

A more complete implementation would also include traps like deleteProperty and defineProperty. Apart from closures, this is probably the approach that is closest to truly-private properties as they are inaccessible from enumeration, cloning, access, or modification.

They are, however, visible in the development console. Only closures are exempt from this fate.

Caching

There are two hard problems in computer science: cache invalidation, naming things, and off-by-one errors.

It is not uncommon to face difficulties synchronizing state between the client and server. Data can change over time, and it can be difficult to know exactly where to place the logic of when to re-sync.

Proxy enables a new approach: wrap objects to invalidate (and resync) properties as necessary. All attempts to access a property first check against a cacheing strategy that decides to returns what’s currently in memory, or to take some other action.

This function is oversimplified: it renders all properties on an object inaccessible after a certain period of time. However, it would not be difficult to extend this approach to set time-to-live (TTL) on a per-property basis and to update it after a a certain duration or number of accesses.

This example simply renders the bank account balance inaccessible after 10 second. For more in-depth, real-world use cases there are several articles on Caching & Logging and Client-Side Caching using Proxy and sessionStorage.

Enums & Read-Only Views

These examples come from Csaba Hellinger’s article on Proxy Use Cases and Mozilla Hacks. The approach is to wrap an object to prevent extension or modification. Although Object.freeze now provides functionality to render an object read-only, it’s possible to extend this approach for better enum objects that throw errors accessing non-existent properties.

Read-Only View

Enum View

Now we can create an Object that throws an exception if you try to access non-exist properties, rather than returning undefined. This makes it easier to catch and address issues early on.

Our enum example is also the first example of proxies on proxies, confirming that a proxy is a valid target object for another proxy. This facilitates code reuse through composition of Proxy functionality.

This approach can be further extended to include “simulated methods” like nameOf that return the property name given an enum value, mimicking the behavior in languages like Javascript.

While other frameworks and language supersets like TypeScript offer an enum type, this solution is unique in that it works with vanilla Javascript without special build tools or transpilers.

Operator Overload

Perhaps the most fascinating Proxy use case syntactically is the ability to overload operators, like the in operator using handler.has.

The in operator is designed to check if a “specified property is in the specified object or its prototype chain.” But it is also the most syntactically-elegant operator to overload. This example defines a continuous range function to compare Numbers against.

Unlike Python, which uses generators to compare against a finite sequence of whole numbers, this approach supports decimal comparison and could be extended to support other numeric ranges–inclusive, exclusive, natural, rational, imaginary, ad infinitum.

Even though this use case does not solve a complex problem, it does provide clean, readable, and reusable code. 🔥

In addition to the in operator, we can also overload delete and new.

Cookies Object

If you have ever had to interface with cookies in Javascript, you have had to deal with document.cookie. It is an unusual API in that the API is a String that reads out all cookies, semi-colon delimited but you use the assignment operator to initialize or overwrite a single cookie.

document.cookie is a String that looks like:

_octo=GH1.2.2591.47507; _ga=GA1.1.62208.4087; has_recent_activity=1

In short, dealing with document.cookie is frustrating and error prone. One approach is the simple cookie framework, which can be adapted to use Proxy.

This function returns an object that acts like any other key-value object, but proxies all changes to document.cookie for persistence.

In 11 lines, we have a better interface for modifying cookies, although additional features like string normalization would be necessary in a production environment.

Fine Print

The devil is in the detail, and Proxy is no exception.

Polyfill

At the time of writing (May 2019), there is no complete polyfill for Proxy. There is, however, a partial polyfill for Proxy written by Google that supports the get, set, apply, and construct traps and works for IE9+.

Is it a Proxy?

It is impossible to determine whether an object is a proxy or not.–2ality

According to the Javascript language specifications, there is no way to determine if an Object is a Proxy. However, on Node 10+ it is possible using the util.types.isProxy method.

What’s the target?

Given a Proxy object, it’s not possible to obtain or change the target object. It is also not possible to obtain or modify the handler object.

The closest approximation is in Ben Nadel’s article Using Proxy to Dynamically Change THIS Binding, which uses an empty object as the Proxy target and closures to cleverly reassign the object the Proxy’s actions are performed on.

Proxy Primitives

Unfortunately, one limitation of Proxy is that the target has to be an Object. That means we cannot use primitives like String directly. 😞

Performance

A major drawback of Proxy is performance. Impact will vary based on browser and use, but Proxy isn’t the best approach for performance-critical code. Of course you can always measure the impact and decide if the advantages of Proxy outweigh the affect on performance.

Why Proxy?

Proxy provides a virtualized interface to control the behavior of any target Object. In doing so, it strikes a balance between simplicity and utility without sacrificing compatibility. Any code that expects an Object can accept a Proxy.

Perhaps the most compelling reason to use Proxy is that many of the examples above are just a few lines long and can easily be composed to create complex functionality. For one final example, we can compose functions from several use cases to create a read-only cookie object that returns a default value for non-existent or “private”, hidden cookies.

I hope these examples have shown that Proxy is more than just an esoteric feature for niche metaprogramming in Javascript.

Resources

• 10 Use Cases for Proxy
• MDN–Proxy
• David Walsh–JavaScript Proxy
• Intro to Javascript Proxy Object
• 2ality–Metaprogramming with ES6 proxies

If you found this article helpful, please leave some claps 👏

Also, feel free to comment — I’d love to hear from you!

Learn More

• 11 Javascript Utility Libraries you Should Know in 2019
• 7 Tools for Building Web Components
• 11 JavaScript Animation Libraries For 2019

A practical guide to ES6 Proxy was originally published in Bits and Pieces on Medium, where people are continuing the conversation by highlighting and responding to this story.

A Symbol

Originally meant to introduce private properties to ES6, Symbols offer an approach to metaprogramming in Javascript that provides extension hooks into language operators and methods without risking user name collisions. A symbol is a primitive data type that is immutable and globally-unique.

There are three kinds of Symbols:

• User-defined symbols created with the Symbol function
• Globally-registered symbols created with the Symbol.for function
• Well-known symbols defined as static properties on the Symbol object

While Symbols provide hooks into language methods, this same feature can be exploited by the developers of libraries and frameworks as well.

Here is a quick summary of how Symbols are used in code.

A user-defined symbol is created with the global Symbol function

They are new to ES6, but not that new

No two symbols are alike

Except when they are equal (i.e. symbols in the global registry)

They really are their own type

They can be set as properties on objects

They can usually be overwritten, even in native object prototypes

They can keep data from prying eyes

Except they are still discoverable through reflection

These are user-defined symbols, originally meant to introduce near-private properties to Javascript. Then there are built-in symbols (aka well-known symbols) designed to provide hooks into the implementation of native functions without risking clashes with user-defined names.

Well-known Symbols

Some Symbols are more well-known than others. There are symbols like Symbol.iterator that allow developers control over how an object can be iterated and spread. Then there are symbols like Symbol.unscopables, which was born out of a need to maintain backwards compatibility with the seldom used with keyword.

As an example, consider how to write a Password class that:

• Accepts an unmasked password string
• Irreversibly masks the password
• Can match (true or false) against a test password

In writing this class, we will use built-in symbols.

To simplicity sake, we will implement the actually “hashing” using an insecure method based on Java’s Object.hashCode function. For secure hashing, consider using a dedicated cryptography library like CryptoJs.

hashCode irreversibly turns a String into a Number with some probability of collision (i.e. two Strings hash into the same Number).

With our hashCode function, the first task is to mask the real password.

As we’ve seen before, a Symbol can be used as a property on an Object and is generally more private than say a String property prefixed with an underscore. However, it is not completely private. We can either use a direct reference to PWD or use reflective methods like Object.getOwnPropertySymbols to obtain a reference to our unique symbol.

For more private properties, we can use IIFE or WeakMap and soon ES7 will support a language-level private field modifier #.

Symbol.match

The next task is to make Passwords comparable. Symbol.match provides a hook into String.match, which is one way for comparing strings. It is generally used to compare against a regular expression and the native implementation even implicitly coerces values into RegExp. However, our Password class will explicitly define its own behavior.

Now we can compare using the Password as we would a RegExp!

Symbol.toPrimitive

While the example above could be achieved without Symbols (either as its own method or by overwriting String.prototype), there are well-known Symbols that alter language and operator behavior like type conversion.

This is where Symbol.toPrimitive comes in. It is used to convert Objects to primitives and is called by operators like the unary plus operator. We will use this Symbol to convert our Password into a masked String.

Now when we coerce our Password into a String, we get asterisks.

The hint parameter hint can be “number”, “string”, or “default”. A more complete implementation could take these values into account and return NaN or other values, as applicable. For now, we will only support String.

When we try to turn our password back into a string we end up with a masked password with asterisks replacing each character.

It is worth noting that Symbol.toPrimitive is used for more than explicit type conversion. It is also used for implicit coercions done by the additional operator + as well as the equality operator ==.

Symbol.toStringTag

You might notice that if instead we call Object.toString on our password, we get the generic tag [object Object]. Symbols allow us to be more descriptive. Using the Symbol.toStringTag symbol, we can provide the Javascript runtime with a better tag for our object.

Interestingly, Symbol.toStringTag is “a string valued property,” not a method. That means we need to use the get keyword in our password class (or Object.defineProperty on an instance) to declare this property.

Now when we call toString on our password, we get [object Password]. Although this is not something many apps would need, it can be useful from a debugging and logging perspective.

Symbol.hasInstance

Another less common, but still powerful Symbol is Symbol.hasInstance. It controls the behavior of the instanceof operator.

Fortunately it is not possible to redefine native implementations of the Symbol.hasInstance function like Function.prototype[Symbol.hasInstance]. However, it is possible to do so on custom classes and objects like Password.

The main use for Symbol.hasInstance is in libraries that want more control over instanceof checks. For example, GraphQL JS uses Symbol.hasInstance to check for specific symbols that identify GraphQLTypes.

Symbol.iterator

Symbol.iterator is among the most well-known and useful symbols. It enables user-defined iterables that implement the iteration protocol.

Here is an example of Symbol.iterator's power. They can be used to extend or override native object’s prototype. That means with just a few lines, we can define iteration for Objects globally.

Whether you should is a different question.

With this code, Objects can be spread into Arrays of [key, value] pairs. Likewise, Objects can be iterated using for...of loops.

In the context of our Password class, we don’t even have to write our own iterator! Since a Password can be coerced into a String, we can use the native String iterator directly.

Doing so allows us to spread or iterate a Password as a sequence of asterisks just as we could with a String.

Putting it all together, our final Password class looks like:

And has some interesting properties:

Future Symbols

The result is a class that masks the original password, is comparable, iterable, and convertible as we might expect from a String. While I have introduced five well-known Symbols, at the time of writing (May 2019) there are actually thirteen! The full list includes:

1. Symbol​.async​Iterator
2. Symbol​.has​Instance
3. Symbol​.isConcat​Spreadable
4. Symbol.iterator
5. Symbol​.match
6. Symbol​.matchAll
7. Symbol​.replace
8. Symbol​.search
9. Symbol​.species
10. Symbol​.split
11. Symbol​.toPrimitive
12. Symbol​.toStringTag
13. Symbol​.unscopables

Be on the lookout for more soon! There are several suggestions and proposals for Private Symbols, Symbol.thenable, Symbol.isAbstractEquals, Symbol.equals, Symbol.inspect, Symbol.inObject, and more.

Symbols are a unique, immutable data type unlike their counterparts in other languages. As the Javascript language continues to evolve, I expect Symbols to become a mainstay in Javascript metaprogramming.

If you found this helpful, please give some claps 👏 👏 👏

You can find me on LinkedIn · GitHub · Medium

A practical guide to ES6 Symbol was originally published in codeburst on Medium, where people are continuing the conversation by highlighting and responding to this story.

Thank u, [Symbol.iterator].next
Thank u, [Symbol.iterator].next
I’m so f****in’ grateful for this spec

Symbol.iterator

Symbol.iterator is the protocol that makes native objects like Array, Set, and Map iterable by providing a hook into language features like for…of loops and the spread operator.

The most obvious use case is in creating new, iterable data structures that are not provided by the language, like a Linked List. However, Symbol.iterator also lets developers redefine navigation patterns without cloning the object or modifying its order in-place.

These patterns can then be written once in a function, class, or module and used anywhere. That means you can loop through an Array forwards, backwards, randomly, or infinitely in a modular fashion.

The actual Symbol.iterator protocol is very simple. It includes:

• Iterable: an object with a function whose key is Symbol.iterator
• Iterator: the above function, used to obtain the values to be iterated

In terms of code:

This tells the Javascript runtime that an Object is iterable. It provides a way to get the next value and check if iteration is done.

If you want more details, there is a great guide on Demystifying ES6 Iterables & Iterators. Otherwise, let’s look at how to use this in practice.

Going Backwards

The two most common ways to iterate an array in reverse are using indices (from array.length — 1 to 0), or with Array.reverse:

But both approaches raise concerns. With indices, it can be easy to forget the -1, leading to the infamous off-by-one error. Using Array.reverse modifies the array in place, which may not always be the intention.

There are entire articles dedicated to this topic, but since we are talking about Symbol.iterator, let me suggest one more way:

Rather than iterating backwards on a given array, we have defined the “going backwards” iteration behavior for any array, all without modifying the order or creating a clone of the original array.

Here is how reverse would work in practice:

Another important aspect of Symbol.iterator is that we do not need an Array-like object to iterate. We can write a function like Python’s range and use it to iterate elegantly over a sequence of numbers.

Note: this implementation of range is very simple. It has not been tested and has clear issues with infinite ranges and negative steps.

We can use range the same way we did reverse:

We can even copy Scala’s Int#to syntax, though you should be cautious when extending a native object’s prototype.

Note: I wrapped the number literal in parentheses to avoid a SyntaxError, but there are other options.

As we will see, with Symbol.iterator the possibilities are literally endless!

Thank U, Next

A music player is a great example of iteration behavior. Most of us do not listen to songs in order. Instead we control playback with shuffle and repeat.

As a programmer, this means building a way to play songs infinitely (repeat), randomly (shuffle), or both. Here is an example that does just that:

The logic underlying these playback options is implemented using Symbol.iterator. Let me quickly break down how each class works.

Repeat

Playing songs on repeat is the simplest task. With InfiniteArray, when we reach the end of the array we just go back to the first element, ad infinitum.

Caution: InfiniteArray might be the simplest, but it is also the most dangerous! You can easily freeze a tab by calling [...infiniteArray].

Shuffle

Popular music services like Spotify used to shuffle songs with the Fisher-Yates algorithm, “but users complained that the playlist wasn’t genuinely random.”

Thankfully this is just an example, and Fisher-Yates is by far the simplest shuffling algorithm so we will go with that.

While RandomArray does not modify the Array in place, we can use it to create a shuffled (randomly-sorted) copy of the original Array.

Note: The actual order will vary each time the spread operator is called.

Repeat & Shuffle

Some users will enable both shuffle and repeat simultaneously. RandomInfiniteArray does just that, iterating both randomly and infinitely.

Caution: same warning as InfiniteArray, be careful looping infinitely!

Each of these classes can be used anywhere an Array is expected. In this example, updateIterator creates a shallow copy of the original album list with a type that depends on which playback options are selected.

In a larger application, Symbol.iterator can cleanly encapsulate iteration behavior in classes or functions, decreasing code repetition and error risk.

Break Free

It is time to break free from the standard for loop! Thus far, I have shown how Symbol.iterator offers significant advantages by encapsulating iteration patterns for a single array. What about multiple arrays?

Looping over two arrays pairwise in Vanilla JS quickly litters code with [i], but because this is a common task libraries like Lodash come with a _.zip function that combines multiple sequential collections into one collection.

Using variadic arguments and Symbol.iterator, we can easily write our own version of zip that iterates over multiple arrays but without needing to clone the entirety of each array.

Now we have a modular and readable way to iterate over multiple arrays at the same time! Extending the previous example to include a third dimension:

In terms of performance, it seems the Symbol.iterator version of zip is significantly faster than the one that clones arrays, while the range example above is comparable to a standard for loop.

Of course accurate, generalizable benchmarks are difficult and it is important to measure in performance-critical settings. But generally speaking, Symbol.iterator uses less memory and should be at least as fast, if not faster, than other approaches.

Yet another use of iterators, is to stop iterating!

This is similar to _.takeWhile, although again it does not require cloning.

One Last Time

Let me end with one more example. In the world of Snapchat and ephemeral messaging, what if we could create an Array using invisible ink?

Ok, not literally but you get the idea.

That is what I call VanishingArray, an Array that removes elements as you loop through them. Iterate once, and only once.

Before you move on too fast, it should be noted that Symbol.iterator has its downsides and should probably not be the default approach to every problem.

For one, it is often more complicated and verbose than simply chaining a combination of Array.filter and Array.map. There are also other approaches with equal or better performance characteristics, like data streams and libraries like Sequency.

That said, Symbol.iterator is versatile and efficient, and I have only presented a small subset of its utility. MDN has examples using generators with Symbol.asyncIterator (that’s right, asynchronous iterators!). If you are curious about the Symbol API more generally, Keith Cirkel has a great article on Metaprogramming in ES6.

If you found this helpful, say thank u before moving on to the next article 👏

And follow me on LinkedIn · GitHub · Medium

thank u, [Symbol.iterator].next was originally published in Frontend Weekly on Medium, where people are continuing the conversation by highlighting and responding to this story.

So the last shall be [0], and the first [length — 1].

– Adapted from Matthew 20:16

I’ll skip the Malthusian Catastrophe and get to it: arrays are one of the simplest and most important data structures. While terminal elements (first and last) are frequently accessed, Javascript provides no convenient property or method for doing so and using indices can be redundant and prone to side effects and off-by-one errors.

A lesser-known, recent Javascript TC39 Proposal offers solace in the form of two “new” properties: Array.lastItem & Array.lastIndex.

Javascript Arrays

In many programming languages including Javascript, arrays are zero-indexed. The terminal elements–first and last– are accessed via the [0] and [length — 1] indices, respectively. We owe this pleasure to a precedent set by C, where an index represents an offset from the head of an array. That makes zero the first index because it is the array head. Also Dijkstra proclaimed “zero as a most natural number.” So let it be written. So let it be done.

I suspect if you averaged access by index you would find that terminal elements are referenced most often. After all, arrays are commonly used to store a sorted collection and doing so places superlative elements (highest, lowest, oldest, newest, etc.) at the ends.

Unlike other scripting languages (say PHP or Elixir), Javascript does not provide convenient access to terminal array elements. Consider a trivial example of swapping the last elements in two arrays:

let faces = ["😀", "😉", "🤑", "😏", "🐐"];
let animals = ["🐒", "🐕", "🐈", "🐎", "🙄"];

let lastAnimal = animals[animals.length - 1];
animals[animals.length - 1] = faces[faces.length - 1];
faces[faces.length - 1] = lastAnimal;

The swapping logic requires 2 arrays referenced 8 times in 3 lines! In real-world code, this can quickly become very repetitive and difficult for a human to parse (though it is perfectly readable for a machine).

What’s more, solely using indices, you cannot define an array and get the last element in the same expression. That might not seem important, but consider another example where the function, getLogins(), makes an asynchronous API call and returns a sorted array. Assuming we want the most recent login event at the end of the array:

let lastLogin = async () => {
  let logins = await getLogins();
  return logins[logins.length - 1];
};

Unless the length is fixed and known in advance, we have to assign the array to a local variable to access the last element. One common way to address this in languages like Python and Ruby is to use negative indices. Then [length - 1] can be shortened to [-1], removing the need for local reference.

I find -1 only marginally more readable than length — 1, and while it is possible to approximate negative array indices in Javascript with ES6 Proxy or Array.slice(-1)[0], both come with significant performance implications for what should otherwise constitute simple random access.

Underscore & Lodash

One of the most well-known principles in software development is Don’t Repeat Yourself (DRY). Since accessing terminal elements is so common, why not write a helper function to do it? Fortunately, many libraries like Underscore and Lodash already provide utilities for _.first & _.last.

This offers a big improvement in the lastLogin() example above:

let lastLogin = async () => _.last(await getLogins());

But when it comes to the example of swapping last elements, the improvement is less significant:

let faces = ["😀", "😉", "🤑", "😏", "🐐"];
let animals = ["🐒", "🐕", "🐈", "🐎", "🙄"];

let lastAnimal = _.last(animals);
animals[animals.length - 1] = _.last(faces);
faces[faces.length - 1] = lastAnimal;

These utility functions removed 2 of the 8 references, only now we introduced an external dependency that, oddly enough, does not include a function for setting terminal elements.

Most likely such a function is deliberately excluded because its API would be confusing and hard to readable. Early versions of Lodash provided a method _.last(array, n) where n was the number of items from the end but it was ultimately removed in favor of _.take(array, n).

Assuming nums is an array of numbers, what would be the expected behavior of _.last(nums, n)? It could return the last two elements like _.take, or it could set the value of the last element equal to n.

If we were to write a function for setting the last element in an array, there are only a few approaches to consider using pure functions, method chaining, or using prototype:

let nums = ['d', 'e', 'v', 'e', 'l']; // set first = last

_.first(faces, _.last(faces));        // Lodash style

$(faces).first($(faces).last());      // jQuery style

faces.first(faces.last());            // prototype

I do not find any of these approaches to be much of an improvement. In fact, something important is lost here. Each performs an assignment, but none use the assignment operator (=).This could be made more apparent with naming conventions like getLast and setFirst, but that quickly becomes overly verbose. Not to mention the fifth circle of hell is full of programmers forced to navigate “self-documenting” legacy code where the only way to access or modify data is through getters and setters.

Somehow, it looks like we are stuck with [0] & [length — 1]…

Or are we? 😏

The Proposal

As mentioned, an ECMAScript Technical Candidate (TC39) proposal attempts to address this problem by defining two new properties on the Array object: lastItem & lastIndex. This proposal is already supported in core-js 3 and usable today in Babel 7 & TypeScript. Even if you are not using a transpiler, this proposal includes a polyfill.

Personally, I do not find much value in lastIndex and prefer Ruby’s shorter naming for first and last, although this was ruled out because of potential web compatibility issues. I am also surprised that this proposal does not suggest a firstItem property for consistency and symmetry.

In the interim, I can offer a no-dependency, Ruby-esque approach in ES6:

First & Last

We now have two new Array properties–first & last–and a solution that:

✓ Uses the assignment operator

✓ Does not clone the array

✓ Can define an array and get a terminal element in one expression

✓ Is human-readable

✓ Provides one interface for getting & setting

We can rewrite lastLogin() again in a single line:

let lastLogin = async () => (await getLogins()).last;

But the real win comes when we swap the last elements in two arrays with half the number of references:

let faces = ["😀", "😉", "🤑", "😏", "🐐"];
let animals = ["🐒", "🐕", "🐈", "🐎", "🙄"];

let lastAnimal = animals.last;
animals.last = faces.last;
faces.last = lastAnimal;

Everything is perfect and we have solved one of CS’ most difficult problems. There are no evil covenants hiding in this approach…

Prototype Paranoia

Surely there is no one [programmer] on earth so righteous as to do good without ever sinning.– Adapted from Ecclesiastes 7:20

Many consider extending a native Object’s prototype an anti-pattern and a crime punishable by 100 years of programming in Java. Prior to the introduction of the enumerable property, extending Object.prototype could change the behavior of for in loops. It could also lead to conflict between various libraries, frameworks, and third-party dependencies.

Perhaps the most insidious issue is that, without compile-time tools, a simple spelling mistake could inadvertently create an associative array.

let faces = ["😀", "😉", "🤑", "😏", "🐐"];
let ln = faces.length

faces.lst = "🙄"; // (5) ["😀", "😉", "🤑", "😏", "🐐", lst: "🙄"]

faces.lst("🙄");  // Uncaught TypeError: faces.lst is not a function

faces[ln] = "🙄"; // (6) ["😀", "😉", "🤑", "😏", "🐐", "🙄"]

This concern is not unique to our approach, it applies to all native Object prototypes (including arrays). Yet this offers safety in a different form. Arrays in Javascript are not fixed in length and consequently, there are no IndexOutOfBoundsExceptions. Using Array.last ensures we do not accidentally try to access [length] and unintentionally enter undefined territory.

No matter which approach you take, there are pitfalls. Once again, software proves to be an art of making tradeoffs.

Continuing with the extraneous biblical reference, assuming we do not believe extending Array.prototype is an eternal sin, or we’re willing to take a bite of the forbidden fruit, we can use this concise and readable syntax today!

Last Words

Programs must be written for people to read, and only incidentally for machines to execute. – Harold Abelson

In scripting languages like Javascript, I prefer code that is functional, concise, and readable. When it comes to accessing terminal array elements, I find the Array.last property to be the most elegant. In a production front-end application, I might favor Lodash to minimize conflict and cross-browser concerns. But in Node back-end services where I control the environment, I prefer these custom properties.

I am certainly not the first, nor will I be the last, to appreciate the value (or caution about the implications) of properties like Array.lastItem, which is hopefully coming soon to a version of ECMAScript near you.

Follow me on LinkedIn · GitHub · Medium

The first shall be last with JavaScript arrays was originally published in We’ve moved to freeCodeCamp.org/news on Medium, where people are continuing the conversation by highlighting and responding to this story.