MuSigma

Retrospective on CVE-2021-44228 (Log4Shell)

Fri, 10 Nov 2023 16:00:00 -0600

Apache Log4j 2 is an open source Java logging library that I’ve been contributing to and maintaining since 2014 as a member of the Apache Logging Services Project Management Committee (aka PMC). Up until late 2021, Log4j 2 was a fairly low profile project in contrast with its widespread usage across the industry. On 2021-11-24, the Logging PMC received a vulnerability report that would put Log4j 2 on the global stage just a couple of weeks later. That email was just one link in the chain of events leading to Log4Shell, one of the most impactful security vulnerabilities in over a decade. In this post, we’ll reflect on the history leading up to CVE-2021-44228 and CVE-2021-45046, how the Apache Way of open source software development prevailed in rapidly responding to these vulnerabilities, and what else we’re doing in Log4j 2 to help prevent any similar issue from happening in the future. Most of the information here was documented over a couple of weeks after the incident, and as we come close to the second anniversary of Log4Shell, I wanted to publish this publicly with some updates.

Background

The Log4Shell vulnerability works through a combination of features added over time that ultimately exploits design flaws in the Java Naming and Directory Interface (JNDI) API—a standard component of Java—to allow an attacker to execute remote code fairly easily. Dating back to the first release of Log4j 2 version 2.0-alpha1 while the project was still an experiment, a commit was added that changed how substituting property placeholders in configuration files work to support a similar configuration feature from Log4j version 1.2.x. Unfortunately, the way this was implemented applied property substitution to both configuration files and log messages when it was only intended to apply to configuration data. As this functionality was introduced during the experimental period of development, there aren’t any Jira issues to refer to for further historical context. While this functionality was not initially dangerous as the types of property lookup plugins available then were fairly benign, this was certainly unexpected behavior which wasn’t discovered until long after version 2.0 was released. The Log4j 2 team has always taken backwards compatibility seriously, and the scope of this compatibility included both the logging API and the default logging configuration behavior. There are other configuration options we’d love to make the default behavior—particularly around performance options—but broad changes like that will have to wait for version 3.0.

After several more pre-2.0 releases, a new feature was contributed to support property substitution for strings from the JNDI API. The provided use case was to support a configuration where log messages would be appended to different files depending on which web application context the logs were coming from. Specifying a common logging config for an entire Java EE cluster was a great way to reduce toil involved in updating logging configurations across a broad deployment. Being July 2013, the exploitability of the JNDI API was still unknown, so this feature was accepted and released as part of version 2.0-beta9. Combined with the log message property substitution behavior, the foundation was laid for making the default logging configuration susceptible to Log4Shell.

Later that year, I discovered the Log4j 2 project while researching what logging library to use in Java applications and began contributing to the project. In early 2014, I was invited to become a committer, and in later 2014, I was invited to join the PMC. It was that year when we finally released version 2.0 for general availability and set the stable compatibility point for the API and default configuration. Near the end of 2014, an issue was filed that discovered the log message property substitution behavior while debugging strange errors encountered from Apache Camel. The details of this issue weren’t made apparent until 2016 when the root cause was discovered and partially fixed in version 2.7.

One year later, another issue was filed and implemented to add a global option to disable message property substitution. This was released in version 2.10.0, and this flag was one of the early mitigations available for Log4Shell five years later. The flag was initially justified for performance reasons as a quadratic lookup function was being executed in a hot loop which didn’t seem to be necessary, though details of the exploitability of JNDI were already publicly known by the security community ever since a Blackhat 2016 talk published details around an earlier zero-day vulnerability exploited during Operation Pawn Storm. This zero-day was a vulnerability that was being exploited by attackers in the wild before Oracle were ever informed about the issue.

The First Vulnerability Report

On 2021-11-24, one day before the American holiday Thanksgiving Day, the Logging PMC received an email with the first vulnerability report. Included were exploitation details and an attached proof of concept project to demonstrate the exploit. Over the next 24 hours, the PMC discussed the implications of the vulnerability, potential solutions to patch the issue, and further details on how the exploit worked. On 2021-11-25, the PMC acknowledged the vulnerability with the reporter and continued internal discussions on how to address the issue over the next few days. During that time, the original reporter followed up with confirmations that Apache Flink, Apache Druid, and Apache Struts 2 were all impacted by this vulnerability. By the 29th of November, the PMC had settled on a root cause of the problem and solution, so a public issue was filed along with a pull request which made no reference to a security vulnerability, though astute observers may have deduced the seriousness of this issue before the CVE was later published. During the following week, we began preparing the codebase for a 2.15.0 release.

On 2021-12-05, the proposed fix for the vulnerability was merged, and version 2.15.0-rc1 was cut the following day. While the release candidate was being reviewed by the PMC during a standard 72-hour voting period, the original reporter emailed us on the 8th to warn that they discovered public discussions among security teams on social media about a Log4j RCE vulnerability, though they weren’t sure if they knew the full extent of the problem. Given the lack of evidence for anyone having exploited the issue before they had first reported the issue back in November, this would disqualify the use of “zero-day” vulnerability, though it did lead to zero-day vulnerabilities in tons of existing software. On the 9th, the reporter emailed us again to show that the proposed fix was insufficient, so the release candidate vote was cancelled. Later that day, a new release candidate was cut with a more comprehensive fix for the issue along with an expedited release vote process. During this release vote, we received more vulnerability reports related to the failed rc1 vote, though these were all the same problem identified by the original reporter earlier that day. The vote passed, and Log4j 2.15.0 was officially released with many items in the changelog, most notably a fix for a CVE rated at 10/10 in severity. The following day, I woke up to a bursting inbox and a handful of text messages from various journalists looking for details on the breaking story that eventually gained the moniker of Log4Shell.

The Second Vulnerability Report

Over the next few days, we received two independent vulnerability reports that the fixes provided for CVE-2021-44228 were insufficient to protect from exploitation in certain non-default configurations. During this time, the Logging PMC gathered in a long-running private video chat while the impact of Log4Shell began to spread throughout the world. It was discovered that the property substitution was still being applied to a couple other Pattern Layout components which could contain attacker-controlled input. Methods to bypass the newly introduced filtering for the JNDI lookup were demonstrated, and other places where JNDI could potentially be abused were identified and hardened. We quickly addressed these issues and created versions 2.16.0 (for Java 8+), 2.12.2 (for Java 7), and 2.3.1 (for Java 6) to publish the fix along with a backport of the two CVEs respectively. CVE-2021-45046 was initially published with a lower severity than it currently has at 9.0, and additional details were later reported to us independently by four separate sources that demonstrated how CVE-2021-45046 was a remote code execution vulnerability. Various other reporters informed us on more variants of this later on, but they were all either caused by the same issue or were less severe than the RCE aspect.

Of course, this wasn’t the final security vulnerability we fixed in December 2021, but these two issues were the only ones that led to serious problems like remote code execution. Since then, an issue related to denial of service and another related to unexpected behavior when using JNDI for configuring other components were fixed in CVE-2021-45105 and CVE-2021-44832 respectively. For more information about these CVEs, see the Log4j security page.

Future of Log4j

These security vulnerabilities raised a lot of questions and laid bare the consequences of the industry not keeping inventory of the software they use let alone the more detailed inventory of subcomponents that make up each piece of software. The security process established at the Apache Software Foundation worked great at delivering security updates for the public as quickly as possible, though it turned out that many of our users were unaware they were even users of Log4j 2. When I first wrote this for a limited audience, 35% of recent downloads of Log4j 2 from Maven central were for vulnerable releases. Nearly two years later, this has reduced to 23% of recent downloads which is an improvement. In the Log4j project, we have a couple ideas we’re working on that will help prevent similar problems in the future.

A primary cause of Log4Shell was a packaging philosophy shared by Java itself until Java 9 finally broke itself up into well-defined modules. To simplify upstream packaging and downstream use, many optional features were included in the standard distribution. During the early 2010s when Log4j 2 was initially developed, build tools like Apache Maven and Gradle were not as dominant while tools like Apache Ant or even Make were used with manually checked-in copies of third party library jar files. The previous major version of Log4j was packaged as a single jar; while convenient for the even more primitive build tooling of the late ’90s and early 2000s, this also had the design flaw of mixing the external and internal APIs together. Version 2.0 was initially packaged into two main jars: the frontend log4j-api and the backend log4j-core. The API has strong compatibility support from version to version, and the core backend contains all the implementations available as plugins for use in a logging configuration. Many optional plugins are included here which can only be used when their required dependencies are present on the classpath. This packaging format was chosen to maximize simplicity of deployment, though even this split-jar setup has caused its own large amounts of confusion to Java developers of all experience levels; some users are first learning what jars are while other users don’t even know the difference between log4j-api (the frontend) and log4j-core (the backend).

For version 3.0, we are planning to break up log4j-core into several additional modules while leaving a slim, secure kernel. We’ve published one alpha release of 3.0 so far, and we want to make a beta release before 3.0.0. While the updated log4j-core will still contain many common plugins, this reduces the required module dependencies to a base requirement of java.base. Users will have to opt in to fancier features that rely on dependencies outside the java.base module provided by Java 11 as those plugins will be packaged in their own modules. We are also adding basic feature flags to select plugins so that they can require explicit opt-in even when the module is present at runtime to further protect from abuse of potentially insecure plugins.

Conclusion

In the time since these issues were fixed, the project has received various levels of sponsorship and donations. This has helped a lot as we’ve been able to dedicate more time to working on improvements to our release process, security metadata, code and test quality, and several other aspects of the project. As world governments debate how to regulate software engineering, we continue to lead by example both in the Logging Services PMC and the Apache Software Foundation more broadly.

Log4j 2 continues to improve over time with a healthy community and a fresh wave of widespread interest in reviewing the project for any sign of potential security issues. As always, we encourage others to contribute to the project and other projects at Apache; community is the power behind the Apache Way. It’s important that everyone using the project make sure they’re using the latest version. Users still on Log4j 1 should upgrade to Log4j 2 or use our Log4j 1 compatibility features which are supported by the PMC.

An abridged guide to using ed25519 PGP keys with GnuPG and SSH

Sun, 09 May 2021 11:00:00 -0500

I came across a great guide to using a YubiKey with SSH and GPG a couple years ago which helped push me over the fence and acquire my own YubiKey. Following that setup guide, I set up my keys offline using a Tails Linux boot USB with a OneRNG hardware random number generator. While a fun exercise, I must note that it’s not for the faint of heart, especially if done on a recent MacBook Pro (with a touchbar) or incompatible hardware. However, it did help explain some of the features available in GnuPG, and this came in handy recently while exploring the new support for elliptic curve cryptography in YubiKey firmware 5.2.3, the version installed in my later YubiKey 5Ci purchase. While I originally created PGP keys using the same guide last year with RSA keys, since those keys were expiring soon, it seemed like a good idea to look in to switching to Curve25519 keys. GnuPG has added some improved support for this algorithm along with supporting this updated YubiKey firmware to transfer these keys to a YubiKey. In this brief guide, I’ll go over how to generate an appropriate PGP key that can be used both in a YubiKey and for use with SSH. For more general info about using smartcards with GnuPG, see this guide from GnuPG.

About PGP Keys

PGP keys are slightly more complicated than a single keypair, and they’re fairly flexible. A key has one or more user ids (name, email address, and an optional comment), one of which is flagged as the primary uid (by default, this is the original uid used on creation). Keys have an optional expiration date along with a list of usages allowed for that specific key. Keys have one or more subkeys which allow for different keys for different use cases all on the same key. These use cases include certification (to create and certify subkeys typically), signatures, encryption, and authentication. Each key or subkey can allow one or more of those capabilities, though certification is typically reserved for the primary key. When using RSA, this is fairly simple as RSA supports all four of those capabilities. However, elliptic curves typically use dual algorithms for signing and encryption, so this will be slightly more nuanced.

Generating PGP Keys

Our master key will be created for certification with no expiration date. This is justified by the fact that the certification key will be physically tied to a YubiKey for which we can use a revocation certificate if the key is lost or stolen. The subkeys will have expiration dates to allow for key rotation as explained in the linked guide, though that is out of scope for this post. We’ll split up the remaining usage capabilities into three subkeys: a signing subkey, an encryption subkey, and an authentication subkey. Note that certification and authentication keys use signature algorithms internally, thus for our key, we’ll use ed25519 for all but our encryption subkey which will instead use cv25519.

The following commands will help us avoid using the UI for most of the work. Use this gpg.conf file in ~/.gnupg/gpg.conf for a more secure default. Then, create a certification master key (note the optional comment can be omitted entirely, no parenthesis necessary) and specify a password for the key when prompted:

gpg --quick-generate-key \
    'Your Name <your.email@example.com> (optional comment)' \
    ed25519 cert never

Next, save the key fingerprint without spaces to an environment variable:

export KEYFP=123456789ABCDEF0...

Now add subkeys for signing, encryption, and authentication. These will have expiration times to allow for key rotation. For example, using a one year expiration time:

gpg --quick-add-key $KEYFP ed25519 sign 1y
gpg --quick-add-key $KEYFP cv25519 encr 1y
gpg --quick-add-key $KEYFP ed25519 auth 1y

Next, verify the keys have been added to your local keyring:

gpg -K

This should give output like:

sec   ed25519/0x0123456789ABCDEF 2021-01-01 [C]
      Key fingerprint = 0000 1111 2222 3333 4444  5555 0123 4567 89AB CDEF
uid                   [ultimate] Matt Sicker <mattsicker@apache.org>
ssb   ed25519/0x9876543212345678 2021-01-01 [S] [expires: 2022-01-01]
ssb   cv25519/0x3141592653589793 2021-01-01 [E] [expires: 2022-01-01]
ssb   ed25519/0x8888ABCDFDEC8765 2021-01-01 [A] [expires: 2022-01-01]

Using SSH

The authentication subkey can be used for authentication in SSH. There are a few different ways to export the PGP key into an OpenSSH format, and we’ll cover a simple one here. This method will use GnuPG as an SSH agent which allows us to both use PGP keys for SSH as well as to import and encrypt existing SSH key files into a GPG-managed SSH keyring. First, obtain a copy of this gpg-agent.conf file and save it to ~/.gnupg/gpg-agent.conf. Uncomment the appropriate pinentry program for your platform (and don’t forget to install it if you haven’t already). Next, add the following lines to your ~/.bashrc, ~/.zshrc, or whatever shell rc file you use:

export GPG_TTY="$(tty)"
export SSH_AUTH_SOCK=$(gpgconf --list-dirs agent-ssh-socket)
gpgconf --launch gpg-agent

Note that this configuration is only relevant for your local machine. If you’re using SSH agent forwarding, don’t copy this configuration to remote machines. After restarting your shell, run ssh-add -L to list the currently known SSH keys in GnuPG. Your locally stored keys should be listed here as well as your YubiKey if you’ve transferred the key there and have it plugged in. These lines correspond to SSH public keys which can be used in your ~/.ssh/authorized_keys file on relevant remote machines to SSH into. Alternatively, you can export a GPG’s authentication key into an SSH format directly using the following command:

gpg --export-ssh-key 0x1234ABCD1234ABCD

For use with GitHub and other git+ssh providers, add this public key to your account’s SSH keys.

Exploring the ChaCha stream cipher

Sat, 06 Feb 2021 18:00:00 -0600

Stream ciphers form the basis for simpler encryption and decryption algorithms than traditional block ciphers like AES. In particular, the ChaCha family of stream ciphers form the basis of the encryption functionality in O(1) Cryptography which we’ll explore in more detail. Originally published as the Salsa family (PDF) of ciphers, ChaCha (PDF) makes some small modifications to Salsa for increased security while maintaining equivalent performance. ChaCha has since been widely standardized in various networking standards and programming language standard cryptography libraries as an alternative to AES and other ciphers. Recall that a stream cipher is an algorithm that takes a secret key and an input stream which returns an output stream of the same size of the input stream. Stream ciphers work by taking the secret key (and usually some sort of nonce or initial value which cannot be reused for the same key) and generating a stream of deterministic random bits called the keystream which is used for multiple purposes. The primary use of this keystream is to xor it with an input stream of plaintext or ciphertext to produce an output ciphertext or plaintext respectively. Given this mode of operation, compared to block ciphers which require complicated key scheduling algorithms, it can be hard to imagine why block ciphers have been so popular historically speaking. Surely a stream cipher isn’t that simple, is it?

Overview

ChaCha and Salsa are stream ciphers that expand a 256-bit key into 2⁶⁴ randomly accessible streams of 2⁶⁴ randomly accessible 64-byte (512-bit) blocks. They are parameterized by a round number suffix, recommended at 20 (as in ChaCha20 or Salsa20), but also available in 8 and 12 round variants with reduced security margin. This round number controls the number of times the round function is applied to the cipher’s internal state and must be an even number. Each round applies a sequence of constant-time operations on an array of 16 32-bit words consisting of four addition, xor, and constant-distance left shift and rotate operations each. The choice for these operations relies on the mechanical sympathy of how CPUs physically implement addition, xor, and shift/rotate instructions, all of which are both fast and operate in constant time regardless of input. This set of operations is also functionally complete, so despite seeming simple, they can simulate any other Boolean expression or logic gate which makes them sufficiently powerful.

Internal State

The internal state of a ChaCha cipher consists of a 512-bit block of data addressed as 16 little endian 32-bit unsigned integers. Keeping the entirety of the cipher state inside this buffer along with the operations performed on it allows CPUs to keep the state in its cache which helps maximize software performance while simultaneously preventing timing attacks based on memory access patterns common to optimized software implementations of AES. This state is initialized with a secret key, a constant value, and some input data formed from a nonce and initial counter integer. The constant value is what is known as a nothing up my sleeve number and is needed as part of the standard initialization of the cipher state, and in ChaCha, this constant is the 16 byte ASCII-encoded string “expand 32-byte k” which fills the first four little endian integer values of this state. A “nothing up my sleeve number” is an arbitrarily chosen initialization constant value that is used in a cipher where one is needed such that it’s clear that the choice of constant was arbitrary and is therefore unlikely to have intentional backdoors, a problem faced by the secrecy behind parameter choices in DES recommended by the NSA in the 1990s. Such numbers are usually encodings of mathematical constants or ASCII strings. The next eight integers consist of the 32-byte secret key interpreted as an array of eight little endian 32-bit integers key₀, …, key₇. Finally, the last four integers consist of the initial counter and nonce encoded similarly. This last aspect has three main variants on how to encode the input data: a 64-bit initial counter with a 64-bit nonce; a 32-bit initial counter with a 96-bit nonce; or a 128-bit nonce. These correspond to the original ChaCha cipher, the IETF standardized variant, and HChaCha respectively, the latter being used in the XChaCha variant of ChaCha which uses an extended nonce. Laying this out in a matrix, the internal state looks like this:

0x61707865	0x3320646E	0x79622D32	0x6B206574
key₀	key₁	key₂	key₃
key₄	key₅	key₆	key₇
input₀	input₁	input₂	input₃

These bottom input values correspond to attacker-controlled input which are positioned to reduce the flexibility attackers have in cryptanalysis of this cipher compared to Salsa which organizes the initial state in a different configuration with attacker-controlled input values in cells 6 through 9. This state is updated by a sequence of invertible operations defined by applying a round function the round number of times.

Round Function

ChaCha defines its round function using a smaller sub-operation known as the quarter-round which is applied four times per round using a specified permutation. This quarter-round is responsible for the entirety of the underlying bit-shuffling taking place to produce a keystream and is defined using the following algorithm.

void quarterRound(int[] state, int a, int b, int c, int d) {
    state[a] += state[b];
    state[d] = Integer.rotateLeft(state[d] ^ state[a], 16);

    state[c] += state[d];
    state[b] = Integer.rotateLeft(state[b] ^ state[c], 12);

    state[a] += state[b];
    state[d] = Integer.rotateLeft(state[d] ^ state[a], 8);

    state[c] += state[d];
    state[b] = Integer.rotateLeft(state[b] ^ state[c], 7);
}

Finally, a round consists of a column round or a diagonal round in alternating sequence. A column round consists of a quarter-round applied to each of the four columns. A diagonal round consists of a quarter-round applied to four diagonal permutations of the state.

void columnRound(int[] state) {
    quarterRound(state, 0, 4,  8, 12);
    quarterRound(state, 1, 5,  9, 13);
    quarterRound(state, 2, 6, 10, 14);
    quarterRound(state, 3, 7, 11, 15);
}

void diagonalRound(int[] state) {
    quarterRound(state, 0, 5, 10, 15);
    quarterRound(state, 1, 6, 11, 12);
    quarterRound(state, 2, 7,  8, 13);
    quarterRound(state, 3, 4,  9, 14);
}

ChaCha20 consists of 20 rounds, and this block shows the application of two subsequent rounds. Therefore, a complete application of ChaCha20 will apply the above two rounds 10 times to produce a 64-byte block in the keystream.

void permute(int[] state) {
    for (int i = 0; i < 10; i++) {
        columnRound(state);
        diagonalRound(state);
    }
}

At the end of each key block, we interpret cells 12 and 13 as a little endian 64-bit counter which is incremented in place before generating the next 64-byte keystream block.

void incrementCounter(int[] state) {
    if (++state[12] == 0) {
        ++state[13];
    }
}

Combined with functions to decode and encode bytes to and from arrays of integers, data streams and keystreams can be fairly easily combined. These ChaCha functions are also used for deriving subkeys in the case of ChaCha20-Poly1305 authenticated encryption and for deriving subkeys and sub-nonce data for implementing XChaCha20-Poly1305. Using the keystream output from ChaCha can be used to implement a deterministic random bit generator. When combined with a message authentication function like Poly1305, ChaCha can form the basis for an authenticated encryption with authenticated data algorithm as standardized in RFC 8439. The ultimate advantage to using a cipher like ChaCha that permits efficient secure software implementations is that it can be widely used by less powerful devices or devices lacking intrinsic AES-related instructions for efficient secure hardware implementations. Many existing software AES implementations are vulnerable to a considerable number of attacks depending on the threat model, and proper software implementations without appropriate operating system or hardware support may suffer performance problems leading to deployment of insecure code in practice. Several of these attacks are detailed in Cache-timing attacks on AES (PDF) along with advice on how to mitigate this in software, though one of the key conclusions you might come to is that AES is something to be avoided where possible.

In a future post, we’ll go over how Poly1305 works, combine it with ChaCha, and ultimately define the authenticated encryption with authenticated data (AEAD) algorithm XChaCha20-Poly1305 used in O(1) Cryptography. In the meantime, you can have a look at some of the academic papers and references cited in O(1) Cryptography.

Building a Deterministic Random Bit Generator

Sat, 30 Jan 2021 17:00:00 -0600

As part of developing the O(1) Cryptography library, I initially relied upon the standard Java cryptography APIs for cryptographic random bit generation. Unlike most of the cryptographic APIs in Java that are frequently misused, java.security.SecureRandom is about as simple as it gets for defining all the relevant operations of a secure random number generator. Naturally, like most of the standard APIs, this one, too, can be misused if configured incorrectly, most importantly in the underlying seed generation strategies which are platform-specific. Java provides a few different strategies out of the box, and in pure Java code, I rely on this API to seed the O(1) Cryptography deterministic random bit generators. In C, there are some nicer integrations with OS-specific system calls and even hardware-specific integrations more readily available, though Java can access some of these if they’re provided via some PKCS11 library. As a technical note, the only relevant limitation we have in Java compared to C is that the Java operations will generally require opening a file or accessing some resource which can potentially fail due to file descriptor leaks or other resource exhaustion, while the C code can make use of system calls that bypass the file interface entirely. Depending on the underlying threat model, this can be a vulnerability if the process cannot open /dev/random or equivalent device files. With that in mind, let’s take a look at how a random bit generator works.

The canonical U.S. standard for cryptographic random bit generators is NIST Special Publication 800-90A Revision 1 which specifies three mechanisms to do so using standard cryptographic primitives. These mechanisms include one based on plain hash functions, one based on keyed hash functions (HMAC specifically), and a third based on block ciphers such as AES. In O(1) Cryptography, I’ve implemented two strategies using the primitives available here: one based on a keyed BLAKE3 hash function, and another using ChaCha20 as a pseudo-block cipher. Both strategies have some commonality beyond their underlying permutations (BLAKE3 uses the same quarter-round mixing function as ChaCha20).

First, a DRBG instance is lazily initialized on a per-thread basis using system-specific seed entropy. Java makes this fairly simple with the ThreadLocal class and relying on SecureRandom for accessing system-specific entropy sources for generating a seed. C raises the challenge of not having a standard runtime, though it does have standardized thread-local storage support. On the other hand, C gives us access to lower level APIs which have their own advantages to the Java equivalents.

One of these APIs is the function getentropy which uses the Linux system call getrandom, a function that is used as the basis for /dev/random and /dev/urandom. On BSDs and macOS, getentropy is itself the system call with the same signature that libc borrowed. For older POSIX platforms that don’t expose a system call, reading seed data from /dev/random can be supported, though I’ve elided support for it currently as it involves file IO which starts to bloat the C library beyond what’s minimally necessary to support the Java API. On Windows, there appears to be a long history of APIs here, the most promising sufficiently low level one being RtlGenRandom, a function from the advapi32 library which has been a fairly standard base library on Windows since the XP days. An interesting source to look at would be non-standard hardware entropy sources like the OneRNG, an open source hardware RNG which is typically accessible in a platform-independent manner via serial port access APIs besides any of the integrations offered specifically for Linux.

Each implementation uses this seed data slightly differently, but they both use the seed as an initial key. The ChaCha20 implementation also uses the seed for a nonce and initial counter. In order to generate random bytes, this uses the keystream output (the resulting ciphertext of encrypting null bytes) for each request and then ratchets itself by using an incrementing nonce to provide forward secrecy. The BLAKE3 implementation generates bytes by finalizing the hash output of an empty input with the first 32 bytes being used solely to rekey the hash function as its ratchet after using the subsequent bytes as the output bytes. Both implementations maintain internal counters to track when reseeding is necessary.

It may be interesting to note how simply the concepts from the underlying stream cipher and extensible output function primitives made implementing a DRBG much simpler than the required steps to do the same with AES or HMAC-SHA2. Since O(1) Cryptography is an opinionated cryptography library with the goal of being easy to use and hard to misuse, this philosophy is apparent in both its APIs and its choice of cryptographic primitives. Using the current NIST standards, there is currently only one primitive available that can be as easily used: SHAKE128/SHAKE256. SHAKE is the extensible output function variant of SHA-3 which has a comparable API to BLAKE3 while being fairly slower. Using this same pattern, it’s fairly simple to build a DRBG using any cryptographic sponge function such as Xoodyak or any sponge-style permutation like Ascon as either cipher-based generators or keyed hash ones. Some of these ideas are also included, though it remains to be seen where lightweight cryptography standards converge, so they are only available as experimental implementations.

I hope this helps demystify how a cryptographic random bit generator can be made using cryptographic primitives. DRBGs can get more complicated than this by adding an interface to accept user-provided seed data to include in its seed input, maintaining buffers, and gathering various system state to use in the reseed algorithm, though I’ve tried to keep things simple by standing on the shoulders of operating systems and hardware support instead. Plus, Java’s standard SecureRandom implementation already handles this for us where necessary.

These random generators will be an important tool in our cryptographic toolbox later on when I go over the design of other parts of O(1) Cryptography in subsequent blog posts. Random data are integral to our ability to generate keys and challenges, and the strength of our cryptosystem is only as good as its fundamental parts. Ensuring that random data can be generated fast and in parallel is a clear requirement for any proper use of cryptography, so perhaps it may help to keep in mind one of the implicit design goals of O(1) Cryptography: speed. Performance problems are a common source of security vulnerabilities when security measures get disabled due to interference with core application logic. Using high performance cryptographic primitives prevents the need to tweak security parameters improperly; using primitives that avoid overly complex configuration options goes a step further by preventing insecure tweaks in the first place.

Introducing the O(1) Cryptography Project

Thu, 28 Jan 2021 18:00:00 -0600

Cryptography is a fascinating subject at the intersection of pure math and computer science that has become nearly ubiquitous over the past several years. Similar to functional programming, cryptography holds a particular interest to me because of the pure math connections, and my work in security software engineering has brought me back into the topic over the past year or so. Diving in to the standard Java cryptography APIs, I quickly discovered a tangle of strange naming conventions and poorly documented cryptographic primitives. There have been numerous academic papers written studying the widespread phenomenon of misuse of Java cryptography APIs, and after spending a bit of time with them, I can easily see why this is. Java, in its quest to remain low level and generic, oftentimes provides overly complicated APIs that, while flexible for low level use, offer very little guidance in their correct use. Some of this might be attributed to typical overengineering common to Java APIs prior to Java 8, but this particular API is more like a C API ported directly to Java. In fact, there’s a fairly innocuous explanation for it: it’s the same basic pattern present in most historical cryptographic libraries, most of which are indeed written in C by academics with little care for engineering. Combined with the historical baggage of having to deal with cryptographic export controls back in the 1990’s and early 2000’s, the Java cryptography API, like most cryptographic software written before export controls were relaxed, suffered from security vulnerabilities by design.

Much has been written and discovered since these dark days, though most software still struggles with outdated cryptography practices and misuse. In an effort to help software developers incorporate strong cryptography into their applications, an effort must be made to create and use cryptographic software components that are both easy to use and hard to misuse. One of the longest running efforts with a similar philosophy is the OpenBSD operating system which has incorporated strong cryptography and security by default for the past 25 years. Unconstrained by American cryptographic export controls in its home of Canada, the project has been one of the great examples of pervasive use of cryptography and secure design throughout its codebase. With simplified export controls, particularly for free and open source software, this philosophy must expand and help improve the security, privacy, and safety of the software we all write.

Using a corpus of public domain algorithms and cryptographic knowledge, I’ve started the O(1) Cryptography Project where I’m developing an opinionated cryptography library that aims to be easy to use and hard to misuse. O(1) Cryptography is a Java and C library that bundles the latest best practices in cryptography by providing abstractions for common cryptographic use cases. The name is a pun on the idea that cryptographic algorithms should generally run in constant-time and with minimal differentiation in its observable state. These properties seem to be somewhat modern to cryptography in that computers are fast enough and powerful enough now to statistically differentiate non-constant-time cryptographic algorithmic output, though they weren’t as big a concern back at the turn of the 21st century when AES was standardized. In fact, most standardized cryptography until recently was developed almost entirely detached from the reality that one day, non-cryptographers might need to use this stuff, too. There simply aren’t enough cryptographers in existence to write custom cryptographic routines for every application that needs it, and older cryptographic libraries were not designed for non-experts. Maybe you’ve heard of the various acronyms from yesteryear like AES, DES, RC4, RSA, DSA, MD5, SHA1, and many more. Some of these primitives are still useful and secure, but they all have problems of one form or another. In particular, they all fail the two-prong test of being easy to use and hard to misuse.

For example, AES is hard to use: as it was standardized before the consensus formed that authenticated encryption was the way to go, it requires pairing with an authentication algorithm which is commonly forgotten in practice. Implementers of AES are not discouraged from writing various optimizations that leak information about the underlying encryption key. It is fairly easy to misuse AES on both the implementer side and the application side. Being a block cipher, in order to encrypt data longer than the length of the encryption key, a mode of operation must be specified, and many commonly implemented modes have their own security vulnerabilities besides a lack of authentication. Another example is RSA, the pair of signature and asymmetric encryption algorithms that are frequently misused and improperly implemented. Due to its large key size, temptations to cut corners have run rampant through history in many implementations. Misunderstandings in the use of symmetric versus asymmetric encryption have led to people using RSA to directly encrypt data, direct use of the Diffie-Hellman shared secret result for encryption, duplicate use of keys, and many other security failures.

Between 2005 and 2010, Prof. Daniel J. Bernstein at University of Illinois at Chicago published a few papers that form the basis for much of the underlying cryptographic primitives central to O(1) Cryptography. In particular, the Salsa20/ChaCha20 family of stream ciphers and extended nonce versions, the Poly1305 one-time authenticator, and the elliptic curve Curve25519, were all detailed during this time. A more detailed listing of these various foundational papers are listed in the O(1) wiki, though the common theme behind the choice of primitives for this library are that they, too, are designed with the philosophy that they should be easy to use and hard to misuse. Another interesting theme is that many of these algorithm choices have been included into various IETF standards such as TLS 1.3 and SSH, so their use has clearly become far more widespread than their initial years.

Naturally, I am not the first to develop such a library, and there is prior art that inspires this library. The general idea behind making a cryptographic library that is easy to use and hard to misuse is the central concept of the polyglot Themis cryptographic framework. The choice of algorithms featured have been strongly influnced by Prof. Bernstein’s old NaCl library which formed the basis for libsodium, the essential C library with a similar philosophy (or at least as easy to use as C can be given that it’s C). Special thanks to Frank Denis, the maintainer of libsodium and developer of much of the zig standard crypto library, for further widening the rabbit hole of cryptographic primitives to explore and support, particularly in the field of lightweight cryptography.

This library is still under heavy development, particularly in the area of documentation and the higher level APIs. Much of the primary cryptographic primitives have been ported to Java where needed, and native implementations are also available. While the primary concern is to develop the Java API first, due to the eventual inclusion of C code from the Fiat Cryptography project for formally verified elliptic curve functions, I am also considering what other languages make sense to provide facades for, especially non-JVM ones that would benefit more from the native code. Pure Java versions of the algorithms are all available, though optimized native versions are also provided as an option. Similar to Apache Log4j, it may make sense to create Scala and Kotlin APIs, but those are currently out of scope for initial release.

Using Travis CI for Continuous Delivery

Tue, 27 Mar 2018 17:00:00 -0500

For almost two years, I’ve been using GitHub Pages to host my blog. This tool handles a lot of things for you behind the scene: it builds your site and deploys it to a web server. GitHub implements this as a great example of continuous delivery, the marriage of continuous integration with continuous deployment. GitHub also lets you use your own custom domain name, but it doesn’t support TLS connections over them. Thus, I had to take matters into my own hands and go back to self hosting.

Enter my old pals NearlyFreeSpeech.NET, Let’s Encrypt, and Travis CI. This host provides SSH access, and Travis provides easy ways to set up CI/CD pipelines. Let’s get started on building the GitHub Pages site first which itself is based on Jekyll. This static site generator is written in Ruby, so in order to build this, we need a recent Ruby environment. Then a few commands can get you bootstrapped:

gem install bundler
bundle install
bundle exec jekyll serve

This command serves an incrementally compiling view of the site. We can change serve to build to simply get the output. Now we need to deploy these output files via SSH. There are several different ways to copy files over SSH, and in today’s post, we’ll be piping data directly through an SSH pipe.

tar -czf - -C _site . | ssh deploy@ssh.example.org 'tar -xzf - -C /var/www'

By using tar, we bundle up the entire _site/ directory and compress it, and we output to stdout. By changing directories into what we want to compress (using -C), we’ll be able to easily extract it wherever we want. Now that we have the basics, let’s integrate it with Travis.

First, let’s get started with continuous integration of our website. We need to add a minimal .travis.yml config file to declare our pipeline:

language: ruby
cache: bundler

before_install:
  - gem install bundler

script:
  - bundle exec jekyll build

In our minimal config, we declare the programming language in use (Ruby), enable package caching for Bundler, setup commands to prepare the build environment, and build commands. After enabling our repository in Travis, we can now see our site get built on commit. This can help detect errors in configuration or anything verifiable that we add tests for.

Next, we need to configure deployment. Travis provides a simple way to encrypt files to store in a git repository which can be decrypted inside the running Travis build container. We’ll use this to encrypt an SSH private key file to be used for copying our site build artifacts. Let’s set up a deploy key.

echo .travis_deploy_key >>.gitignore
ssh-keygen -C 'deploy@travis-ci.org' -f .travis_deploy_key

Next, we need to install the Travis command line tool and log in.

gem install travis
travis login

Next, we encrypt the file and update our Travis config.

travis encrypt-file .travis_deploy_key -a

This will output a file named .travis_deploy_key.enc which can be safely committed to git. The unencrypted file should not be committed, and the provided example adds it to .gitignore to prevent it as such. Now we can add some more settings to .travis.yml.

language: ruby
cache: bundler

addons:
  ssh_known_hosts: ssh.example.org

before_install:
  - openssl aes-256-cbc ...
  - gem install bundler

script:
  - bundle exec jekyll build

after_success:
  - eval "$(ssh-agent -s)"
  - chmod 600 .travis_deploy_key
  - ssh-add .travis_deploy_key
  - "tar -czf - -C _site . | ssh deploy@ssh.example.org 'tar -xzf - -C /var/www'"

branches:
  only:
    - master

Of note here is the after_success section. The first three commands are used to set up ssh-agent so that any ssh program can use its keys without user input. The last command is the same one from earlier for deploying the site artifacts. We also have a branches section which we use to restrict this pipeline to only execute on the master branch. This way we can use branches to store blog post drafts and other things without them being deployed as well.

We can adapt this process to build unsupported programming languages as well such as LaTeX. To do so, we can use the apt addon for Travis.

addons:
  ssh_known_hosts: ssh.example.org
  apt:
    packages:
      - texlive-full

before_install:
  - openssl aes-256-cbc ...

script:
  - pdflatex cv.tex
  - pdflatex cv.tex

after_success:
  - eval "$(ssh-agent -s)"
  - chmod 600 .travis_deploy_key
  - ssh-add .travis_deploy_key
  - scp cv.pdf deploy@ssh.example.org:/var/www/

branches:
  only:
    - master

The Art of Logging

Mon, 06 Nov 2017 08:00:00 -0600

All developers have attempted to debug their programs by printing lifecycle and state information to the console. This concept, sometimes known as printf debugging, can be far more powerful of a tool than one might first expect. The essence of this debugging technique is the concept of logging, where developers add relevant information about the state of the running program to a log. The use of logging is vital to both developers and operators, and it is important to understand how and why to use logging from both perspectives.

Logging Fundamentals

Logging is far more than just printing to stderr, however. Typical logging systems are divided into a set of logging levels that generally define the audience and semantics of the log event. For example, in Apache Log4j 2, logging levels are divided into the following set:

Fatal: error messages that indicate that some subsystem or the entire program cannot continue execution and will terminate.
Error: error messages regarding a problem that should be handled by a human. These are generally useful for operators to alert on.
Warn: warning messages regarding potential problems that may need to be handled by a human. This level is often misused and ignored as a result.
Info: informative messages about the state of a program. These types of messages tend to be related to the lifecycle of a program and can be viewed as a way to debug the macro state of the program.
Debug: debugging information about internal states of the program. These messages are usually only helpful to the developers maintaining a program.
Trace: messages tracing the execution flow of a program. These messages are usually very low level and simply mirror the micro state of a program and generally don’t offer more information than a debugger would.

Some logging systems define other levels, but most logging systems categorize their log messages into similar buckets with similar use cases. Each level can be selectively enabled or disabled, though generally disabling one level will disable all levels below it as well. For example, if we used a logging configuration that was set to WARN as its level, then only warnings, errors, and fatal messages would be shown.

By simply adding severity information to log messages, we have already surpassed the functionality offered by printf, but we’ve only scratched the surface. Any given program is generally large enough to be made up of some sort of concept of modules or subsystems, so it seems like it would be useful to extend this configurable flexibility to subsystems as well. In Java programs, these tend to be separated by packages and classes, though the important concept to use here is that of a named logger. By naming the loggers used in a program, each subsystem can be independently configured to only output logs that are desired. For example, suppose a third party library is misusing the warning level and causing operations to be concerned about the health of your application. After verifying that every warning log message under the logger name prefix com.example.subsystem are not real warnings, we can use a higher level threshold for that set of loggers specifically while not having to disable warnings globally or modify the third party library’s source code. This also relates to the idea that logger names form a hierarchy; com.example is the parent of com.example.subsystem. This allows for simpler ways to configure entire subsystems in one setting.

At this point, we have a rather powerful abstraction over log event filtering using both a level and a name, but we can do better! An additional piece of metadata can be attached to log events: markers. A marker is a simple text string to mark some sort of cross-cutting concern of a particular log message. This can be used to help route specific log messages to different logging systems. For example, suppose a log message is marked with the ALERT marker. The logging configuration could have a filter for that marker which would route these messages regardless of level or logger name to a particular destination. This might be an alerts channel in Slack or an alerts mailing list.

In some programming languages such as Java, string manipulation is considered a somewhat low level operation, thus there are certain string templating features not present here that would be useful for logging. For example, logging a message that contains values from some local variables would normally require string concatenation, and if that log message is never displayed, then said concatenation was wasted CPU effort. Little things like this can add up over time to form a significant performance overhead, so we can certainly do better! Enter the parameterized log message which is quite similar to a parameterized SQL query in spirit. In Log4j and many other logging systems, parameters are specified by {} placeholders in the log message and provided as additional parameters to the logging method. For example:

logger.debug("User {} logged in", user.getName());

The placeholder is filled in only if debug logging is enabled, so the full string is never computed unless absolutely necessary. This technique is mostly relevant to languages like Java. In the Scala version of Log4j, for example, string templates are a built in feature to the language, and macros are used behind the scenes to avoid the template rendering when logging is disabled. Example:

logger.debug(s"User ${user.getName} logged in")

One more related API that is handy to know is that a lambda function can be provided instead of a string in order to defer some code needed to assemble a log message only when enabled. For example, suppose we wish to go fetch some additional metadata from a database for some debug log message. This overhead might be unacceptable most of the time, but we may wish to selectively enable it once in a while. The entire body of the function can be encapsulated into a lambda function and passed to the logger. This is generally cleaner than surrounding the code with if checks for the relevant log level or other noisy techniques.

There are far more features that can be covered regarding how to use a logging API from the developer’s point of view, but these are mostly convenience features regarding repetitive things like thread-local information always included in a log message, or structured log messages, generic event logging, and others. Far more information about these features are available in the Log4j manual.

Where Do Log Events Go?

Now that we’ve established a general framework for writing and filtering log events, what can we do with them? The simplest implementation of handling log events would be to print each log message to the console separated by new lines. Since quite a bit of context would be lost doing it this way, we generally include additional information from the log event such as the timestamp, log level, marker (if defined), logger name, and thread name for multithreaded programs. All the fields we wish to output should be configurable, and in fact, there are several different fields available which we can add to provide context about the log message. The output format could also use a structured format such as JSON which is more easily parsed than line-oriented log messages, though all log aggregation and search tools have powerful tools to extract log event information from all sorts of formats.

In some use cases, writing log events to stderr is acceptable. For example, during development of a program, the developer may wish to view log events while running the program in the console. On the other hand, perhaps you’re using an orchestration framework such as Apache Mesos to execute all your applications. Such a framework can be configured to watch the stderr streams of all running applications in order to collect log messages to a central location.

For many use cases, however, simply printing to a console that nobody looks at is not a valid strategy. Any website that requires a reasonable SLA and has more than, say, a few hundred users, generally requires multiple servers to distribute load. As the number of nodes increase, it simply becomes infeasible to watch the console. In fact, each node may be executing multiple applications, so without a program like tmux, we’d have to redirect the stderr of each program to a file anyways. With that in mind, we can directly configure the logging framework to output log events to a file instead of stderr. Each file can be monitored using a program such as tail to continually watch for new log events being appended to the file. This style of logging is pervasive in typical GNU/Linux and BSD systems where many running services will output log information to /var/log/ directories. However, if this is not configured to periodically rotate log files and delete old ones, then the server’s disk space can eventually fill up with log information! This job is typically filled by a program such as logrotate, though Log4j has a rolling file appender which provides similar functionality.

Simply outputting to a log file can be a good strategy for operators who are still stuck in the “do it by hand” mindset, but we can do better! Our main goal here should be to collect logs from all our servers into a central, searchable location. One such way to accomplish this is by using a product such as ELK, Fluentd, or Graylog. These tools offer more than just log aggregation; they offer ways to filter, sort, search, and alert based on the contents of the logs. However, by relying on log files, we’re also relying on the stability of the individual servers. Obtaining logs in disaster scenarios is generally more difficult but also far more important, so let’s improve on that.

Apache Flume is a project for collecting and aggregating large volumes of log events in a distributed computing environment. This is very useful in cluster scenarios such as running dozens or hundreds of Apache Hadoop or Apache Spark nodes for example. Individual nodes can pass along log events to a Flume agent, and each agent is responsible for reliably delivering the log events elsewhere. Combined with the Flume appender, this can be easily utilized in a distributed environment to collect all log events to a central log aggregator. Said aggregator may be something complex like Logstash or Graylog, or perhaps it may be something simple like a single master log file.

Now that we have our logs all in one place, we can really step up the operations game. We can set up alerts based on log level thresholds, number of messages, frequency of messages, and triggers based on any metadata contained within. If we want to get really fancy, we can train some machine learning models via Spark or Apache Mahout combined with any other exported metrics data to attempt to predict failure of our services. Such a technique could also be used for all sorts of observability of clusters and microservices. Combined with scripts to automatically scale or restart services, operations can become more proactive in maintaining their systems.

There are dozens more frameworks, libraries, and tools that could be covered here. Logging is something all developers do whether they’re using the proper tools or not, so it’s a great idea to get familiar with the tools and concepts in order to improve the metadata being created by applications. Developers should work closely with operators (devops) in order to find a good balance of logging verbosity and observability. Managing logs is a complex topic that many people tend to overlook, but having a good logging architecture in place can help save the day during a production issue. As a final note, to those using the Java Platform, Apache Log4j 2 is the premier logging library for Java, Scala, Groovy, Kotlin, and any other JVM language. It is common for logging to add noticeable overhead to applications, and the typical solution is to simply disable logging, but this removes all the advantages to logging in the first place! Instead, take a look at the numbers and see how Log4j can be used with very minimal overhead, even in high frequency trading applications.

Using Akka and Scala for CQRS and Event Sourcing

Mon, 03 Jul 2017 09:00:00 -0500

In the past, we covered using Lagom to implement Java microservices using the event sourcing and CQRS patterns that framework relies on. Today, we’ll be revisiting our blog microservice example using the Scala programming language and Akka, one of the main components underlying Lagom.

Event Sourcing

First, let’s take a quick review of what event sourcing and CQRS are along with the general design patterns to use to implement these ideas. Event sourcing is an architectural pattern where all activity in a system is captured by immutable event objects, and these events are stored and published in a sequential log. The state of a system at a given time can therefore be derived from the sequence of events prior to the state. For example, most SQL databases are implemented using this sort of architecture where all data mutation operations are stored as events in a write-ahead log. This pattern works very well with the physical storage underlying these devices as they tend to be based on magnetic disks which can work very fast in sequential access but become a lot slower in random access.

Using the concept of event sourcing, Command Query Responsibility Segregation is an architectural pattern that is rather self-describing: the responsibility of a system to handle commands and queries are separated rather than intertwined as in typical applications. Commands are used to update state while queries are used to inspect state. The separation of these two concerns allows for much greater scalability in distributed systems than traditional approaches such as a single backing database for both reading and writing data. While CQRS is not required in order to implement a write-side and read-side pair of databases, it does offer a strong pattern to follow to do so. For example, an application like Twitter may wish to store all tweets in a distributed database such as Cassandra, but in order to effectively search those tweets, it would be useful to copy written data into Elasticsearch for later querying. Provided that we follow an event sourcing pattern, then the source streams of data are well defined and make it much simpler to implement this responsibility segregation.

In CQRS, we can issue a command to a system to update its state. This command, after validation, will create an immutable event which is appended to the event log. After persisting this event, the state of the system can be updated in response. By doing this, the system can always be brought back to any given state by replaying the events in the event log. For performance reasons, the state of a system is periodically snapshotted and persisted so that less events from the main event log need to be replayed when restarting the system. The state obtained from the event log can be as simple as the denormalized representation of entities, or it could be more complex views of the event stream which can be further combined down the line. It’s important to note that following this pattern means we need to manually reimplement some things we take for granted in a normal RDBMS such as transactions, indexing, joins, constraints, and triggers. Ideally, these sorts of things can be abstracted well enough to be provided mostly through libraries or frameworks, but some things such as transactions are still application-specific in how to make compensating transactions to roll back erroneous data.

Actor Model and Akka

The actor model is essentially a model of distributed systems where processes are modeled as “actors”, things that have an inbox for receiving messages that are processed asynchronously. Actors can only communicate with other actors via messages, and this allows actors to be scaled outward into fully distributed systems. A simple way to think of an actor is as a thread that cannot directly access the state of any other thread and must pass immutable message objects to other threads to coordinate anything. This avoids the use of synchronization, locks, and all the extremely difficult to debug concurrency issues that plague typical concurrent code.

Actors are lightweight and can be spawned and restarted many times with very little overhead (far less than an actual thread). Thus, following the ideals of programming languages like Erlang, actors are small enough to allow for a more robust form of error handling known as “let it crash”. Forming a suitable hierarchy of actors, parent actors can automatically restart child actors when errors occur or perform other fallback strategies. This is particularly useful for long-running systems where either a full restart is infeasible or the cost of resources to double up the system for blue green deployments is prohibitive. It may also be the case that recovering the full state of the system would take too long, so being able to selectively restart only single components without affecting the rest of the system is very useful.

Akka is an implementation of this actor model inspired by Erlang. Akka is written in Scala and provides both a Scala and Java API for actors, clustering, persistence, distributed data, reactive streams, HTTP, and integration with various external systems.

All code samples in this post are from my GitHub repository. We’ll be using Scala 2.12 which requires Java 8. Note that if you’re still stuck on Java 6 or 7, Scala 2.10 and 2.11 both only require Java 6. Code samples may be adaptable to previous releases of Scala with little or no modifications. Anyways, let’s jump in to using Scala!

Blog API

Our API to implement today will be the same as in my Lagom tutorial. As such, we’ll have a blog data type consisting of three fields: title, author, and body. Blog posts are identified by UUIDs. The REST API will consist of a GET, POST, and PUT endpoint for looking up, creating, and updating blog posts respectively. The first thing to implement, then will be our blog data type.

final case class PostContent(title: String, author: String, body: String)

You may note that there isn’t much to it, and you’re right! Let’s review the syntax here. First of all, we don’t need to define things as public in Scala as they’re all assumed to be public by default. In fact, public is not even a keyword in Scala. After a class name comes its parameters which can be thought of as the parameters to its constructor. A class can still have multiple constructors by defining methods named this within the class, but we have no need for that here. Parameters and variables are declared in the opposite order as Java; in Java, we would say String foo, whereas in Scala we would say foo: String. The other keywords used here are final which makes the class final as in Java, and case which makes this a “case class”.

Recall that in Java, we can use Lombok to automatically create an all-args constructor, getters for all fields, equals, toString, and hashCode, builders, withers, and other boilerplate code. In Scala, we can simply add case to a class to get a lot of that for free. A case class makes all its class parameters (fields) available with something similar to getters, adds sensible equals, hashCode, and toString method implementations, and handles some other Scala-specific features we’ll cover later. Essentially, a case class can be thought of as a data value class, and we’ll be using it as such.

Next, we’ll define a post id class instead of directly using UUID.

class PostId(val id: UUID) extends AnyVal {
  override def toString: String = id.toString
}

object PostId {
  def apply(): PostId = new PostId(UUID.randomUUID())

  def apply(id: UUID): PostId = new PostId(id)

  implicit val postIdDecoder: Decoder[PostId] =
    Decoder.decodeUUID.map(PostId(_))
  implicit val postIdEncoder: Encoder[PostId] =
    Encoder.encodeUUID.contramap(_.id)
}

We have several new keywords to discuss here. First, by adding val to the id class parameter, this will add a Scala-style getter for id by providing a way to access the id field directly. Classes use the extends keyword to extend other classes or implement traits (interfaces). This AnyVal class is a special class in Scala that works essentially as a boxed value type. This value class can only contain a single public field, and at compile time, the compiler will attempt to remove all indirect access of that field and replace it with direct access. Thus, we can create a wrapper type for UUID without sacrificing performance at runtime.

The override keyword is used just like the @Override annotation in Java: while not required, if the keyword is present but the method doesn’t actually override anything, this will cause a compiler error. This can be useful for catching typos or unknown API changes. The def keyword is used for defining methods. When a method has zero arguments, the parenthesis are optional. The conventional style here is that parentheses are omitted in pure functions while they remain in side-effecting functions. As with variables, the types of parameters and method return values come afterwards.

The object keyword here defines what is essentially a singleton instance of a class of the same name. Scala does not have static, but these objects provide an equivalent feature. When an object is named the same as a class, it is called the class’s “companion object”. A class and its companion object have equal access to internals of each other similar to how static and non-static members work in Java.

The apply methods here are used as factory methods to create PostId instances. The apply method has special meaning in Scala: we can omit the name of the method when calling it. For example, PostId.apply() is the same as calling just PostId(). As can be seen here, the return keyword is optional when the last line of executing code of a method is the return value.

The implicit keyword is used for a lot of things in Scala, and in this case, an implicit variable is one that is available for implicit parameters to methods that take them. We’ll be using it here to fill in custom encoder and decoder implementations for our PostId class to ensure that the JSON marshaller and unmarshaller do not think that this is a complex object.

The square bracket syntax, Decoder[PostId], is Scala’s generic type parameter syntax. The equivalent in Java would be Decoder<PostId>. Finally, these codecs are derived from existing UUID codecs, so we map and contramap them using some lambda functions. Both of these lambda functions use anonymous parameter syntax, both of which can be expanded to:

Decoder.decodeUUID.map(id => PostId(id))
Encoder.encodeUUID.contramap(postId => postId.id)

Next, let’s stub out a service interface. We’ll fill in the details later.

trait BlogService {
  def getPost(id: PostId)
  def addPost(content: PostContent)
  def updatePost(id: PostId, content: PostContent)
}

The trait keyword is very similar to an interface in Java. In fact, as written, this trait will compile directly into an interface. Traits are far more powerful than Java interfaces, so this isn’t always possible, but Java has started making interfaces more powerful starting with default methods in Java 8, so these two concepts may converge one day. The main difference between a trait and a class in Scala is that a trait cannot have its own parameters, but a class can extend (or mix in) multiple traits. So far, this is still very similar to Java. However, traits can contain fields, private and protected members, default implementations, and even constraints on what the concrete implementing class must conform to. We’ll omit the return types of the methods for now, but we’ll return to add them later once we’ve defined them.

Blog Entity

Next, let’s dive down to the entity level. Recall that in Lagom, a persistence entity is associated with three related things: commands, events, and state. Following a similar pattern here, we’ll implement a BlogEntity class by first defining our commands, events, and state classes.

object BlogEntity {

  sealed trait BlogCommand

  final case class GetPost(id: PostId)
    extends BlogCommand

  final case class AddPost(content: PostContent)
    extends BlogCommand

  final case class UpdatePost(id: PostId, content: PostContent)
    extends BlogCommand

  sealed trait BlogEvent {
    val id: PostId
    val content: PostContent
  }

  final case class PostAdded(id: PostId, content: PostContent)
    extends BlogEvent

  final case class PostUpdated(id: PostId, content: PostContent)
    extends BlogEvent

  final case class PostNotFound(id: PostId)
    extends RuntimeException(s"Blog post not found with id $id")

  type MaybePost[+A] = Either[PostNotFound, A]

  final case class BlogState(posts: Map[PostId, PostContent]) {
    def apply(id: PostId): MaybePost[PostContent] =
      posts.get(id).toRight(PostNotFound(id))

    def +(event: BlogEvent): BlogState =
      BlogState(posts.updated(event.id, event.content))
  }

  object BlogState {
    def apply(): BlogState =
      BlogState(Map.empty)
  }

}

The first new thing of note here is the sealed keyword. A trait or class marked sealed indicates that all subclasses of that trait or class must be contained in the same file. This makes pattern matching on these types of classes easier as it aids the compiler in detecting missing patterns checked by the programmer and can help prevent certain classes of bugs.

We have three types of commands: GetPost, AddPost, and UpdatePost. These are all rather trivial and mirror the same commands from the Lagom version of this microservice.

Next, we defined a sealed trait BlogEvent which contains two public values. The val keyword is similar to marking a variable as final in Java, while the var keyword is similar to a normal variable in Java. When defining a val or var on a class, this is technically creating a field in the class along with a getter-style method (named the same as the field) and a setter-style method if using var. What this effectively does is allows the variable to be accessed as if it were a public field of the class. Note that by not giving a val or var an initial value, this makes them abstract. It may be worth noting here that semicolons are optional in Scala and tend to be omitted; otherwise, this would have been our first usage of them.

For our events, we only care about commands, not queries (in the CQRS sense of the words, not a BlogCommand type of command), so we defined two events: PostAdded and PostUpdated.

Next, we abuse the case class feature to implement our own exception class, PostNotFound, to obtain some handy features for free. This demonstrates the syntax to call the constructor of a superclass as well. There is one other nifty feature in use here: interpolated strings. Scala provides an extensible feature to make interpolated strings which are expanded out and filled in at compile time. The syntax s"Hello, $foo!" would be essentially the same as "Hello, " + foo + "!" at compile time. We could use more advanced expressions inside the string by wrapping the variable name in curly braces, so for instance, we could say s"Hello, ${foo.capitalize}!".

After that, we define a type alias named MaybePost[+A]. There are a few things going on in this, so let’s break it down. First of all, Scala allows us to define type aliases which can be used to alias a larger type into something more readable, or it can even be used simply to rename types. In the generic type parameter, the +A bit indicates that, if B is a subclass of A, then MaybePost[B] is a subclass of MaybePost[A]. This is called a covariant type parameter. If we omitted the plus, we’d have an invariant type parameter which means that regardless of how A and B are related, Foo[A] and Foo[B] would not be related. There is a third type of variance available called contravariant type parameters which use -A and imply the opposite subclass relationship between Foo[A] and Foo[B] from A and B. We alias this MaybePost from the Either[A, B] type from the Scala standard library which is a type that can be either a Left or Right value containing a type A or B respectively. This class is normally used as a more powerful form of Option (Optional in Java) where the left side type is an exception type and the right side type is the success value type.

Finally, we come to the BlogState class and companion object. We have some strange names chosen here for methods: apply and +. Wait, +? Yup! In Scala, you can name a method pretty much anything you like. Scala will translate the names into valid Java identifiers at compile time. As explained above, the apply method is treated specially by allowing the word apply to be omitted when calling the method. There are several other special method names that allow for syntax features in Scala such as update, unapply, map, flatMap, filter, withFilter, foreach, and methods named after mathematical operators such as +, -, and *. We will not be covering most of them, but it’s worth noting their names. One other thing to note here is that the default Map[K, V] type used here is an immutable hash map, and in keeping with the spirit of immutability, we will be updating our BlogState by returning a new BlogState with a new Map which contains the old map plus an additional item. Thus, the use of posts.updated(key, value) returns a new map with the addition or modification of the provided key value mapping.

With all that out of the way, let’s move on to the entity implementation.

class BlogEntity extends PersistentActor {

  import BlogEntity._
  import context._

  private var state = BlogState()

  override def persistenceId: String = "blog"

  override def receiveCommand: Receive = {
    case GetPost(id) =>
      sender() ! state(id)
    case AddPost(content) =>
      handleEvent(PostAdded(PostId(), content)) pipeTo sender()
      ()
    case UpdatePost(id, content) =>
      state(id) match {
        case response @ Left(_) => sender() ! response
        case Right(_) =>
          handleEvent(PostUpdated(id, content)) pipeTo sender()
          ()
      }
  }

  private def handleEvent[E <: BlogEvent](e: => E): Future[E] = {
    val p = Promise[E]
    persist(e) { event =>
      p.success(event)
      state += event
      system.eventStream.publish(event)
      if (lastSequenceNr != 0 && lastSequenceNr % 1000 == 0)
        saveSnapshot(state)
    }
    p.future
  }

  override def receiveRecover: Receive = {
    case event: BlogEvent =>
      state += event
    case SnapshotOffer(_, snapshot: BlogState) =>
      state = snapshot
  }

}

Note that we need to extend PersistentActor to use akka-persistence, the mechanism we’ll be using to save blog events to disk. This class is dense with syntax we haven’t covered yet, so let’s go over it all. First of all, note how we imported all the members of the BlogEntity object. The _ in the import is equivalent to using * in Java (Scala doesn’t use * here because * is a valid class and method name in Scala). Note that we don’t ever use an import static like in Java as Scala does not have static things (though it can import static things from Java without having to specify it’s a static import). Since BlogEntity is already in scope, we did not have to specify the full package name preceding it in the import. We also import the members of the context variable which is defined in a superclass. We do this to gain access to some implicitly available variables used in the asynchronous code below.

Next, note that we were able to construct a BlogState object by omitting the keyword new because we defined a method named apply. Thus, we are actually calling BlogState.apply() here. In the next line, we override an abstract method to identify this aggregate root for persistence purposes. An alternative name here may be the fully qualified class name.

Another neat syntax to note here is that in Scala, a zero-arg method can omit its parenthesis. This feature allows for a lot of neat things such as allowing a val to override an arg-less def in the parent class. Our next method, receiveCommand, returns an Akka type called Receive which is a type alias for a lambda function that takes a single parameter of type Any and returns nothing. This warrants a quick overview of some of the standard Scala types. The Any class is the root class, and from there are AnyVal and AnyRef. The AnyVal class is for types like Int, Boolean, Double, etc., along with user-defined value classes as explained in the PostId description, while AnyRef is equivalent to Object in Java. There is also the Unit class which is equivalent to void in Java. The difference here is that technically, all methods must return a value in Scala, so for an empty return type, Unit can be used which has only one possible value: (). There are also two bottom-most types in Scala: Nothing and Null. The Nothing type is generally used as a return value for a method that does not actually return (e.g., it always throws an exception). The Null type is the type of the null reference which is a subtype of everything. Try not to confuse these with the None type of an empty Option or the Nil instance of an empty List.

The receive method of an actor tends to be implemented as a pattern match expression. A pattern match takes the form of foo match { case a => ...; case b => ...; ... }, and this is a very powerful feature used pervasively in Scala. In our use case here, we can omit the foo match part to create a lambda function that matches an anonymous value. In our cases, we’ll be looking for messages that match our command types. Since our commands are all case classes, Scala has generated an unapply method for each which makes the classes destructurable so to say within pattern match expressions. For example, the expression case GetPost(id) => will match when the matched object is an instance of GetPost, and it will subsequently bind its one field to the new variable named id. We can use nested patterns here if we wanted, but that is a more advanced feature.

In the GetPost example, we already have a handy BlogState.apply(id) method available as a lookup to find the content for a given id. This returns our MaybePost[PostContent] type which can be used to determine whether or not we got the content. Using the sender() method, we can send a message back to the actor that sent the initial message. The ! method used here is an alias for the tell method. Note that we omitted the dot and parenthesis here by using the infix notatation syntax of calling methods. This syntax allows, for example, the method call 1.+(1) to be written as 1 + 1. This is particularly useful for methods that have an operator syntax like +, but it can also be useful in functional programming contexts as well. Thus, our line of code is equivalent to sender().!(state.apply(id)) when expanded out.

Next, to handle the AddPost command, we create a PostAdded event with a new id and its content. The result of handling the event is a Future[PostAdded], so we use the pipeTo pattern in Akka to send the result of that future back to the sender. As mentioned for the infix method call syntax, this line can be equivalently written as handleEvent(PostAdded(PostId(), content)).pipeTo(sender()). We’ll be reusing the event handling logic in UpdatePost, so we’ll come back to that. The next line contains a Unit to return explicitly. We do this because we are using a set of strict compiler flags which would make discarding a return value an error otherwise. Note that this is not required in the default Scala compiler settings, but we’re trying to stick to high quality Scala code.

The UpdatePost command requires a bit more work than AddPost did. This is to validate that the id provided already exists. Thus, first we look up in our state for an existing post. If it does not exist, we’ll get a Left(error) value with some exception error; if it does exist, we’ll get a Right(content) value with content being the PostContent value. Thus, we combine this with a pattern match to check for both types. The syntax, case response @ Left(_), uses two pattern types concurrently: binding the matched expression to the new variable named response, and matching that it is a Left type with any content (the _ is a wildcard match in this context as a throwaway variable). If the post doesn’t exist, we send back the error. Otherwise, we create a PostUpdated event, handle it, and send it back.

Next, we look at handling the event. Let’s break down the new syntax. The [E <: BlogEvent] bit is a generic method type parameter, where <: is equivalent to extends in a Java generic method. Conversely, the >: generic syntax would be equivalent to super in Java. For example, this method may be written in Java as private <E extends BlogEvent> Future<E> handleEvent(E event). The other syntax of note here is the e: => E parameter. This is similar to a zero-arg lambda function, but when called as such, does not require being wrapped in a lambda. Had we used e: () => E as the parameter instead, then we would have had to call the method as handleEvent(() => PostAdded(...)) instead.

In order to create a Future here, we’re using the Promise class. A Scala Promise works rather similarly to promises in JavaScript and other languages. To handle the event, we call the persist method defined in the PersistentActor superclass. We use another new syntax here where a method with a single lambda function parameter can be replaced with curly braces. A lambda function can always be surrounded in curly braces, so this syntax is taking advantage of some infix method call syntax features. This syntax is rather similar to how Closures work in Groovy. The lambda provided to persist here is called after the event has been successfully persisted. In this, we complete our promise which is returned as a Future to the caller. After that, we call state += event. Since we never defined a method called += on BlogState, Scala sees that state is a var, so it can rewrite that expression into state = state + event. Since we do in fact have such a method available, the whole expression is equivalent to calling state = state.+(event). After that, we publish the event to the system event stream which can be used as an application-wide event bus. Finally, we add in a periodic call to saveSnapshot every 1000 events which will allow us to restart and recover the state a lot faster than rereading the entire event log.

The last bit of code we needed to implement in this actor was the receiveRecover method which is used to recover the latest snapshot on startup if available. We can populate this actor’s state directly from the snapshot, and then subsequently handle all the events that weren’t included in that snapshot. The only new syntax used here is the case event: BlogEvent pattern match syntax which matches when the object is an instance of BlogEvent and binds it to the new variable event.

Blog Service

Now that we’ve written our entity actor, we can fill in the stub BlogService trait defined earlier.

trait BlogService extends AkkaConfiguration {

  import BlogEntity._

  private val blogEntity = actorRefFactory.actorOf(Props[BlogEntity])

  def getPost(id: PostId): Future[MaybePost[PostContent]] =
    (blogEntity ? GetPost(id)).mapTo[MaybePost[PostContent]]

  def addPost(content: PostContent): Future[PostAdded] =
    (blogEntity ? AddPost(content)).mapTo[PostAdded]

  def updatePost(id: PostId, content: PostContent): Future[MaybePost[PostUpdated]] =
    (blogEntity ? UpdatePost(id, content)).mapTo[MaybePost[PostUpdated]]

}

The AkkaConfiguration trait we mix in here is a trait we defined for easy access to Akka-related objects such as an ActorRefFactory to create actors, a Materializer which is used to run Akka Streams, an ExecutionContext which is used for coordinating Futures and other asynchronous functionality, and a Timeout which is used for a default timeout value when making request/reply-style ask calls to actors.

As noted before, we can import all the members of the BlogEntity companion object here. It’s also worth noting that Scala allows you to import things to whatever scope you want, so we can limit our imports to the closest spot it is actually used.

To use our BlogEntity actor, we need to spawn it first. Using our ActorRefFactory, we can spawn a BlogEntity using the default Props of a BlogEntity. Spawning the actor will create it and start it asynchronously, returning an ActorRef instance which can be used to interact with the actor. Since actors can be restarted, replaced, or even located on completely different processes or machines, we never directly access an actor’s instance and instead use its ActorRef to send messages, watch it, etc. Since actors are written in a rather type-unsafe fashion, we’re wrapping the actor’s messaging API into a typesafe trait.

In order to receive responses from our actor when sending a message, we must do so asynchronously. Thus, our API returns futures. We utilize the ask pattern to send a message to the actor and wait for a response message. Since actors are not typed, the message response comes back as a Future[Any], so we use the mapTo[A] method on Future to verify it matches our expected type and cast it. Other than that, this trait is rather self-explanatory based on our BlogEntity.

Blog REST API

Next up is defining our REST API. We’ll be using the high level Akka HTTP route DSL for this. Using our BlogService trait, we’ll mix that in to another trait to define our API.

trait BlogRestApi extends RestApi with BlogService {
  override def route: Route =
    pathPrefix("api" / "blog") {
      (pathEndOrSingleSlash & post) {
        // POST /api/blog/
        entity(as[PostContent]) { content =>
          onSuccess(addPost(content)) { added =>
            complete((StatusCodes.Created, added))
          }
        }
      } ~
      pathPrefix(JavaUUID.map(PostId(_))) { id =>
        pathEndOrSingleSlash {
          get {
            // GET /api/blog/:id
            onSuccess(getPost(id)) {
              case Right(content) => complete((StatusCodes.OK, content))
              case Left(error) => complete((StatusCodes.NotFound, error))
            }
          } ~
          put {
            // PUT /api/blog/:id
            entity(as[PostContent]) { content =>
              onSuccess(updatePost(id, content)) {
                case Right(updated) => complete((StatusCodes.OK, updated))
                case Left(error) => complete((StatusCodes.NotFound, error))
              }
            }
          }
        }
      }
    }
}

There are a couple new syntax features here along with many DSL-specific things going on. First of all, this introduces the keyword with which is used when extending multiple traits. The RestApi trait is one we made that mixes in the Akka HTTP route DSL along with support for marshalling and unmarshalling our case classes and primitive types into JSON. The only other new syntax here is the tuple syntax. A tuple is an abstraction of an ordered pair. A tuple can have two or more values that do not have to be the same type. They are contained in parenthesis and separated by commas. Due to syntax ambiguity, when sending an inline tuple to a method, we need to double up on the parenthesis to avoid it being interpreted as a call to a method with multiple parameters. All other syntax in this trait are features of the route DSL. For example, URI paths can be matched using implicit conversions from strings into URI patterns, and those patterns can be composed with / which matches a slash in the URI. Segments of the path can be extracted into parameters in the lambda function provided after. We can easily unmarshal request bodies using the entity(as[A]) { a: A => ... } DSL. Using our service mixin, we can forward these requests and get response which can be chained back into the HTTP response. Finally, the ~ function is used to chain two routes together into a single route. Far more comprehensive information about the routing DSL can be found in the Akka documentation.

In order to run this code, we still need to create our HTTP server and set up Akka in general. That code is not very interesting in itself and is included in the code samples. While this post only scratches the surface of Scala, it provides a broad overview of various Scala-specific syntax features that really differentiate it from Java. There is one other feature that has only been mentioned by name so far, and that is implicits. Implicits are a powerful feature specific to Scala that helps reduce boilerplate typing in various scenarios. Implicits can generally be used for a few different things:

Implicit values can be used to provide parameters to functions or class constructors without being explicitly written as long as it’s in scope. This is useful for passing around parameters that are needed by tons of methods such as the ExecutionContext object mentioned above.
Implicit parameters can be used to automatically be filled in by an implicit value that is in scope so that the parameter doesn’t have to be typed out over and over again.
Implicit methods can be used to convert from one type to another when the original type is not compatible where it is used. For example, this feature is used to automatically convert an Int to a Long when passed to a method parameter that takes a Long. This can be a very dangerous feature when misused.
Implicit classes can be used to provide extension methods to an existing API. This is used to add methods to java.lang.String, for example, which cannot normally be done as String is a final class that cannot be extended. Extension methods provide a compositional way to extend APIs in a type safe fashion without having to explicitly wrap the API everywhere an extension method is desired.
Implicits can be used to provide type classes to Scala, a feature common to functional programming languages. Type classes are used very seldomly in object oriented programming languages. An example of a simple type class in Java is the Comparable<T> or Enum<E> interface.

Overall, Scala is a fantastic programming language with a simple core set of features and tons of extensibility. Staying true to the “scalable language” concept behind its name, Scala works well in small scripts all the way up to large distributed applications. The Akka framework provides the building blocks for writing distributed systems using event sourcing and CQRS, while Scala itself enables much more concise, readable DSLs and removal of verbose boilerplate common to many Java frameworks. Akka is far more flexible than its opinionated cousin, Lagom, and it works well with various technologies common to event sourced systems like Kafka and Cassandra.

Why Use a Microservice Architecture

Thu, 20 Apr 2017 09:00:00 -0500

Microservice architecture is the latest fad in software development, and as such it comes with numerous conflicting definitions. To help clear up this confusion, we’ll discuss what microservices are, how they relate to older development architecture patterns, and why and when they are useful. In the enterprise world, service oriented architecture was the established paradigm in software development, and this typically incorporated the SOAP standard and web services. Some concepts from SOA, such as loose coupling and domain-driven design, translate well into microservices, but other concepts such as global transactions and global data consistency do not scale well in such an architecture. One major pain point in SOA that microservices helps to address are monolithic applications and the pains behind deploying new versions of them. Another problem stems from incompatible implementations of SOAP being widely used which tend to nullify the benefits of using a web services standard for loose coupling. Expensive proprietary software is also a large downside to typical SOA style development which has not yet reappeared to a similar extent in microservices. Data scalability and schema evolution are another hard issue in monolithic architectures which tend to be more manageable in a microservices architecture.

Due in part to the existing complexity of SOA usage, many developers eventually found themselves trapped by applications that could not be iterated on rapidly in an agile development environment. Deployments required complicated build systems easily described as “magic”, and entire teams would need to be formed just to ensure a single deployment worked properly. This sort of process worked well enough in the waterfall development days (or at least as well as waterfall development worked itself), but as the software industry migrated to agile development methodologies, the friction of maintenance, deployment, and onboarding new developers into overly complicated monoliths became untenable.

At this point, some developers began experimenting with what would later be named microservices. Standardizing on the ubiquitous web standards, individual services were written to communicate over HTTP using XML and later JSON, relying on extensive library support for these standards. This borrowed the concept of RESTful APIs, where services provided resources which were identifiable by URIs, and each resource defined HTTP methods (or verbs) that could be performed on those resources. There are other ideas on how to implement microservices besides via HTTP (e.g., OSGi in the Java world provides their own style of microservices which can communicate in-memory, but are still physically separated by the JVM via separate ClassLoaders), but the general concept behind them all is that each service requires physical separation and must communicate over well-defined boundaries such as through a network socket. This provides several advantages over traditional monoliths, many of which we will discuss.

It’s important to note before we go into detail that in general, microservice architecture provides far more benefits the larger an organization becomes. A small enterprise with only a single team or two working on these services may find it overly complicated to break their systems into microservices, particularly with purely CRUD-style applications. Many of the advantages discussed here can also be extended into small monolithic applications with proper developer discipline towards modularity, loose coupling of functionality, and high cohesion of individual units of code as organized into modules.

First of all, breaking up a monolith into smaller pieces simplifies the individual build and deployment systems of each microservice. While the overall complexity of the system may increase, the savings in developer time and pain is well worth the cost. Each individual microservice is much simpler to understand in isolation, and in theory, this decreases the amount of time it takes to onboard new developers into a system. This does generally require the adoption of devops into an organization, but this is generally considered a good thing to adopt regardless.

By breaking into microservices, another advantage is provided in that each service may have different technologies that are more appropriate to use depending on the domain in question. For example, simple CRUD microservices may be more rapidly developed using a language like Kotlin, Ruby, or even JavaScript, while more complicated asynchronous message-oriented services may be more appropriately developed in Scala, Elixir, or another functional programming language. Choosing appropriate frameworks for each service is also an important advantage as developers no longer have to rely on overly broad enterprise style frameworks. This allows more room for experimentation with newer technology and design patterns which can then be shared with other developers to help improve the overall system.

Choosing appropriate programming languages, libraries, and frameworks are only one part of the many choices required for each microservice. Since most applications contain some sort of persistent state, most applications require some sort of database or general persistent storage area. When separating an application into microservices, each microservice can and should have ownership over the data it needs while providing access to it through its own public APIs. A common problem in many actual implementations of microservices is sharing databases between services. Not only does this tightly couple those applications in regards to scaling, schema management, and other administrative tasks, but it also destroys a lot of the advantages of using a microservice architecture. Though in practice, a single database cluster can be used to serve multiple microservices, as long as each microservice uses its own database within that cluster, the choice to physically separate that database is much easier than when multiple applications share a database, even if they use their own tables.

The choice of database, cache, and other infrastructural addons to applications should be left to the individual microservices, though in order to make the operations sides of things at least somewhat practical, it is generally a good idea to limit the number of different choices to use. For example, an organization may wish to standardize on a single type of relational database, a single NoSQL database, a single time-series database, a single distributed cache, etc. Of course, this should remain more of a guideline rather than a hard rule in order to prevent wasted effort fighting the tools rather than using what works best in each use case.

An important consequence of the physical separation between microservices is that it makes it more difficult to create minimally cohesive systems. No longer can code be simply dumped wherever the developer feels like it. Instead, reusable libraries can be promoted where appropriate, and business logic remains exactly where it belongs: within its appropriate domain. This can also provide a prime opportunity for an organization to contribute back to the greater community via free and open source software with said common library components (incidentally, this is how many Apache projects are started). Developers must make a conscious decision to make remote calls into other microservices, so this helps enforce loose coupling. Even when developers attempt to tightly couple microservices, this problem becomes apparent much sooner rather than at the most inopportune time when a service hits a spike in usage one day.

This leads into another advantage of microservices: the ability to independently build, deploy, and scale these services. As alluded to before, build scripts can be immensely simplified just due to the decrease in the amount of code needed to be packaged together into a single distributable artifact. Deployments can be simplified out of necessity of supporting deployment of numerous microservices, a goal that is often ignored when deploying monoliths. As for scalability, each microservice can be independently copied and deployed multiple times across a cluster of servers. In order for microservices to communicate in this scenario, a form of client or server load balancing is required so that all instances of each microservice can be evenly used. This relates to the elasticity aspect of reactive architecture where services can scale up and down to meet the physical requirements of that service at any given time. In general, scaling horizontally like this is simpler than vertical scaling as it doesn’t require specialized or expensive hardware.

There are disadvantages to using a microservice architecture, however. Deployments are a double-edged sword in that they may be simpler on their own, but in the real world, this doesn’t negate the need for communication with other teams. Due to limitations in creating RESTful services, maintaining backwards compatibility can become difficult. Since REST APIs aren’t exactly statically typed, programmatic refactoring between services is not very possible at this time. If a backwards incompatible change is required in one service, then its dependent services may also need to make a deployment at the same time with updates, and this can slowly lead back into the problems of coordinating monolithic deployments. Renewed research into RPC-style protocols is working on solving these types of problems, but there is no silver bullet. Following semantic versioning strictly can help smooth migrations between incompatible versions of APIs, but this comes with its own set of limitations such as requiring to support multiple versions of an API while migrations take place as well as challenges in migrating or synchronizing different versions of data stores behind the APIs.

A common pitfall when breaking a monolith into microservices is splitting them up too finely. This can be compared to a similar problem of dropping the use of relational databases purely in exchange for using a NoSQL style database. While certain domains may find this pattern lends well to use, other domains are relational in nature and can’t be physically decoupled without introducing numerous performance problems. If this happens, though, not all is lost! Using a distributed trace logging library such as Zipkin can help identify tightly coupled microservices and performance issues in context of the physical flow of data in relation to particular requests or events. After identifying these parts of a system, they might be rejoined, or the overall architecture of how data relate to each other can be revisited.

Another difficult issue in microservice architectures is that the divide between frontend and backend needs can become ever greater. While the backend attempts to remain logically pure, frontend needs begin to clash with UI requirements that join data from numerous microservices into a single page. Supporting such UIs is generally much simpler in monolithic applications, but the microservice world is quickly researching and experimenting with alternative approaches to exposing APIs to applications.

An approach that stays pure to RESTful APIs is to layer APIs similar to how code may be organized in a traditional SOA application. The base layer of APIs is made up of the individual microservices in a system. Another layer of APIs can be created to compose those microservices into more specific aggregate domains. A third layer of APIs can then be added on to expose a more stable, UI-centric API that in essence translates all the underlying domains into synthetic ones that are relevant to the user experience. This approach is generally more complicated, of course, but composite API layers, a common design pattern from enterprise application integration, can be easily implemented using integration libraries. This also allows for more intelligent caching and prefetching of data based on common usage patterns.

Another approach is to use an alternative method to interacting with APIs such as via GraphQL. Instead of exposing RESTful APIs directly to the frontend, a GraphQL endpoint is provided which can be used to specify the data structures required for a page along with filters on individual fields of the data, and individual fields can be updated via the same mechanism. While this is not RESTful in the slightest, it does tend to work better in practice when defining an ever-changing API that both frontend and backend developers can agree on. This also makes it much simpler to refactor individual microservices without disrupting the frontend as the GraphQL server itself performs the data translations into the requested structure. This also provides an interesting opportunity for caching and prefetching of data.

A fully asynchronous approach (which does not currently have a buzzword name that I’m aware of) may be to fully rely on using message queues and topics from frontend to backend. This idea was original inspired by a joke I made with old co-workers about implementing an “everything” API endpoint that would simply return all the user data we know about so that the frontend developers would stop requesting more and more combined REST APIs. To take this idea to its logical conclusion, the only realistic way I could think to implement it without an extremely long response latency would be to interpret the request as a message, and then, using websockets, individual pieces of the user’s data could be sent back as messages which would feed into a Redux-style state management library which would fill in relevant UI elements as it loaded. Existing messaging protocols such as STOMP may be useful in implementing this idea. While this approach of preloading all of a user’s data is not exactly realistic, the underlying idea of passing messages between the frontend and backend is interesting and should be considered as another alternative to traditional synchronous request/response APIs.

The patterns behind microservice architectures generally parallel the wider internet. Instead of attempting to abstract away distributed systems, developers should be embracing them. A monolithic architecture attempts to group everything together under the illusion that a simpler local programming model will work out, but in reality, these patterns do not scale well on the web, in big data, in the internet of things, in artifical intelligence, or really any non-trivial application of software engineering or computer science. As technology continues to improve, past assumptions that allowed monolithic software to thrive no longer work well for most applications. Embrace the horizontal nature of interconnected technology as this is still only the beginning as distributed systems of the future continue to spread out.

Security in a reactive microservices architecture

Mon, 26 Dec 2016 09:00:00 -0600

Applying the reactive manifesto to microservice architecture is a difficult problem to solve. One of the more difficult facets of this type of architecture is designing secure microservices. One common way to secure microservices is by using JSON web tokens to securely share authentication information between services and clients. This standard is great as the authentication information can be encoded directly in the token instead of needing to be queried from a central location on every service call. For example, upon logging in to an SSO page, the authentication service can create a JWT using a private RSA key to sign the token, and this JWT is returned to the client. The client can then use this JWT on subsequent calls, and the integrity of this token can be checked against the public RSA key of the authentication service. As a neat side effect, microservices do not need to be proxied by an authentication gateway as the authentication information is now self-contained and verifiable in a distributed fashion.

However, security is not only authentication (authn). Security also covers authorization (authz), the concept of what roles a user has along with what roles are required to perform a certain action. These user roles can be encoded into the JWT payload in addition to the authentication info. This way, each microservice can decode the provided JWT, verify it against the public key of the auth service, and perform local authorization checks using that information. Again, no centralized authorization proxy is required!

Notice, however, that this architectural design seems to rely heavily on stuffing everything to do with authentication and authorization into a single microservice. This means that all secured microservices need to share information with the auth service. Well, in a reactive system, we still want our microservices to work even when other microservices go down (resiliency). There is also the concern that a typical auth service tends to become a catch-all service for any cross-cutting concerns, so in designing a microservices architecture, it’s important to prevent such a problem from evolving. To preemptively avoid this mess, it may be useful to separate the concerns of authentication and authorization entirely.

One important architectural principle to follow when creating reactive microservices is that each service should own its own data, and other services should obtain that data through its public API rather than through a shared data source. Such a design tends to create the opposite of relational data normalization as much data tends to be duplicated across the entirety of your services. In order to make microservices responsive and resilient, this replication of data is completely alright. In that regard, every microservice contains resources and actions on those resources, and each of those resource/action pairs require a particular user role to be present in order to authorize a user to perform said action on said resource. With that in mind, it only seems natural that each microservice owns the authorization data for each authenticated user.

Following this idea to its logical conclusion, let’s design a reactive authorization architecture. Recall that a reactive system is responsive, resilient, elastic, and message driven. This architecture will have to fulfill all of these assets.

Message Driven Security

Beginning with messages, what would such a system look like? First, we’ll have an authentication microservice which owns a user login database. This service will also be responsible for generating OAuth tokens whether they be simple bearer tokens or JWTs. Upon successful authentication (e.g., via a web form), this login service will broadcast a login event message to all interested microservices. For a more concrete implementation, we could use Apache Kafka to broadcast this login event. This event would be encoded as a JWT, and the payload would contain an access token, a login username, an expiration timestamp, and the requested OAuth scopes that were authorized by the user. These scopes will later correspond to local authorization roles in each service.

This login message might not be entirely necessary, but it does allow each microservice to preemptively cache relevant user data that is expected to be queried soon after login. Using a login message would allow for larger payloads than allowed in an HTTP header which is particularly useful if a large payload is required. This also reduces message overhead by allowing JWTs to be accessed by access tokens. In general, the decision on whether to use JWTs directly or indirectly is similar to the decision in certain programming languages on whether to pass data by value or by reference to functions. Do keep in mind that network traffic is more expensive than in-memory traffic, so the more fine-grained roles you have, the better an idea it is to use JWTs indirectly.

Elastic Security

In this design, every microservice (including the login service) can be independently scaled in resource usage in response to changes in input rate to each service. The fact that we will not be relying on this service to proxy service requests makes this a lot simpler to scale. Since this service only contains a small amount of data for each user (along with some additional role/scope configuration), replication of data is also simpler to handle.

Responsive Security

As each service will be performing local authorization checks using data sent to it in each request, the responses generated by each service should not be affected by complicated authorization checks involving multiple services or proxies. In order to maintain low overhead in authorization checks, each service should contain a local cache of authorization information that can be recomputed from a hard data store on startup (e.g., by utilizing CQRS patterns). For example, we could use an event log to store the login message broadcast earlier, and using a read side processor to maintain an in memory table of unexpired authentication states. Lagom provides a great way to design a Java microservice around this pattern.

Resilient Security

This is perhaps the most interesting aspect of this design. The goal here is to stay responsive even in the face of failure. For example, if the login service crashes or is overloaded, we still want the rest of our services to continue working for authenticated users during the time it takes to fix, restart, or replicate the login service. To do this, the idea is to apply a sort of inversion of control over a centralized directory server. In a typical directory service such as LDAP, authorization checks are queried against this central server. While easier to configure and easier to code against, this provides a central point of failure. Instead, we’ll design our services to maintain their own local stores of authorization data. As each service knows what roles are available for its APIs, these roles can be broadcast in a sort of “role discovery” mechanism similar to service discovery. This feature can be used for an administrative service for configuring which users are allowed which roles. Said service is unneeded for continued operation of every other service, so we maintain service isolation.

Each service is now responsible for maintaining two aggregate roots related to security: user roles and login events. The login events can be uniquely identified by access tokens for a low overhead message header, or JWTs can be used directly. While this may sound like a bit of extra coding needed in every microservice, it can be abstracted into a security provider library used by every microservice. For example, a custom Spring AuthenticationProvider and Authentication class could be written to handle translations of these access tokens. The Spring Security OAuth2 library can be used to implement a lot of this functionality, though custom code would still be necessary to configure it to work in this distributed environment.

When a service calls another service, the access token (or JWT) is provided in a message header. For example, an HTTP REST call would use an HTTP header, while a generic message broker would use either a message header or a custom payload format that allows for encoding message headers. For example, the Message Security Layer framework can be used for securely encoding and decoding messages in such an architecture.

Is This Overkill?

This security architecture started as a shower thought on how to implement security in a reactive microservices architecture. As of the end of 2016, I have been unable to find any resources online detailing any sort of solution to this problem. However, it may be unlikely that most applications do not need this level of resiliency if enough resources are thrown at a secure API gateway proxy that all microservices are required to communicate with. It is also possible to allow microservices to trust each other completely and perform all necessary access control checks at the gateway level. However, any single microservice can theoretically be compromised, and if all it takes is anonymous access to one microservice, then that compromised microservice could be used to perform actions on any other microservice. On the other hand, proxying all inter-service communication creates a single point of failure, though some people may find that acceptable.

Implementing proper security is a difficult problem to solve, especially since the development process tends to follow the steps of: make it work, make it right, make it fast, and make it secure. Security is oftentimes an afterthought, and proper implementation of security also requires a carefully designed balance between security and usability. If a security architecture is too cumbersome to use as a user or developer, then users or developers will take shortcuts around the system or else lose an unacceptable amount of productivity. In that regard, it is important not to go overboard creating too many different authorization roles, and it is also important to make use of your security layer easy to use for developers. By localizing authorization to each service, this makes it easier to test in isolation as well while also maintaining some sort of security over time instead of waiting until integration testing to find out that something is insecure.