Nathan Kleyn

Scala by day — Rust, Ruby and Haskell by night. I hail from London.

Hi! I currently work for Intent HQ as Head of Engineering, writing functional Scala by day and Rust by night. I write for SitePoint about Ruby, and I’m on Twitter and on GitHub. You can get in touch with me at mail/at/

Homebrew >= v1.9.0 No Longer Links MacOS Provided Software

15 Jan 2019

I recently had a bit of fun after upgrading to Homebrew v1.9.0, so wanted to write a quick PSA for others who may hit the same issue.

Homebrew >= v1.9.0 has a breaking change to brew link --force:

brew link --force will not link software already provided by macOS.

That is, this change means that Homebrew will no longer allow brew link to override anything MacOS already ships with.

So for example, if you used Homebrew to install a recent version of Ruby, before v1.9.0 Homebrew’s version would’ve been in your path — now the MacOS system version will be in your path:

$ ruby -v
ruby 2.3.7p456 (2018-03-28 revision 63024) [universal.x86_64-darwin18]
$ /usr/local/opt/ruby/bin/ruby -v
ruby 2.6.0p0 (2018-12-25 revision 66547) [x86_64-darwin18]

If you try to brew link it, Homebrew will tell you why it’s refusing to, and give you a snippet of script for prepending it to your PATH in your chosen shell (in my case, fish):

 $ brew link --force ruby
Warning: Refusing to link macOS-provided software: ruby
If you need to have ruby first in your PATH run:
  echo 'set -g fish_user_paths "/usr/local/opt/ruby/bin" $fish_user_paths' >> ~/.config/fish/

For compilers to find ruby you may need to set:
  set -gx LDFLAGS "-L/usr/local/opt/ruby/lib"
  set -gx CPPFLAGS "-I/usr/local/opt/ruby/include"

For pkg-config to find ruby you may need to set:
  set -gx PKG_CONFIG_PATH "/usr/local/opt/ruby/lib/pkgconfig"

You can also see this information for any given package by running brew info <package>.

The upgrade to v1.9.0 does not give any warning about this — the change is effectively silent — so beware of scripts depending any binaries that clash with MacOS provided ones being in your default system PATH!

Top and Bottom

06 Sep 2018

As a follow-up to my previous post on Unit vs null, I thought it might be useful to talk about some of the other special types we have in statically typed languages.

There are actually two other special types in the type-system that you may be interested in: namely, the bottom and the top type.

The Bottom Type

The bottom type (written as Nothing in Scala) is a type that is impossible to create. It is commonly used as a placeholder when a type isn’t known to the compiler. See for example what happens when we don’t tell the compiler the types of the keys and values in an empty Map:

scala> Map.empty
res0: scala.collection.immutable.Map[Nothing,Nothing] = Map()

As it has no other information to go on, the types for both end up as Nothing.

It’s also useful for saying that a function never returns — for example, a function that exits the program before returning could be written as:

def doSomething(): Nothing = sys.exit(1)

It should be clear from the type that it is impossible for this function to produce a return value. In fact, it’s not even possible to write a function that has a real return value if you have Nothing as the return types:

scala> def doSomething(): Nothing = 123
<console>:11: error: type mismatch;
 found   : Int(123)
 required: Nothing
       def doSomething(): Nothing = 123

You therefore know that upon seeing a function returning Nothing, it will definitely never return if you call it — all guranteed because it’s impossible to make a value of type Nothing!

The Top Type

The opposite of the bottom type is the top type. Also called the universal type (and written in Scala as Any), this is simply a way to say “this could be any possible type”. For example:

def doSomething(): Any = 123

This function actually returns an Int but typing it Any works — what’s going on here? This is because Any is actually the ancestor of all types in Scala, even Object:

scala> val x: Object = new Object {}
x: Object = $anon$1@b428830

scala> x.isInstanceOf[Any]
res1: Boolean = true

You’ll often see Any used by the compiler when it is trying to fill in a missing type (so called “type inference”) but couldn’t find anything more specific to choose.

Why Do We Need These Types?

The reason both bottom and top types have to exist comes down to how static-typing works — compilers of statically typed languages are, fundamentally, trying to solve two problems:

The bottom type is a natural way to represent an impossible scenario during the first of these tasks. The bottom type is the default type if a type cannot be inferred — that is, when you don’t annotate a type, Nothing will be the end result if the compiler cannot find another more specific type after searching through everything else.

The top type is the default for constraint reduction. It gives a compiler a final ultimate result if nothing can be proven when doing constraint reduction. Any is what you end up with when two types constrain each other so much that they have no other ancestor in common, and works because every single type has Any at least in common.

Unit vs Null

05 Sep 2018

The other day, I was asked by somebody why we had Unit in Scala, and what was the difference over just using null. I thought this was a really interesting question — so here’s my response in a slightly longer form!

The Boolean Type

Let’s start from something we know well — the Boolean type! When writing code, we use these all the time. As you know, they can only have two possible values — true or false.

The boolean type is the smallest possible type with which we can represent information. It can show something was true, or false, on or off, yes or no, 1 or 0 — just a single bit is all it has to work with, but it’s enough 1.

Imagine the following example function:

def doSomething(bool: Boolean): Boolean = ???

We have a function that takes a Boolean and returns a Boolean. We know immediately some things about this function:

We know all of these things because the number of possible values for Boolean is limited. This limitation is what allows us to reason about what a function can do just from it’s type — and this is an incredibly useful tool that static-typing gives you.

Parameters and Return Values As Tuples

Remember that function we wrote before?

def doSomething(bool: Boolean): Boolean = ???

We can think about functions in a slightly different way: we can think about their parameters being passed as tuples and them returning tuples:

// Is the same thing as:
val doSomething: (Boolean) => (Boolean) = ???
//               ^ tuple      ^ tuple

This is how a compiler actually sees the functions you wrote — names given to parameters and syntax around showing the return type is just sugar that makes it read better to our human eyes. Really, behind the scenes, it’s more like passing tuples of arguments and receiving tuples of results back.

Keep this tuple thing in mind!

The Unit Type

Sometimes we do things where we don’t care about or can’t get a result — things that may have side-effects, for example. Often we need a way to say “this function returns nothing useful” — how can we do that without wasting space on a boolean if a boolean is the smallest piece of information we have to work with?

Well, it turns out there is a type for saying “I have nothing useful to return” — and that type is called Unit.

Unit is a type that only has a single value (hence the name). This single value is also called “unit” but to differentiate the value from the type, in some languages we write it as ().

Notice something? That looks like a tuple! And indeed, that’s not a mistake — Unit is just a way of saying “an empty tuple”!

Unit is useful because of an important property about the empty tuple — there is only one possible value of it, (). Where, for example, (Boolean) could become (true) or (false), Unit will always be () only. As a result of only having a single value, this means it is entirely useless for actually encoding any information — after all, what information can you impart if you’re only allowed to respond with one thing?

Being able to say “I have no information” is still a useful thing, and so we use Unit and () to do just that:

def doSomething(file: File): Unit = ???

Just from looking at the definition of this function, and as you were able to with the boolean example, you can tell this function must have some side-effect 2. We could even use this information to guess what this function does (perhaps it deletes the file, or maybe calls touch?)

Null: The Odd One Out

So now to the question at hand: what makes Unit useful over null?

The issue with null is that it is, basically, a hack! null is a special value that can take the place of any type, but it is not a type itself. This completely breaks our ability to reason using just types. Take for example this function:

def doSomethingpath: Path): File = ???

We might look at this and think “oh cool, maybe it’s opening a File for us?”. Well, if we didn’t have null you might possibly be right3 — but unfortunately it’s completely valid for this function to be implemented thusly:

def doSomethingpath: Path): File = null

What’s wrong about null is that it is a magic value not represented in the type-system, and therefore it limits a lot of our ability to use the types to their full power. This is why most idiomatic Scala eschews the use of null over things like Unit, Option[A], or Either[A, B] — these types tell the full story of what is going on, without hacks, and allow us the full power of deductive reasoning. null in Scala exists purely because of Java compatibility, and you would be wise to avoid it at all costs!

  1. Space is an important limitation in computers, and it is natural for us to seek out the smallest way to represent things — booleans are as small as it gets. It also fundamentally encodes the entirety of boolean logic, which enables us to write code that can make decisions based on conditional statements. 

  2. Assuming you aren’t making empty functions, of course. 

  3. Technically, exceptions are also another thing that isn’t represented in the type-system. Languages like Rust are taking this thinking to the mainstream and using an Either like type (Result) and Option completely instead of exception! 

Data-Only Images In Docker By Abusing COPY --from

02 Jan 2018

I recently employed a little trick to make Docker data-only images which can be joined into other Docker images as needed.

Imagine you have a set of configuration that you want to store somewhere separately to the code. Well, you can make a Docker image of just this configuration very easily:

FROM scratch
ADD ./ /fake-config

After building it with docker build -t test/fake-config ., the only thing inside this image will be the directory you added. This is the most barebones image it is possible to make outside of scratch itself 1.

Now, you may wonder what the use of an image like this is — after all, it can’t be executed or run because it has no binaries inside. Enter the COPY --from command:

FROM <some base image>
# ...
COPY --from=test/fake-config /fake-config /fake-config
# ...

With one command, the contents of your data-only image has been added to another image. This is super useful for adding configuration that you want shared between applications to the Docker images for them 2.

  1. There is actually some low-level plumbing within an image based on scratch

  2. This is especially true since you can’t use symlinks within a Docker context, making it difficult to otherwise do this without some script to wrap the call to docker build that copies the common stuff around — messy. 

Using Transient Lazy Val's To Avoid Spark Serialisation Issues

29 Dec 2017

Occasionally within a Spark application, you may encounter issues because some member of a (case) class / object is not serialisable. This manifests most often as an exception when the first task containing that item is attempted to be sent to an executor. The vast majority of the time, the fix you probably will reach for is to make the object implement Serializable. Sometimes however, this may not be easy or even possible (for example, when the type in question is out of your control).

It turns out that there is another way! If the object in question can be constructed again inexpensively, the @transient lazy val pattern may be for you:

case class Foo(foo: SomeType) {
  @transient lazy val bar: SomeOtherType = SomeOtherType(foo)

The @transient annotation has the effect of excluding the annotated item from the object it is contained within when that object is serialised. In conjunction with the lazy val, this means the field will be constructed again when first accessed on each of the executors, rather than being sent to each executor as a series of serialised bytes to deserialise as part of the task.

Sometimes this trick can actually result in modest performance improvements — for example, if the object is question is large when serialised but is cheap to construct again (like large collections computed from some smaller seed). However, carefully note that it is constructed again once per executor making it only useful for stateless items.

The next time you hit a serialisation issue in Spark, give @transient lazy val a try and see if it’s a fit for you!