Today I just learned something about shell/bash that I never knew before which caused me quite a bit of confusion.
If you have set -e on in a bash/shell script, as you probably know the shell will exit if any line has a non-zero exit status and it isn’t caught and handled by something.
But what I did not know before today was that this is “weird” inside functions:
set -e
foo () {
echo "I should be printed first!"
false
echo "I should not be printed?"
}
foo || echo "I should be printed last!"
Maybe somewhat surprisingly the output of this is:
I should be printed first!
I should not be printed?
This happens because if you handle the output of an entire function then it kind of acts like every line inside the function is also handled. This means lines can run that you were not expecting!
Additionally, the last line that runs will be the exit status of the function, so in this case being echo my “I should be printed last!” line didn’t even run because echo always returns a zero exit status!
Therefore, it’s more correct to do:
set -eo pipefail
foo () {
echo "I should be printed first!"
false || echo "I should be printed last!" && return 1
echo "I should not be printed?"
}
foo
That is, handle each line failing where it happens, and don’t try to handle whole functions.
This is documented in some of the various versions of documentation for set -e:
-e errexit
Exit immediately if any untested command fails in non-interactive mode. The exit status of a command is considered to be explicitly tested if the command is part of the list used to control an if, elif, while, or until; if the command is the left hand operand of an “&&” or “||” operator; or if the command is a pipeline preceded by the ! operator. If a shell function is executed and its exit status is explicitly tested, all commands of the function are considered to be tested as well.
But I think this is almost certainly surprising behaviour for most users of shell.
Following in the footsteps of @tpolecat’s wonderful “Recommended Scalac Flags for 2.12”, and previous version for 2.11, I’d like to share with you an updated list for Scala 2.13.
Scala 2.13 has removed a significant number of old lints and compiler flags, so you will find the previous list no longer works as-is with Scala >= 2.13.0.
This list is current as of Lightbend Scala 2.13.0-RC1.
scalacOptions ++= Seq(
"-deprecation", // Emit warning and location for usages of deprecated APIs.
"-explaintypes", // Explain type errors in more detail.
"-feature", // Emit warning and location for usages of features that should be imported explicitly.
"-language:existentials", // Existential types (besides wildcard types) can be written and inferred
"-language:experimental.macros", // Allow macro definition (besides implementation and application)
"-language:higherKinds", // Allow higher-kinded types
"-language:implicitConversions", // Allow definition of implicit functions called views
"-unchecked", // Enable additional warnings where generated code depends on assumptions.
"-Xcheckinit", // Wrap field accessors to throw an exception on uninitialized access.
"-Xfatal-warnings", // Fail the compilation if there are any warnings.
"-Xlint:adapted-args", // Warn if an argument list is modified to match the receiver.
"-Xlint:constant", // Evaluation of a constant arithmetic expression results in an error.
"-Xlint:delayedinit-select", // Selecting member of DelayedInit.
"-Xlint:doc-detached", // A Scaladoc comment appears to be detached from its element.
"-Xlint:inaccessible", // Warn about inaccessible types in method signatures.
"-Xlint:infer-any", // Warn when a type argument is inferred to be `Any`.
"-Xlint:missing-interpolator", // A string literal appears to be missing an interpolator id.
"-Xlint:nullary-override", // Warn when non-nullary `def f()' overrides nullary `def f'.
"-Xlint:nullary-unit", // Warn when nullary methods return Unit.
"-Xlint:option-implicit", // Option.apply used implicit view.
"-Xlint:package-object-classes", // Class or object defined in package object.
"-Xlint:poly-implicit-overload", // Parameterized overloaded implicit methods are not visible as view bounds.
"-Xlint:private-shadow", // A private field (or class parameter) shadows a superclass field.
"-Xlint:stars-align", // Pattern sequence wildcard must align with sequence component.
"-Xlint:type-parameter-shadow", // A local type parameter shadows a type already in scope.
"-Ywarn-dead-code", // Warn when dead code is identified.
"-Ywarn-extra-implicit", // Warn when more than one implicit parameter section is defined.
"-Ywarn-numeric-widen", // Warn when numerics are widened.
"-Ywarn-unused:implicits", // Warn if an implicit parameter is unused.
"-Ywarn-unused:imports", // Warn if an import selector is not referenced.
"-Ywarn-unused:locals", // Warn if a local definition is unused.
"-Ywarn-unused:params", // Warn if a value parameter is unused.
"-Ywarn-unused:patvars", // Warn if a variable bound in a pattern is unused.
"-Ywarn-unused:privates", // Warn if a private member is unused.
"-Ywarn-value-discard", // Warn when non-Unit expression results are unused.
"-Ybackend-parallelism", "8", // Enable paralellisation — change to desired number!
"-Ycache-plugin-class-loader:last-modified", // Enables caching of classloaders for compiler plugins
"-Ycache-macro-class-loader:last-modified", // and macro definitions. This can lead to performance improvements.
)
I recently had a bit of fun after upgrading to Homebrew v1.9.0, so wanted to write a quick PSA for others who may hit the same issue.
Homebrew >= v1.9.0 has a breaking change to brew link --force:
brew link --force will not link software already provided by macOS.
That is, this change means that Homebrew will no longer allow brew link to override anything MacOS already ships with.
So for example, if you used Homebrew to install a recent version of Ruby, before v1.9.0 Homebrew’s version would’ve been in your path — now the MacOS system version will be in your path:
$ ruby -v
ruby 2.3.7p456 (2018-03-28 revision 63024) [universal.x86_64-darwin18]
$ /usr/local/opt/ruby/bin/ruby -v
ruby 2.6.0p0 (2018-12-25 revision 66547) [x86_64-darwin18]
If you try to brew link it, Homebrew will tell you why it’s refusing to, and give you a snippet of script for prepending it to your PATH in your chosen shell (in my case, fish):
$ brew link --force ruby
Warning: Refusing to link macOS-provided software: ruby
If you need to have ruby first in your PATH run:
echo 'set -g fish_user_paths "/usr/local/opt/ruby/bin" $fish_user_paths' >> ~/.config/fish/config.fish
For compilers to find ruby you may need to set:
set -gx LDFLAGS "-L/usr/local/opt/ruby/lib"
set -gx CPPFLAGS "-I/usr/local/opt/ruby/include"
For pkg-config to find ruby you may need to set:
set -gx PKG_CONFIG_PATH "/usr/local/opt/ruby/lib/pkgconfig"
You can also see this information for any given package by running brew info <package>.
The upgrade to v1.9.0 does not give any warning about this — the change is effectively silent — so beware of scripts depending any binaries that clash with MacOS provided ones being in your default system PATH!
As a follow-up to my previous post on Unit vs null, I thought it might be useful to talk about some of the other special types we have in statically typed languages.
There are actually two other special types in the type-system that you may be interested in: namely, the bottom and the top type.
The Bottom Type
The bottom type (written as Nothing in Scala) is a type that is impossible to create. It is commonly used as a placeholder when a type isn’t known to the compiler. See for example what happens when we don’t tell the compiler the types of the keys and values in an empty Map:
scala> Map.empty
res0: scala.collection.immutable.Map[Nothing,Nothing] = Map()
As it has no other information to go on, the types for both end up as Nothing.
It’s also useful for saying that a function never returns — for example, a function that exits the program before returning could be written as:
def doSomething(): Nothing = sys.exit(1)
It should be clear from the type that it is impossible for this function to produce a return value. In fact, it’s not even possible to write a function that has a real return value if you have Nothing as the return types:
scala> def doSomething(): Nothing = 123
<console>:11: error: type mismatch;
found : Int(123)
required: Nothing
def doSomething(): Nothing = 123
You therefore know that upon seeing a function returning Nothing, it will definitely never return if you call it — all guranteed because it’s impossible to make a value of type Nothing!
The Top Type
The opposite of the bottom type is the top type. Also called the universal type (and written in Scala as Any), this is simply a way to say “this could be any possible type”. For example:
def doSomething(): Any = 123
This function actually returns an Int but typing it Any works — what’s going on here? This is because Any is actually the ancestor of all types in Scala, even Object:
scala> val x: Object = new Object {}
x: Object = $anon$1@b428830
scala> x.isInstanceOf[Any]
res1: Boolean = true
You’ll often see Any used by the compiler when it is trying to fill in a missing type (so called “type inference”) but couldn’t find anything more specific to choose.
Why Do We Need These Types?
The reason both bottom and top types have to exist comes down to how static-typing works — compilers of statically typed languages are, fundamentally, trying to solve two problems:
- Type inference: That is, filling in missing types by finding the most specific type that something should be.
- Constraint solving: Proving correct or finding a set of constraints on types given a heirachy of types. This is used for things like subtyping and supertyping, and is called “variance”.
The bottom type is a natural way to represent an impossible scenario during the first of these tasks. The bottom type is the default type if a type cannot be inferred — that is, when you don’t annotate a type, Nothing will be the end result if the compiler cannot find another more specific type after searching through everything else.
The top type is the default for constraint reduction. It gives a compiler a final ultimate result if nothing can be proven when doing constraint reduction. Any is what you end up with when two types constrain each other so much that they have no other ancestor in common, and works because every single type has Any at least in common.
The other day, I was asked by somebody why we had Unit in Scala, and what was the difference over just using null. I thought this was a really interesting question — so here’s my response in a slightly longer form!
The Boolean Type
Let’s start from something we know well — the Boolean type! When writing code, we use these all the time. As you know, they can only have two possible values — true or false.
The boolean type is the smallest possible type with which we can represent information. It can show something was true, or false, on or off, yes or no, 1 or 0 — just a single bit is all it has to work with, but it’s enough .
Imagine the following example function:
def doSomething(bool: Boolean): Boolean = ???
We have a function that takes a Boolean and returns a Boolean. We know immediately some things about this function:
- It is going to take either a
true or false value.
- It is going to return either a
true or false value.
- It cannot take or return any other values.
We know all of these things because the number of possible values for Boolean is limited. This limitation is what allows us to reason about what a function can do just from it’s type — and this is an incredibly useful tool that static-typing gives you.
Parameters and Return Values As Tuples
Remember that function we wrote before?
def doSomething(bool: Boolean): Boolean = ???
We can think about functions in a slightly different way: we can think about their parameters being passed as tuples and them returning tuples:
// Is the same thing as:
val doSomething: (Boolean) => (Boolean) = ???
// ^ tuple ^ tuple
This is how a compiler actually sees the functions you wrote — names given to parameters and syntax around showing the return type is just sugar that makes it read better to our human eyes. Really, behind the scenes, it’s more like passing tuples of arguments and receiving tuples of results back.
Keep this tuple thing in mind!
The Unit Type
Sometimes we do things where we don’t care about or can’t get a result — things that may have side-effects, for example. Often we need a way to say “this function returns nothing useful” — how can we do that without wasting space on a boolean if a boolean is the smallest piece of information we have to work with?
Well, it turns out there is a type for saying “I have nothing useful to return” — and that type is called Unit.
Unit is a type that only has a single value (hence the name). This single value is also called “unit” but to differentiate the value from the type, in some languages we write it as ().
Notice something? That looks like a tuple! And indeed, that’s not a mistake — Unit is just a way of saying “an empty tuple”!
Unit is useful because of an important property about the empty tuple
— there is only one possible value of it, (). Where, for example, (Boolean) could become (true) or (false), Unit will always be () only. As a result of only having a single value, this means it is entirely useless for actually encoding any information — after all, what information can you impart if you’re only allowed to respond with one thing?
Being able to say “I have no information” is still a useful thing, and so we use Unit and () to do just that:
def doSomething(file: File): Unit = ???
Just from looking at the definition of this function, and as you were able to with the boolean example, you can tell this function must have some side-effect . We could even use this information to guess what this function does (perhaps it deletes the file, or maybe calls touch?)
Null: The Odd One Out
So now to the question at hand: what makes Unit useful over null?
The issue with null is that it is, basically, a hack! null is a special value that can take the place of any type, but it is not a type itself. This completely breaks our ability to reason using just types. Take for example this function:
def doSomethingpath: Path): File = ???
We might look at this and think “oh cool, maybe it’s opening a File for us?”. Well, if we didn’t have null you might possibly be right — but unfortunately it’s completely valid for this function to be implemented thusly:
def doSomethingpath: Path): File = null
What’s wrong about null is that it is a magic value not represented in the type-system, and therefore it limits a lot of our ability to use the types to their full power. This is why most idiomatic Scala eschews the use of null over things like Unit, Option[A], or Either[A, B] — these types tell the full story of what is going on, without hacks, and allow us the full power of deductive reasoning. null in Scala exists purely because of Java compatibility, and you would be wise to avoid it at all costs!