Scala implicits: uses and pitfalls

Implicits in Scala aim to make your code simpler and more concise by abstracting away some of the details of your implementation into some implicitly-resolved utilities. This keeps your concerns separated, keeping low-level logic like serialisation or the specifics of accessing data structures away from higher-level logic.

Today we'll take a look at how to write implicits, how they're resolved at compile time, and good and bad use cases for them.

Types of implicits

There are a few types of implicits in Scala, and they all work at compile time, by providing the compiler with a means of resolving what would otherwise be a type error or missing argument.

Implicit arguments

The most straightforward implicits to understand are implicit argument lists in a function or class. Consider the following, a very common use case for implicits:

class FishRepository(host: String)(implicit ec: ExecutionContext) {
  def findOne(id: String): Future[Fish] = ???
}

Here we define a repository which deals with fetching records about scaly sea creatures from a database. Since our repository will work with concurrency using Futures, it's going to need to know which ExecutionContext to use. We want that to be configurable further up the stack, since we might need to tweak our execution contexts based on load in different parts of the application, and most likely want database IO to run on a dedicated future pool.

To save us the effort of passing in an explicit ExecutionContext everywhere it may be needed throughout the stack, we simply declare it an implicit argument and allow it to be resolved automatically.

implicit val ec: ExecutionContext = ExecutionContext.global

val fisher = new FishRepository("localhost")  // implicit ec is filled in for us

To fill in these implicit arguments, the compiler will do the following:

  1. Notice that you've not provided the implicit argument list
  2. Look for definitions in scope marked implicit with type ExecutionContext
  3. Having found exactly one, provide it to the function
  4. Repeat from (2) for any other arguments in this set of brackets

It's important to note that point (3) here means you have to be careful with scoping your implicits, and with which types are declared as implicit arguments. If an implicit can be fulfilled in multiple ways it will simply fail, and you'll have to either change the scope of your implicits or provide the arguments explicitly instead. In general they should be very specific types, to avoid confusion or ambiguity about what is expected.

If you defined an implicit String argument, for example, it could mean anything – a name, a hoststring, a password, a type of root vegetable. An implicit Int might be a buffer size, a rate limit, a concurrency limit, or all of the above. To avoid this, use a specific type, creating one if needbe – e.g. case class Parallelism(val value: Int) is one I've used in the past to encapsulate the parallelism setting desired in akka stream components.

Implicit classes

Implicit classes are typically used in Scala to provide type class functionality. This is a means of achieving "ad hoc polymorphism" – adding a capability to a type on the fly, and without needing to modify or have any control over the type in question.

An excellent example of type classes in Scala, and one which I touched upon last week, is adding JSON serialisation functionality to your models. Here's a simplified example which acts as CSV encoding behaviour, currently for a single case class:

case class Fish(species: String, age: Int, endangered: Boolean)

implicit class FishEncoder(fish: Fish) {
  def toCsv: String = Seq(fish.species, fish.age, fish.endangered).mkString(",")
}

This allows us to treat toCsv as a method of Fish, even though it's not defined on the class:

> Fish("Salmon", 12, false).toCsv
res1: String = "Salmon,12,false"

Defining a type class to a single specific type like this isn't very helpful in the case of a JSON or CSV encoder, so we'll revisit this example shortly to make it more generic.

Here's what the compiler is doing in this case:

  1. Look for a method called toCsv defined on Fish or one of its parents
  2. Fail to find that, so now look for in-scope implicit classes for Fish which might provide one
  3. Find FishEncoder, instantiate it with our Fish, and call toCsv on this instance instead

Implicit conversions

Firstly, note that implicit conversions are no longer recommended for use, and you should use implicit classes to achieve the same thing instead. In fact, standard library tooling like the implicit conversion-based scala.concurrent.JavaConversions was deprecated in favour of the implicit class-based scala.concurrent.JavaConverters several versions ago, and since removed entirely in 2.13.

Implicit conversions convert one type to another using an implicit function. These are the "most magical" types of implicits in that the process is entirely implicit – unlike implicit classes, where you call a method, hinting that there's an implicit class involved if it's not defined on the class itself. With implicit conversions, you can simply use the source type as if it were the target type.

Consider the following:

import scala.language.implicitConversions

case class Fish(species: String, age: Int, endangered: Boolean)
case class FishSummary(fishCount: Int, hasEndangered: Boolean)

implicit def fishToSummary(fish: Seq[Fish]): FishSummary = {
  FishSummary(fish.length, fish.exists { _.endangered })
}

With the implicit conversion defined and in scope, we can simply treat a Seq[Fish] as if it were a FishSummary:

> val summary: FishSummary = Seq(Fish("Salmon", 12, false), Fish("Atlantic Halibut", 9, true))
summary: FishSummary = FishSummary(2, true)

// --- or, pass it into a function which expects a FishSummary, like:
def printSummary(summary: FishSummary): Unit = print(summary)

> printSummary(Seq(Fish("Salmon", 12, false), Fish("Atlantic Halibut", 9, true)))
FishSummary(2,true)

This time, the compiler resolves the implicit as follows:

  1. Check that the argument is the correct type, i.e. Seq[Fish] conforms to type FishSummary
  2. Seeing that it doesn't, look for an in-scope implicit conversion which takes a Seq[Fish] and produces FishSummary
  3. Call the implicit conversion to produce the type which the compiler expected, FishSummary

But, to reiterate: implicit conversions are no longer recommended since they're too prone to be confusing: since version 2.10, the compiler will produce a warning if you define them. You can acknowledge this and switch off the warning by importing scala.language.implicitConversions as shown above. In extremely narrow use cases a locally-scoped implicit conversion may produce a small readability benefit, so this will allow you to use them without warnings if you really need one.

To perform the same operation shown above using an implicit class, you could use:

implicit class RichFishSeq(fish: Seq[Fish]) {
  def toSummary: FishSummary = FishSummary(fish.length, fish.exists { _.endangered })
}

val summary: FishSummary = {
  Seq(Fish("Salmon", 12, false), Fish("Atlantic Halibut", 9, true)).toSummary
}

Note that this is, regardless, quite a contrived example, and likely too specific to be a good use case for an implicit at all. A simple utility function to create the summary is likely to be much easier to understand in this case. Exercise restraint in deciding when to use an implicit.

Organising implicits

For implicits to work, they have to be in scope when they're used. Given that the purpose of implicits is to make your code concise and readable and avoid mixing layers of logic, you'll likely want to package them separately and reusably.

Typically, implicits are packaged like this:

trait FishImplicits {
  implicit class RichFishSeq(fish: Seq[Fish]) {
    def toSummary: FishSummary = ???
  }

  // More fish-related puns / converters
}

object FishImplicits extends FishImplicits

This is a standard pattern you'll see in many libraries, as it allows you to use either import FishImplicits._ or extends FishImplicits to bring your implicits into scope. This gives you some flexibility to compose traits to collect together implicits, or limit their scope by importing. This, in turn, allows you to control the exact scope of your implicits to minimise confusion.

Note also that we've also used Implicits in the name, making it very clear that we're bringing in some implicits. The name may be slightly more specific where appropriate, e.g. Spray JSON implicits are defined in XXXJsonProtocol traits, and the standard library scala-to-java collection converters are called AsScalaConverters. In all cases we aim for clarity that implicits are involved and that we're effectively bringing in a new DSL. This gives us a heads up of where to look if we're experiencing unexpected behaviour or wondering about unexpected syntax, as unannounced implicits can cause real pain.

Finally, a specific call-out: for all of the reasons above, do not be tempted to place implicits in the package object. Package objects are generally best avoided entirely, as they can make scoping confusing regardless, but that's further compounded when you're using implicits. Implicits should always involve a clear import as a heads up that they're present.

Generic implicits

Implicit classes become even more powerful when they're combined with generic types, as I hinted at earlier when describing how a typical JSON (or CSV) library might work. A full discussion of this is best left for another day, but to complete our earlier thought, let's have a quick look at how a typical JSON library works, by thinking about how our toCsv API might be completed for generic types:

// We need to provide a CsvEncoding subclass for each type we want to encode to CSV
trait CsvEncoding[T] {
  def encode(t: T): String
}

// The core of the CSV encoder implicits; this defines any type as encodable to CSV
// if an implicit CsvEncoding is available, using the toCsv method
trait CsvImplicits {
  implicit class CsvEncodable[T : CsvEncoding](t: T) {
    def toCsv: String = implicitly[CsvEncoding[T]].encode(t)
  }
}

// Our type-specific encodings
trait FishCsvEncodings {
  implicit object FishCsvEncoding extends CsvEncoding[Fish] {
    override def encode(fish: Fish): String = {
      Seq(fish.species, fish.age, fish.endangered).mkString(",")
    }
  }
}

// Finally, just bring both the CSV implicits and our fish CSV encodings into scope
object App extends CsvImplicits with FishCsvEncodings {
  val fishy: String = Fish("trout", 11, false).toCsv
}

This works by defining a CsvEncodable type which will work for any type which has an implicit CsvEncoding available. The .toCsv behaviour comes from CsvEncodable, and it falls back on the definition you've provided via a CsvEncoding instance. Bring both the implicit class and your set of implicit encoders into scope, and the API works as expected.

Two things to clear up:

With that sugar removed, this is defined as:

implicit class CsvEncodable[T](t: T)(implicit ev: CsvEncoding[T]) {
  def toCsv: String = ev.encode(t)
}

There are a few ways which this could all be improved, of course, as our CSV "encoder" is very naive. We may also consider an additional DSL to add toCsvString to types to determine how individual fields should be converted to strings before inserting into the CSV, rather than assuming calling .toString on them produces the desired result. I leave it to the reader to think about other improvements we could make here.

A horror story

Since I've touched a few times on some potential pitfalls of using implicits poorly, with too-weak types, poor scoping, or simply overzealousness, I'd like to share a horror story from a codebase I had to work with several years ago.

The codebase in question was a large monolithic application which generally had a lot of problems, including very weak types used, frequent use of Map[String, Any] and similar instead of case classes, large numbers of fields combined in difficult-to-follow ways, a 55-minute build time, very difficult to parse logs, poor test coverage, and a structure which made unit testing difficult.

All of this contributed to slow development, but one particular implicit incident is engraved in my memory.

I first stumbled into this by attempting to call a method like:

def store(id: ObjectId, title: String, encoding: String, date: Instant): Future[Unit]

like this:

repository.store(title, id, encoding, Instant.now())

If you're keen-eyed, you'll see that I've transposed the first two arguments. This is often a drawback of having too-weak types, and if both were Strings I'd likely only find this out during a test run, or in this application, by carefully scrutinising logs or database records to find that my ID was wrong. But with one argument having type ObjectId this simply won't compile, and I'll immediately see the problem. Right? Wrong.

Enter an implicit conversion, defined in the package object, in a package with 100+ classes, hidden from sight:

implicit def any2ObjectId(a: Any): ObjectId = a match {
  case id: ObjectId => id
  case str: String => ObjectId(str)
  case _ => throw new Exception("Bad object ID")
}

Unhelpfully, this broad implicit was trying to correct my compile error by forcing the type to ObjectId if I provided a String, or anything else for that matter. Combined with other issues in the codebase, the resulting camouflaged error took a staggering two weeks to identify.

If you're keen-eyed again, you may also notice that the first branch of the match here is unreachable: the compiler will not reach for the implicit conversion if it already has an ObjectId, so this could not be called with one. With that in mind, I'd like to also give an honourable mention to another implicit I found alongside it:

implicit def any2Any(a: Any): Any = a match {
  case id: ObjectId => id
  case str: String => ObjectId(str)
  case _ => throw new Exception("Bad object ID")
}

...which, thankfully, does nothing at all.

We collectively resolved to eliminate these implicits, which was a long process, as deleting some of them spawned several hundred compile errors all across a monolith a dozen developers were actively working on.

tl;dr

An executive summary: