I've recently stumbled upon a post on Scala Contributors by Kit Langton describing how he managed to create the illusion of adding automatically derived (publicly visible) members to a companion of a class. Something that's no longer possible to achieve with macro annotations in Scala 3. This was done by daisy-chaining a number of features together: transparent macros, Selectable types, and implicit conversions. Read the full post for the details.

As Kit mentions, the solution is just 2% shy of syntactic perfection, which is good enough for me. But another downside is the fact that it uses transparent macros, meaning that IDEA cannot infer anything about the result. And that's a bummer.

In an entry below Kit's post, Guillaume Martres, mentions that we can get rid of the implicit conversions using a new feature: "computed field names", a sub feature of named tuples. This got me curious, as I suspected that we can make it even better and get rid of the transparent macros as well. Maybe IntelliJ can join the party after all?

You can see the full example code for this post in the repo.

Computed Field Names

For the purposes of this post I'll assume that you're already familiar with named tuples, and if not, please watch Jamie Thompson's excellent talk from Scalar. Additionally, basic knowledge of the mechanics of macros is also assumed (see here if not).

But let's see what "computed field names" actually means with a small example (the video above contains an explanation as well):

object Foo extends Selectable: // 1
  type Fields = (hello: String, meaningOfLife: Int) // 2

  def selectDynamic(field: String): Any = // 3
    field match // 4
      case "hello" => "world"
      case "meaningOfLife" => 42
      case _ => sys.error("cannot happen unless `selectDynamic` is called directly") // 5

Here we are defining a new Selectable object (1) to enable "dynamically" computed fields. In (2) we define the type member Fields which is a named tuple that specifies the members that the compiler will allow us to select on Foo. These are the "computed field names". The named tuple that we use here can come from an arbitrary (type-level) computation. Something that'll we'll take advantage of later on.

Next we implement field access with selectDynamic (3), which receives the name of the field being accessed as a string, and returns Any. Although nothing is enforcing the correctness of the selectDynamic implementation, type safety is somewhat retained by the fact that the user is only allowed to call the fields that are specified in the Fields named tuple (almost, see below).

With the field name in hand, we match on it (4), and produce the appropriately typed result (as specified by Fields). Though we return Any here, the compiler will cast the value to the correct type at runtime. We must be careful to obey the contract, otherwise we'll get a runtime cast exception.

Unfortunately, despite having a solid, typeful contract with the Fields tuple, nothing is stopping the user from calling selectDynamic directly with an arbitrary string. We handle that with a custom error (5), just in case.

With the computed field names, Foo now behaves as if we added the hello: String and meaningOfLife: Int fields to it. Which we can safely use, like so:

val a: String = Foo.hello
val b: Int = Foo.meaningOfLife

But this won't compile:

Foo.stuff

As stuff was not specified in Fields¹.

This new capability of Selectable is a nontrivial improvement over what we had before. Previously we would typically have to use macros to compute the precise type of a Selectable. Now we can use this new power to recreate Kit's example sans the transparent macros.

Automatic Lenses

The example in the original post tackles the problem of automatically adding lenses for the fields of a case class to its companion. I'll shamelessly steal the same example here.

In the post we have the following code that we want to automate:

case class Person(name: String, age: Int, isAlive: Boolean)

object Person:
  val name = Lens[Person](_.name)(p => name => p.copy(name = name))
  val age = Lens[Person](_.age)(p => age => p.copy(age = age))
  val isAlive = Lens[Person](_.isAlive)(p => isAlive => p.copy(isAlive = isAlive))

As we can see the code here is very mechanical and tedious. It can be described algorithmically as:

For each field of the case class
Add a Lens val to the companion with the field's name
Implement the val with a getter/setter pair for the field

In Scala 2 we could use annotation macros to automate this completely:

@deriveLenses
case class Person(name: String, age: Int, isAlive: Boolean)

This would automatically add all the necessary vals to the companion of Person, without us even needing to name it ourselves. Unfortunately, this is no longer possible in Scala 3, as macro annotations are not allowed to define new, externally visible, members on a class.

Computed field names to the rescue!

Computed Lens Fields

As we saw before, we can create the illusion of added field names by specifying the appropriate Fields named tuple on our class. What would it be for the example above?

type PersonLenses = (name: Lens[Person, String],
                     age: Lens[Person, Int],
                     isAlive: Lens[Person, Boolean])

If we pass this as the Fields member of a Selectable, we will get the appropriately typed values on our target class. But of course, we wouldn't want to do it manually, we want to compute this automatically from the Person type.

We can use the various tuple utilities to achieve this. The first stop is to convert the Person type to the equivalent named tuple. The magic NameTuple.From lets us do just that:

scala> type PersonFields = NamedTuple.From[Person]
// defined alias type PersonFields = (name : String, age : Int, isAlive : Boolean)

PersonFields now has the same labels and types as the Person case class. Now we need to manipulate the types so that they have the corresponding Lens wrappers.

First, let's define a type-level "function" that can convert a type to the matching Lens:

type ToLens[S] = [T] =>> Lens[S, T]

This is the equivalent of a curried function of two arguments, but at the type-level. The first argument is the source of the lens, the second is the target. We just curried the Lens type constructor. We can use it like so:

summon[Lens[Person, String] =:= ToLens[Person][String]] // compiles

(Using =:= to check the result as the REPL refuses to expand the resulting type.)

Now we can use this anonymous function to map over the named tuple:

import scala.NamedTuple.Map

type LensesForPerson = PersonFields `Map` ToLens[Person]

Leaning on the intuition that tuples are like collections, we import the Map "function" for named tuples², and then we use it as an infix operator to apply the ToLens "function" over the fields of Person. We partially apply ToLens to the Person type, so that it becomes the source of all the lenses that we get.

This automatically converts Person to the corresponding field lenses:

summon[
  (name: Lens[Person, String], age: Lens[Person, Int], isAlive: Lens[Person, Boolean])
  =:=
  LensesForPerson
] // compiles

Cool, we managed to automatically compute the lenses fields for Person. Of course, what we really care about is doing this generically for any case class, not just Person. It's easy enough to generify this code, just replace Person with A all over the place:

type LensesFor[A] = NamedTuple.From[A] `Map` ToLens[A]

Short and to the point.

This still works the same for Person:

summon[
  (name: Lens[Person, String], age: Lens[Person, Int], isAlive: Lens[Person, Boolean])
  =:=
  LensesFor[Person]
] // compiles

This is the key ingredient that we need to be able to avoid transparent macros. Once we computed the desired type of the output, transparency is no longer needed. Let's tie this all together into automatic lenses.

Automatic Lensy Members

We need one more ingredient to be able to implement the lens members: the actual lenses. For this we'll use a combination of givens and macros. First a helper type:

case class LensesMap[S](lenses: Map[String, Lens[S, ?]])

This is just a wrapper around a map of lenses, all pointing to the same source S. The intent being that S stands for our target class, e.g., Person. Each map entry is keyed by a field name with the value being the field's lens. Since the fields can have different types, we cannot type the lenses with anything better than ?. But that's okay, as we are going to hide this unsafety behind a safe Selectable interface.

For our Person example this map would look something like this:

LensesMap[Person](Map(
  "name" -> Lens[Person, String](...),
  "age" -> Lens[Person, Int](...),
  "isAlive" -> Lens[Person, Boolean](...),
))

This is pretty much the same structure that we wanted to generate on the companion object but written out in an unsafe and stringly manner. Creating these values is also completely mechanical, and we can automate this with a macro:

inline given derived[S]: LensesMap[S] = ${ derivedImpl[S] }

def derivedImpl[S: Type](using Quotes): Expr[LensesMap[S]] = ???

We auto-derive LensesMap for a given type S. The implementation is a trivial adaptation of the code from Kit's original example, which you can see in the repo. Crucially, the macro is not transparent, it has a well-defined type known at the definition site of the macro.

With this building block we can finally create our automatic lens deriver:

// 1
trait DeriveLenses[A](using lensesMap: LensesMap[A]) extends Selectable:
  type Fields = LensesFor[A] // 2

  inline def selectDynamic(name: String): Lens[A, ?] =
    lensesMap.lenses.getOrElse( // 3
      name,
      sys.error(s"Invalid field name [$name]"))

The implementation is very short, but it packs a punch due to the power of named tuples. Let's break it down:

DeriveLenses take a type argument, the case class we want to create lenses for (1)
We also take as a given argument for the automatically derived lenses for the same type (1)
And extend Selectable to be able to dynamically add fields (1)
Next we compute the lens fields that correspond to the type we got (2), this ensures that we can safely access all the field lenses
Lastly, we implement selectDynamic by extracting the lens that matches the string field name we got as input, this is safe as LensesMap and LensesFor both refer to the same type

Done! But does it work?

First we need to apply derivation to the Person companion:

object Person extends DeriveLenses[Person]

Now all the lenses are available to us as if they are regular fields:

val person = Person("Alice", 42, true)

val name = Person.name.get(person)
val age = Person.age.get(person)
val isAlive = Person.isAlive.get(person)

println(s"Name: $name, Age: $age, Is Alive: $isAlive")
// Name: Alice, Age: 42, Is Alive: true

Great success. It all works, fully type-safe, with zero transparent macros. Computed fields really boosted our expressivity here.

Some observations:

This is not as efficient as the Scala 2 version, where the code compiled to regular field access. Here we have to convert from a string to get to the actual lens ³.
We lowered our syntactic perfection score to be 3% shy of perfection, due to the extra Person type parameter we have to specify. It may be possible to avoid it with another macro, but I don't see how it can be done without using transparent macros again.
The NameTuple.From type is very magical indeed, without it we would probably have to resort to using Mirror, which in turn would force us into implicit conversions like in the original code. Although the proposed modularity improvements might make this a non-issue even without From.

Having said that, I think that we got one step closer towards whiteboxish macro capabilities from the days of yore, without actually needing to be whitebox. Well done named tuples!

Aftermath

Recall that the motivation to avoid transparent macros (other than to just show that we can) was to enable IDEA to infer the lenses. We succeeded in removing the transparent macros, but as of writing IntelliJ is yet to support computed field names with named tuples.

There goes trying to invite IDEA to the party⁴...

But this unfortunately does compile and fails at runtime: Foo.selectDynamic("stuff")↩
To gain more intuition you can see more examples of generic tuple manipulation in a talk of mine, or this blog post.↩
We could somewhat improve efficiency by using a macro that creates a pattern match for the lenses rather than a Map. But this is still probably not on par with direct field access. Maybe some kind of trickery with inline matches can get us closer to direct field access.↩
Jokes aside though, the Scala team at Jetbrains deserves much credit for having quite a bit of support for named tuples, even though they are still marked experimental.↩

Whiteboxish Macro Powers with Named Tuples