A false end

In the last post, a type class named TypeRelation was added to the implementation of platform-dependent types, enabling the creation of mappings between platform-dependent types and basic integral types like Long based on the Platform type of the application.

Using this type class, methods like certain became possible to define. These methods have signatures that change how you use them depending on the platform the application is running on. In this blog post, we'll define more methods and type classes in this vein; beginning with the replacement of asLong by a safer, platform-dependent method.

What's wrong with CIntegral.asLong?

When CIntegral was defined in the previous blog posts, it was backed de facto by the Long type. To recap, de facto is something that's enforced in practice, while de jure is something that's enshrined in law, but might not be enforced. The speed limit on a road is de jure, but the speed you can go on the road before the police pull you over is the de facto speed limit.

With regards to type safety, de facto type safety is something that is not enforced by the compiler, but rather the structure of the program, while de jure type safety is type safety that should be enforced by the compiler. De facto type safety is weaker than de jure type safety but easier to produce.

CIntegral was de facto backed by Long because the methods to create CIntegral stored and accepted Long values exclusively; the de jure backing was Matchable, as that is the backing for CVal and the only type the compiler can provide guarantees for.

Since the TypeRelation mapping of types has changed the behavior of the constructors of CLong and CIntegral, the de facto guarantee that CIntegral is equivalent to Long at runtime no longer holds. Now the runtime type of these types depends on the TypeRelation mappings defined for them. In the case of CLong, it should be backed by the following:

• Long on MacOSX
• Long on Linux X64
• Int on Windows X64

Making the backing dependent on the platform-dependent type and platform has the side-effect of breaking the definition of .asLong. This method assumed that the Option returned by CVal.as[Long] would always be defined, an assumption that held when CIntegral was effectively equal to Long. That's no longer given, so a safer data extraction method is needed.

import scala.reflect.TypeTest

opaque type CVal = Matchable

object CVal:
  def apply(a: Matchable): CVal =
    a

  extension [A <: CVal](
      cval: A
  )
    inline def extract[
        P <: Platform
    ](using
        p: P,
        tr: TypeRelation[P, A],
        tt: TypeTest[
          Matchable,
          tr.Real
        ]
    ) = cval match
      case ret: tr.Real =>
        ret
      case _ =>
        throw Error(
          "the backing type wasn't what was expected"
        )

This new method is called extract and it has been defined as an extension method of types that are subtypes of CVal. CVal's companion object is the definition site for this method rather than CIntegral's because extract should be available for all platform-dependent types that have a defined mapping for the platform that's in scope. One might note that its definition is fairly similar to that of CIntegral.apply; this stems from the fact that extract is the inverse operation of apply, deconstructing a platform-dependent type rather than constructing one. The only major addition to its signature when compared to apply is the TypeTest, allowing one to check at runtime that the platform-dependent value in question houses the type it should according to the TypeRelation mappings.

Math on CLong

Now that CVal.extract has been defined, an attempt can be made at defining math operations on CLong. Defining extract beforehand was necessary due to the nature of platform-dependent types; by their nature, opaque types have no methods or operations defined, so defining generic math operations for a set of platform-dependent integral types (ie: the subtypes of CIntegral) requires extracting the underlying integral primitives.

So the question comes: "What properties should one expect from the math operations on platform-dependent integral types?". The following seems like a good starting point, based on the original requirements for the design of platform-dependent types:

1. The math operations must be as usable as possible. The less work a user has to do to perform math on platform-dependent integrals, the better.
2. The math operations must be as fast as possible barring constraints from the JVM
3. The math operations must require as little work on the part of someone implementing a platform-dependent integral as possible. This reduces the chances for errors and increases the velocity of development for people defining platform-dependent integrals.
4. The definition of math operations must be type-safe.

Integral[CLong]

A first attempt at enabling math on a platform-dependent integral will be defining Integral for one of those types. Integral is a good choice because it's part of the Scala standard library, and it has a great deal of math operations available to it.

  given [P <: Platform](using
      p: P,
      tr: TypeRelation[P, CLong],
      int: Integral[tr.Real],
      tt: TypeTest[
        Matchable,
        tr.Real
      ]
  ): Integral[CLong] with
    override def fromInt(
        x: Int
    ): CLong = CIntegral(
      int.fromInt(x)
    )

    override def parseString(
        str: String
    ): Option[CLong] = int
      .parseString(str)
      .map(CIntegral.apply)

    override def minus(
        x: CLong,
        y: CLong
    ): CLong = CIntegral(
      int.minus(
        x.extract,
        y.extract
      )
    )

    override def toDouble(
        x: CLong
    ): Double =
      int.toDouble(x.extract)

    override def toLong(
        x: CLong
    ): Long =
      int.toLong(x.extract)

    override def times(
        x: CLong,
        y: CLong
    ): CLong = CIntegral(
      int.times(
        x.extract,
        y.extract
      )
    )

    override def rem(
        x: CLong,
        y: CLong
    ): CLong = CIntegral(
      int.rem(
        x.extract,
        y.extract
      )
    )

    override def compare(
        x: CLong,
        y: CLong
    ): Int = int.compare(
      x.extract,
      y.extract
    )

    override def negate(
        x: CLong
    ): CLong = CIntegral(
      int.negate(x.extract)
    )

    override def toInt(
        x: CLong
    ): Int = int.toInt(x.extract)

    override def quot(
        x: CLong,
        y: CLong
    ): CLong = CIntegral(
      int.quot(
        x.extract,
        y.extract
      )
    )

    override def toFloat(
        x: CLong
    ): Float =
      int.toFloat(x.extract)

    override def plus(
        x: CLong,
        y: CLong
    ): CLong = CIntegral(
      int.plus(
        x.extract,
        y.extract
      )
    )

Adding this code to CLong's companion object makes an Integral[CLong] instance available whenever a TypeRelation between CLong and the Platform in scope is defined.

  test("demo 2") {
    val value = CLong(5)

    import scala.math.Integral.Implicits.*

    val res =
      summon[Platform] match
        case given Platform.WinX64.type =>
          (value + value).toInt
        case given (Platform.MacOSX64.type |
              Platform.LinuxX64.type) =>
          (value + value).toInt

    assertEquals(res, 10)
  }

The above code demonstrates the usage of this Integral instance, showing that math can be done on CLong types, and other functions like .toInt now exist for the type in certain situations.

However, this definition doesn't meet the requirements on the following points:

1. It must be defined by someone implementing a platform-dependent integral, and its definition is quite long.
2. It requires instantiating an object just to do math. That can reduce performance.
3. It is not easy to use, requiring one to fully deduce the platform running the application to enable math operations.

All three of these can be worked around, but one can't work around all three without violating the principles that type classes typically follow. The following definition, for example, would handle the instantiation problem, but make definition harder, and toss away type safety to boot:

  private var integralInstance
      : Integral[?] | Null =
    uninitialized

  given [P <: Platform](using
      p: P,
      tr: TypeRelation[P, CLong],
      int: Integral[tr.Real],
      tt: TypeTest[
        Matchable,
        tr.Real
      ]
  ): Integral[CLong] =
    if integralInstance == null
    then
      integralInstance =
        new Integral[CLong]:
          override def fromInt(
              x: Int
          ): CLong = CIntegral(
            int.fromInt(x)
          )

          //imagine the rest of the overrides here

    integralInstance
      .asInstanceOf[Integral[
        CLong
      ]]

Please note that the omission of many definitions for Integral here is done for the sake of the brevity of this blog post.

The code above is not sound along with not being type safe. The specific issue is that we do not currently have any mechanism forcing Platform to be a singleton value for the entire application runtime. This could be done, but even then, it would be a de facto safety that this mutability relied on, and it hasn't been proven yet that performance is degraded enough for mutability to become necessary. Mutability always comes with costs and considerations that make it inadvisable until one's proven that it's necessary.

A better option that meets safety concerns for math on platform-dependent integrals would be to define a new type class that takes the platform at the call site.

trait CIntegralMath[
    A <: CIntegral
]:
  def parseString(str: String)(
      using Platform
  ): Option[A]
  def minus(x: A, y: A)(using
      Platform
  ): A
  def toDouble(x: A)(using
      Platform
  ): Double
  def toLong(x: A)(using
      Platform
  ): Long
  def times(x: A, y: A)(using
      Platform
  ): A
  def rem(x: A, y: A)(using
      Platform
  ): A
  def compare(x: A, y: A)(using
      Platform
  ): Int
  def negate(x: A)(using
      Platform
  ): A
  def toInt(x: A)(using
      Platform
  ): Int
  def quot(x: A, y: A)(using
      Platform
  ): A
  def toFloat(x: A)(using
      Platform
  ): Float
  def plus(x: A, y: A)(using
      Platform
  ): A

The advantage of this approach is that a single instance of this type class can be defined per platform-dependent type without requiring mutability. It still suffers from the other issues mentioned, but they can probably be overcome with more work.

One thing to note about this new type class is that it doesn't have fromInt like Integral did. That method was ignored because it is unsafe. Testing Integral[Byte].fromInt shows that values greater than Byte.MaxValue overflow. Adding such a method is doable, but for now, the development focus will be on safety.

Going forward, it's necessary to remove the need for a user to define step by step the CIntegralMath type class for their types. To meet the ease of use requirement, one or two lines of code should be required at best. For that to be possible, one needs to implement a method of derivation for the type class.

Derivation in Scala 3

Enabling derivation of type classes typically means that someone who wants an instance of a type class for their type can make one by just providing the type to a method. A common example of derivation in Scala 2 was deriving a JSON codec for a case class with Circe.

case class Test(a: Int, b: Float)

object Test {
  implicit val codec: Codec[Test] = deriveCodec[Test]
}

The codec instance generated here is based on the properties of the case class type given to the deriveCodec method. It will look at the fields of the case class, consider their types and names, and generate a codec that encodes or decodes JSON with similar properties defined and with data that matches the expected types.

In the case of CIntegralMath, one would want to define a method that, when given a type, determines the platform mappings and how to perform math on that type. Before one can implement such a method for CIntgeralMath it's necessary to resolve how it will perform mathematical operations despite the needed type information only being known at runtime.

A possible solution is to try to get the Integral instances for the types backing the platform-dependent type. However, it's wasteful to summon all the Integral instances in the method signature of the derivation method, since we only need one integral instance per integral type per run of an application. What would be helpful is to selectively summon the single Integral instance for the backing type of platform-dependent type while the application is running.

Inline methods in Scala 3

In Scala 2, derivation was the domain of macros or libraries that relied upon them, and it was relatively hard to implement. Thankfully, Scala 3 has improved when it comes to metaprogramming, meaning that one does not need to write macros to derive type classes for a type.

With Scala 3, several metaprogramming facilities are available aside from macros. Noteworthy is the addition of the inline soft keyword for method definitions, allowing one to define inline methods.

Inline methods are methods whose bodies are inlined at their call sites. An example of how this works is the following:

inline def inlineAdd(
    x: Int,
    y: Int
): Int = x + y

def add(x: Int, y: Int): Int =
  x + y

val x = inlineAdd(5,2)
val y = add(5,2)

After compilation, the bytecode emitted for x and y would look something like this:

val x = 5 + 2
val y = add(5,2)

Because of how inline methods work, they have more information than normal about the type parameters passed into them, and can even perform tricks one would not be able to do in a normal function. An example is the summonInline inline method, which allows the summoning of context to be deferred until after inlining has been completed, allowing the summon to depend on the context of the inline method call site, rather than the context of the method itself. To make it clearer what that means, take the following example:

import compiletime.summonInline
inline def inlineGenericAdd[A](
    x: A,
    y: A
): A =
  val integral =
    summonInline[Integral[A]]
  integral.plus(x, y)

val z = inlineGenericAdd(5,2)
//val zz = inlineGenericAdd(5f,2f) //doesn't compile

In this example, z's bytecode will look something like the following after compilation:

val z = 
  val integral = summon[Integral[Int]]
  integral.plus(5,2)

An inline derivation method for CIntegralMath

Armed with knowledge about inline methods, it should be possible to summon the Integral instances for the platform mappings of a type. One can test that with a simple inline method that tries to summon an Integral[?] for a given type A <: CIntegral based on the platform that's available at runtime.

object CIntegralMath:
  inline def test[
      A <: CIntegral
  ](using
      p: Platform
  ): Integral[?] =
    p match
      case given Platform.WinX64.type =>
        val tr = summonInline[
          TypeRelation[
            Platform.WinX64.type,
            A
          ]
        ]
        summonInline[Integral[
          tr.Real
        ]]
      case _ => ???

In this code, there are two uses of summonInline, one to summon the type relation for the platform-dependent integral A, and one to summon the Integral for tr.Real. A wildcard type is used for the Integral return type because trying to make its retrieval typesafe too is trying to do too much at once.

Using a unit test, one can see if this summoning code works or not:

  test("demo 4") {
    assertNoDiff(
      compileErrors(
        "CIntegralMath.test[CLong]"
      ),
      ""
    )
  }

Running this test fails though. There's a compiler error.

    """|error: No implicit Ordering defined for tr.Real.
       |        summonInline[Integral[
       |                   ^
       |""".stripMargin

This error indicates that tr.Realis not de-aliasing into the real type (in the WinX64 case it should try to summon Integral[Int]). The reason for this is that tr is widened to TypeRelation[WinX64.type, CLong] instead of the more specific type TypeRelation[WinX64.type, CLong] { type Real = Int }.

This problem can be tough to avoid. TypeRelation uses the path-dependent type Real to indicate the real type behind the platform-dependent type, and to reference the type Real, one must have a stable identifier. Assigning something to a value is one way of making a stable identifier, as seen above, but it tends to fall prey to widening. Another option is passing something in as a parameter. Redefining the test method like so could potentially work.

  inline def summonIntegral[
      P <: Platform,
      A <: CIntegral
  ](tr: TypeRelation[P, A]) =
    summonInline[Integral[
      tr.Real
    ]]

  inline def test[
      A <: CIntegral
  ](using
      p: Platform
  ): Integral[?] =
    p match
      case Platform.WinX64 =>
        summonIntegral(
          summonInline[TypeRelation[
            Platform.WinX64.type,
            A
          ]]
        )
      case _ => ???

If one reruns the unit test named "demo 4", they would see that the unit test now succeeds. Type widening is such a fickle thing...

With this test function properly functioning, one can go ahead and make a simple first derivation of CIntegralMath. However, this integral summoning method test uses a match expression. Is that performant? Well, with the usage of the switch annotation, the compiler can make sure it's damn near perfectly performant:

  inline def test[
      A <: CIntegral
  ](using
      p: Platform
  ): Integral[?] =
    (p: @switch) match
      case Platform.WinX64 =>
        summonIntegral(
          summonInline[TypeRelation[
            Platform.WinX64.type,
            A
          ]]
        )
      case _ => ???

This little annotation is similar to the @tailrec annotation when it comes to optimization. If this annotation is used on a match expression that can't be optimized into a Java switch expression then a compiler error is thrown. Since Platform is an enum, the Scala compiler can easily optimize this match expression, and the JVM can further optimize the emitted bytecode. Likewise, since this match expression should only ever be evaluated into one result for the duration of the application, branch prediction should never have any problem predicting what path to take, making this switch expression fairly lightweight.

Now all that's left is to implement the derivation of CIntegralMath and test it.

  inline def derive[
      A <: CIntegral
  ]: CIntegralMath[A] =
    new CIntegralMath[A] {
      def getIntegral(using
          p: Platform
      ): Integral[A] =
        val integral
            : Integral[?] =
          (p: @switch) match
            case Platform.WinX64 =>
              summonIntegral(
                summonInline[
                  TypeRelation[
                    Platform.WinX64.type,
                    A
                  ]
                ]
              )
            case Platform.LinuxX64 =>
              summonIntegral(
                summonInline[
                  TypeRelation[
                    Platform.LinuxX64.type,
                    A
                  ]
                ]
              )
            case Platform.MacOSX64 =>
              summonIntegral(
                summonInline[
                  TypeRelation[
                    Platform.MacOSX64.type,
                    A
                  ]
                ]
              )

        integral
          .asInstanceOf[Integral[
            A
          ]]

      override def minus(
          x: A,
          y: A
      )(using Platform): A =
        getIntegral.minus(x, y)

      override def rem(
          x: A,
          y: A
      )(using Platform): A =
        getIntegral.rem(x, y)

      override def parseString(
          str: String
      )(using
          Platform
      ): Option[A] =
        getIntegral.parseString(
          str
        )

      override def toDouble(
          x: A
      )(using Platform): Double =
        getIntegral.toDouble(x)

      override def plus(
          x: A,
          y: A
      )(using Platform): A =
        getIntegral.plus(x, y)

      override def toFloat(x: A)(
          using Platform
      ): Float =
        getIntegral.toFloat(x)

      override def quot(
          x: A,
          y: A
      )(using Platform): A =
        getIntegral.quot(x, y)

      override def negate(x: A)(
          using Platform
      ): A =
        getIntegral.negate(x)

      override def toLong(x: A)(
          using Platform
      ): Long =
        getIntegral.toLong(x)

      override def toInt(x: A)(
          using Platform
      ): Int =
        getIntegral.toInt(x)

      override def compare(
          x: A,
          y: A
      )(using Platform): Int =
        getIntegral.compare(x, y)

      override def times(
          x: A,
          y: A
      )(using Platform): A =
        getIntegral.times(x, y)
    }

One might notice that the test method from before wasn't completed and used and that the getIntegral method that serves in its place is not an inline method at all. There are three reasons for that:

1. Inline methods were needed to enable usage of summonInline. derive here is itself inline, so getIntegral can use summonInline just fine.
2. One cannot define an inline method within an inline method. This is a fundamental restriction in Scala 3.
3. The properties of an inline method would be harmful in the case of getIntegral. Earlier in this blog post it was noted that inline methods replace their call sites with their method bodies. That means that if getIntegral was inline, the match expression it houses would be inlined into every method defined in CIntegralMath, making the class files for the definition extremely large (and grow more with each platform defined). Leaving getIntegral as a regular method allows the JVM to inline it if and when it makes sense, which can increase performance and reduce compile times.

That is to say, inline methods are powerful tools, but they must be handled with care as they can result in massive code generation if one uses them without regard to their special properties.

In any case, this derivation method looks about as complex as the definition of Integral[CLong] from before, but there's an important thing to note: it only needs to be written once, for all types, and it will be part of a library rather than needing to be user-defined. To create CIntegralMath for CLong for example, one needs only add a single simple line to the companion object:

  given CIntegralMath[CLong] =
    CIntegralMath.derive

Or at least, it should only require that. In reality, the compiler is going to complain that "No given instance of type op2.TypeRelation[(op2.Platform.LinuxX64 : op2.Platform), op2.CIntegral] was found". This is because there isn't a type relation defined for LinuxX64.type, but rather for LinuxX64.type | MacOSX64.type. A quick change to the definition of TypeRelation fixes the issue though:

trait TypeRelation[
    -P <: Platform,
    A <: CVal
]:
  type Real <: Matchable

Adding - to the P type parameter of TypeRelation makes it contravariant with regards to Platform, meaning a type relation defined for a less specific Platform can serve when a request for one with a more specific Platform is made.

Finally, some math!

  test("demo 5") {
    val a = CLong(5)
    val b = CLong(6)
    val math = summon[
      CIntegralMath[CLong]
    ]

    assertEquals(
      math.toInt(
        math.plus(a, b)
      ),
      11
    )
  }

Running this unit test shows that math can finally be done on CLong without knowing which specific platform the application is running on and with a single type class definition for CLong. Likewise, it's easy for users to derive this math definition for their platform-dependent types. However, there's a fly in the ointment. One may have noticed that the getIntegral helper method used in CIntegralMath derivation is not type-safe. This method casts the Integral[?] that it summons into an Integral[A], throwing away type safety in the name of expedience. Is this necessary? That's a question that will be answered in the next blog post...

The code for this blog post can be found at this GitHub repository under the opaque-types-2 folder.

I sincerely hope you enjoyed reading the third post in this series, and hope you continue to follow along with me as I continue to refine these platform-dependent types.

Happy Scala hacking!