Shapeless for Mortals

Sam Halliday

Scala eXchange 2015

http://fommil.github.io/scalax15

https://www.youtube.com/watch?v=RbhHytSHmgo

Introduction

Sam Halliday @fommil

  • Chartered Mathematician
    • DSP, optimisation, quantum, machine learning, etc
  • Free (Libre) Education and Software
    • 5mil textbooks in South Africa (Siyavula)
    • FSF Fellow (they fight for BSD / Apache too!)
    • netlib-java (underpinning Spark ML)
    • ENSIME core developer

Raise your hand if…

  • you use ENSIME
    • hack day tomorrow!
  • you have knowingly used the type class pattern
  • you have used Shapeless
    • experienced users, see me after class
  • you understand:
    • “A free monad is a free object relative to a forgetful functor whose domain is a category of minds.”
    • Mere Mortal, this talk is for you!

Workshop Format

No shapeless experience required

  1. Running example: spray-json
  2. Shapeless fundamentals
  3. scala-compiler workarounds
  4. spray-json-shapeless step by step
  5. Exercise!
  6. Discuss solutions
  7. More shapeless

Running Example

spray-json

// note "sealed"
sealed abstract class JsValue

// note case classes or case objects
// note recursive types
case class JsObject(fields: Map[String, JsValue]) extends JsValue
case class JsArray(elements: Vector[JsValue]) extends JsValue
case class JsString(value: String) extends JsValue
case class JsNumber(value: BigDecimal) extends JsValue

sealed trait JsBoolean extends JsValue
case object JsTrue extends JsBoolean
case object JsFalse extends JsBoolean

case object JsNull extends JsValue

JsonFormat

@implicitNotFound(msg = "Cannot find JsonFormat type class for ${T}")
trait JsonFormat[T] {
  def read(json: JsValue): T
  def write(obj: T): JsValue
}

implicit class EnrichedAny[T](any: T) {
  def toJson(implicit f: JsonFormat[T]): JsValue = f.write(any)
}

implicit class EnrichedJsValue(v: JsValue) {
  def convertTo[T](implicit f: JsonFormat[T]): T = f.read(v)
}

Fundamentals

Shapeless

“Empty your mind, be formless.
Shapeless, like water.
If you put water into a cup, it becomes the cup.
If you put water into a bottle, it becomes the bottle.
If you put water into a teapot, it becomes the teapot.
Water can flow and it can crash.
Become like water my friend."
– Bruce Lee

WARNING: Idealised

  • some code does not compile
  • bugs / limitations of scala-compiler
  • for 10 mins, we’re running TypeLevel scalac 3.0
  • coming next: workarounds

import shapeless._, labelled._, syntax.singleton._

Type Classes

trait JsonFormat[T] {
  def read(json: JsValue): T
  def write(obj: T): JsValue
}
implicit object StringJsonFormat extends JsonFormat[String] {
  def read(value: JsValue) = value match {
    case JsString(x) => x
    case other => throw new DeserializationError(other) // sic
  } // Either[String, T]
  def write(x: String) = JsString(x)
}
implicit class EnrichedAny[T](val any: T) extends AnyVal {
  def toJson(implicit f: JsonFormat[T]): JsValue = f.write(any)
}

implicit class EnrichedJsValue(val v: JsValue) extends AnyVal {
  def convertTo[T](implicit f: JsonFormat[T]): T = f.read(v)
}

Singleton Types

"bar".narrow : String("bar") // <: String
42.narrow    : Int(42)       // <: Int
'foo.narrow  : Symbol('foo)  // <: Symbol
true.narrow  : Boolean(true) // <: Boolean
Nil.narrow   : Nil.type
'a ->> "bar" : String  with KeyTag[Symbol('a), String]
'b ->> 42    : Int     with KeyTag[Symbol('b), Int]
'c ->> true  : Boolean with KeyTag[Symbol('c), Boolean]
val foo    = implicitly[Witness[String("foo")]].value  : String("foo")
val answer = implicitly[Witness[Int(42]]].value        : Int(42)
field[Symbol('a)]("bar") : FieldType[Symbol('a), String]  // <: String
field[Symbol('b)](42)    : FieldType[Symbol('b), Int]     // <: Int
field[Symbol('c)](true)  : FieldType[Symbol('c), Boolean] // <: Boolean

​ Product HList

"hello" :: 13L :: true :: HNil
                              : String :: Int :: Boolean :: HNil
('a ->> "hello") :: ('b ->> 13L) :: ('c ->> true) :: HNil
                              : FieldType[Symbol('a), String] ::
                                FieldType[Symbol('b), Int] ::
                                FieldType[Symbol('c), Boolean] ::
                                HNil
                          // <: String :: Int :: Boolean :: HNil
case class Teapot(a: String, b: Int, c: Boolean)
implicit object HNilFormat extends JsonFormat[HNil] {
  def read(j: JsValue) = HNil
  def write(n: HNil) = JsObject()
}
implicit def hListFormat[Key <: Symbol, Value, Remaining <: HList](
  implicit
  key: Witness[Key],
  jfh: JsonFormat[Value],
  jft: JsonFormat[Remaining]
): JsonFormat[FieldType[Key, Value] :: Remaining] = new JsonFormat {
  def write(hlist: FieldType[Key, Value] :: Remaining) =
    jft.write(hlist.tail).asJsObject :+
      (key.value.name -> jfh.write(hlist.head))
  def read(json: JsValue) = {
     val fields = json.asJsObject.fields
     val head = jfh.read(fields(key.value.name))
     val tail = jft.read(json)
     field[Key](head) :: tail
  }
}
val f = implicitly[JsonFormat[
          FieldType[Symbol('a), String] ::
          FieldType[Symbol('b), Int] ::
          FieldType[Symbol('c), Boolean] ::
          HNil]]

val teapot = ('a ->> "hello") :: ('b ->> 13L) :: ('c ->> true) :: HNil
val expected = "{'a': 'hello', 'b': 13, 'c': true}".parseJson

f.write(teapot) shouldBe expected
f.read(expected) shouldBe teapot 
=> JsonFormat[String]
 + JsonFormat[FieldType[Symbol('b), Int] ::
              FieldType[Symbol('c), Boolean] ::
              HNil]
=> JsonFormat[Int]
 + JsonFormat[FieldType[Symbol('c), Boolean] ::
              HNil]
=> JsonFormat[Boolean]
 + JsonFormat[HNil]

LabelledGeneric

val hlist = ('a ->> "hello") :: ('b ->> 1) :: ('c ->> true) :: HNil
case class Teapot(a: String, b: Int, c: Boolean)

val teapot = Teapot("hello", 1, true)
val generic = LabelledGeneric[Teapot]

generic.Repr : FieldType[Symbol('a), String] ::
               FieldType[Symbol('b), Int] ::
               FieldType[Symbol('c), Boolean] ::
               HNil

generic.from(hlist) shouldBe teapot

generic.to(teapot) shouldBe hlist
implicit def familyFormat[T](
  implicit
  gen: LabelledGeneric[T],
  sg: JsonFormat[T.Repr],
  tpe: Typeable[T]
): JsonFormat[T] = new JsonFormat[T] {
  if (log.isTraceEnabled)
    log.trace(s"creating ${tpe.describe}")

  def read(j: JsValue): T = gen.from(sg.read(j))
  def write(t: T): JsValue = sg.write(gen.to(t))
}
implicit val TeapotJsonFormat = cachedImplicit[Teapot]

teapot.toJson // {"a": "hello", "b": 1, "c": true}

​ CoHist Coproduct

sealed trait Receptacle
case class Glass(a: String) extends Receptacle
case class Bottle(a: Int) extends Receptacle
case class Teapot(a: Boolean) extends Receptacle

val generic = LabelledGeneric[Receptacle]
generic.Repr: FieldType[Symbol('Glass),   Glass] :+:
              FieldType[Symbol('Bottle), Bottle] :+:
              FieldType[Symbol('Teapot), Teapot] :+:
              CNil
sealed trait Coproduct
sealed trait CNil extends Coproduct
sealed trait :+:[+H, +T <: Coproduct] extends Coproduct
final case class Inl[+H, +T <: Coproduct](head : H) extends :+:[H, T]
final case class Inr[+H, +T <: Coproduct](tail : T) extends :+:[H, T]
def show(o: Coproduct): String = o match {
  case Inl(head) => "\"" + head + "\""
  case Inr(tail) => "(nil . " + show(tail) + ")"
}

showCoproduct(generic.to(Glass("foo"))) // "Glass(foo)"
showCoproduct(generic.to(Bottle(99)))   // (nil . "Bottle(99)")
showCoproduct(generic.to(Teapot(true))) // (nil . (nil . "Teapot(true)"))
implicit object CNilFormat extends JsonFormat[CNil] {
  def read(j: JsValue) = throw new GuruMeditationFailure
  def write(n: CNil)   = throw new GuruMeditationFailure
}
implicit def coproductFormat[Name <: Symbol, Head, Tail <: Coproduct](
  implicit
  key: Witness[Name],
  jfh: JsonFormat[Head],
  jft: JsonFormat[Tail]
): JsonFormat[FieldType[Name, Head] :+: Tail] = new JsonFormat {
  def read(j: JsValue) =
    if (j.asJsObject.fields("type") == JsString(key.value.name))
      Inl(field[Name](jfh.read(j)))
    else
      Inr(jft.read(j))
  def write(lr: FieldType[Name, Head] :+: Tail) = lr match {
    case Inl(found) =>
      jfh.write(found).asJsObject :+ ("type" -> JsString(key.value.name))

    case Inr(tail) =>
      jft.write(tail)
  }
}

✨ Shapeless Magic ✨

Glass("foo").toJson  // { "a":"foo" }
Bottle(99).toJson    // { "a":99    }
Teapot(true).toJson  // { "a":true  }

(Glass("foo"):Receptacle).toJson // { "type":"Glass",  "a":"foo" }
(Bottle(99)  :Receptacle).toJson // { "type":"Bottle", "a":99    }
(Teapot(true):Receptacle).toJson // { "type":"Teapot", "a":true  }

Enter the Dragon

Singleton Symbols

"bar".narrow : String("bar") // CRASH!
42.narrow    : Int(42)       // BANG!
true.narrow  : Boolean(true) // POWIE!
scala> "bar".narrow
res0: String("bar") = bar

scala> 42.narrow
res1: Int(42) = 42

scala> true.narrow
res2: Boolean(true) = true

scala> 'foo.narrow
res3: Symbol with Tagged[String("foo")] = 'foo

field[Symbol('a)]("bar")  // KERPLOP!

implicit val a = Witness('a)
scala> field[a.T]("bar")
res4: FieldType[a.T,String] = bar

Hipster.Aux (SI-823)

trait A { type T }

def f(a: A, t: a.T) = ...
// parameter a must appear in a parameter list
// that precedes dependent parameter type a.T
def f(a: A)(t: a.T) = ...
def f(implicit a: A)(implicit t: a.T) = ... // THWAPP!
// TODO https://github.com/typelevel/scala/issues/8
trait Hipster[T] { type Repr }
object Hipster {
  type Aux[T, Repr0] = Hipster[T] { type Repr = Repr0 }
}

def f[T, Repr](implicit hip: Hipster.Aux[T, Repr]) = ...
implicit def hListFormat[Key <: Symbol, Value, Remaining <: HList](
  implicit
  key: Witness.Aux[Key],
  jfh: JsonFormat[Value],
  jft: JsonFormat[Remaining]
): JsonFormat[FieldType[Key, Value] :: Remaining] = ...
implicit def coproductFormat[Name <: Symbol, Head, Tail <: Coproduct](
  implicit
  key: Witness.Aux[Name],
  jfh: JsonFormat[Head],
  jft: JsonFormat[Tail]
): JsonFormat[FieldType[Name, Head] :+: Tail] = ...
implicit def familyFormat[T, Repr](
  implicit
  gen: LabelledGeneric.Aux[T, Repr],
  sg: JsonFormat[Repr],
  tpe: Typeable[T]
): JsonFormat[T] = ...

Hipster??

def validate[F[_], G, H, V <: HList, I <: HList, M <: HList, A <: HList, R]
  (g: G)(v: V)(implicit
  hlG: FnHListerAux[G, A => R],
  zip: ZipApplyAux[V, I, M],
  mapped: MappedAux[A, F, M],
  unH: FnUnHListerAux[I => F[R], H],
  folder: LeftFolderAux[M, F[A => R], applier.type, F[HNil => R]],
  appl: Applicative[F]
) = unH((in: I) => folder(zip(v, in), hlG(g).point[F]).map(_(HNil)))

Higher Order Unification (SI-2712)

implicit def getTraversableformat[E, T <: GenTraversable[E]]( // WHAMMM!!!
  implicit
  cbf: CanBuildFrom[T, E, T],
  ef:  JsonFormat[E]
): JsonFormat[T] = ...
  • T has kind ★
  • GenTraversable[E] has kind ★→★
  • scalac can’t equate ★ to a ★→★
import scala.language.higherKinds

implicit def genTraversableFormat[T[_], E](
  implicit
  evidence: T[E] <:< GenTraversable[E], // both of kind *->*
  cbf: CanBuildFrom[T[E], E, T[E]],
  ef: JsonFormat[E]
): JsonFormat[T[E]] = ...

Implicit Resolution: Recursion

sealed trait Tree
case class Branch(left: Tree, right: Tree) extends Tree
case object Leaf extends Tree
trait Smell[T]

implicit def leafSmell: Smell[Leaf] = ???

// recursive
implicit def treeSmell(implicit
  branch: Smell[Branch],
  leaf: Smell[Leaf]): Smell[Tree] = ...
implicit def branchSmell(implicit
  tree: Smell[Tree]): Smell[Branch] = ...
implicit def treeSmell(implicit
  lazyBranch: Lazy[Smell[Branch]],
  leaf: Smell[Leaf]): Smell[Tree] = {
  val branch = lazyBranch.value
  ...
}
implicit def branchSmell(implicit
  lazyTree: Lazy[Smell[Tree]]): Smell[Branch] = {
  val tree = lazyTree.value
  ...
}
implicit def hListFormat[Key <: Symbol, Value, Remaining <: HList](
  implicit
  key: Witness.Aux[Key],
  lazyJfh: Lazy[JsonFormat[Value]],
  lazyJft: Lazy[JsonFormat[Remaining]]
): JsonFormat[FieldType[Key, Value] :: Remaining] = new JsonFormat {
  val jfh = lazyJfh.value
  val jft = lazyJft.value
  ...
}
implicit def coproductFormat[Name <: Symbol, Head, Tail <: Coproduct](
  implicit
  key: Witness.Aux[Name],
  lazyJfh: Lazy[JsonFormat[Head]],
  lazyJft: Lazy[JsonFormat[Tail]]
): JsonFormat[FieldType[Name, Head] :+: Tail] = new JsonFormat {
  val jfh = lazyJfh.value
  val jft = lazyJft.value
  ...
}
implicit def familyFormat[T, Repr](
  implicit
  gen: LabelledGeneric.Aux[T, Repr],
  lazySg: Lazy[JsonFormat[Repr]],
  tpe: Typeable[T]
): JsonFormat[T] = new JsonFormat {
  val sg = lazySg.value
}
// a convenience for implicitly[Lazy[JsonFormat[T]]].value
// but also consider using shapeless' cachedImplicit
object JsonFormat {
  def apply[T](implicit f: Lazy[JsonFormat[T]]): JsonFormat[T] = f.value
}

Watch out for Strict by @alxarchambault (shapeless 3.0).

Implicit Resolution: Cycles

Always

package com.domain.api
package com.domain.formats
package com.domain.app

Never

Use com.domain.formats from com.domain

Implicit Resolution: Priority

How it’s supposed to work:

  • Normal Scope
    • Local / outer / ancestors / package object / imports
  • Implicit Scope
    • Given parameter type
    • Expected parameter type
    • Type parameter (if there is one)

IMPLICIT RESOLUTION

How it actually works:

trait FamilyFormats extends LowPriorityFamilyFormats {
  this: StandardFormats =>
}
object FamilyFormats extends DefaultJsonProtocol with FamilyFormats

private[sjs] trait LowPriorityFamilyFormats {
  this: StandardFormats with FamilyFormats =>
  ...
}
implicitly[JsonFormat[Symbol]]            // => familyFormat
implicitly[JsonFormat[Left[String, Int]]] // => familyFormat
package brucelee.api {
  sealed trait Receptacle
  case class Glass(a: String) extends Receptacle
  case class Bottle(a: Int) extends Receptacle
  case class Teapot(a: Boolean) extends Receptacle
}
package brucelee.format {
  object MyFormats extends FamilyFormats {
    implicit override def eitherFormat[A, B](implicit
      a: JsonFormat[A],
      b: JsonFormat[B]) = super.eitherFormat[A, B]
    implicit val symbolFormat = SymbolJsonFormat

    implicit val ReceptacleF: JsonFormat[Receptacle] = cachedImplicit
  }
}
package brucelee.app {
  import spray.json._
  import brucelee.format.MyFormats.ReceptacleF

  Glass("half").toJson
}

Crappy Errors

sealed trait Dragon
case object Chinese extends Dragon
case object Japanese extends Dragon
case class Khmer(heads: Seq[Head]) extends Dragon

class Head
implicit val DragonF: JsonFormat[Dragon] = cachedImplicit

We want…

cannot find implicit for JsonFormat[Head]

We get…

cannot find implicit for JsonFormat[Dragon]

Your Turn!

Practicalities

Stringy Map for Big Data

java.util.HashMap[String, AnyRef]

type StringyMap = java.util.HashMap[String, AnyRef]
type BigResult[T] = Either[String, T]
trait BigDataFormat[T] {
  def label: String
  def toProperties(t: T): StringyMap
  def fromProperties(m: StringyMap): BigResult[T]
}
trait SPrimitive[V] {
  // e.g. Int => java.lang.Integer
  def toValue(v: V): AnyRef
  def fromValue(v: AnyRef): V
}
  • Exercise 1.1: derive BigDataFormat for sealed traits.
trait BigDataFormatId[T, P] {
  def key: String
  def value(t: T): P
}
  • Exercise 1.2: define identity constraints using singleton types.

S-Express

sealed trait Sexp

case class SexpCons(x: Sexp, y: Sexp) extends Sexp
sealed trait SexpAtom extends Sexp

case class SexpString(value: String) extends SexpAtom
case class SexpNumber(value: BigDecimal) extends SexpAtom
case class SexpSymbol(value: String) extends SexpAtom
case object SexpNil extends SexpAtom
...

(a . (b . (c . nil)))

(a b c) ;; list syntax

data

(:keyA . (valueA . (:keyB . (valueB . nil))))

(:keyA valueA
 :keyB valueB) ;; data syntax with keywords

(:file "Foo.scala"
 :line 13)

;; complex map structure
((1 2 3) "Foo.scala"
 (:key value) 13)

alist

((keyA . valueA) . ((keyB . valueB) . nil))

((keyA . valueA)
 (keyB . valueB)) ;; alist syntax

((file . "Foo.scala")
 (line . 13))

vs JSON

  • JSON keys are String
  • JSON maps are unordered
  • S-Exp naturally encodes structure
trait SexpFormat[T] {
  def read(value: Sexp): T
  def write(obj: T): Sexp
}
  • Exercise 2.1: implement SexpFormat[T] for sealed traits.
  • Exercise 2.2: customise products as “data” or “alist” forms.

Customise JsonFormat

  • Exercise 3.1: customise product field names
  • Exercise 3.2: customise coproduct (flat vs nested)
  • Exercise 3.3: customise handling of null and Option
  • Exercise 3.4: handle default values on products

Go!

More Goodies

everywhere

import poly._

object choose extends (Set ~> Option) {
  def apply[T](s : Set[T]) = s.headOption
}
scala> choose(Set(1, 2, 3))
res0: Option[Int] = Some(1)

scala> choose(Set('a', 'b', 'c'))
res1: Option[Char] = Some(a)

object Canon extends Poly1 {
  implicit def caseFile[F <: File] = at[F](_.getCanonicalFile)
}
everywhere(Canon)(List(new File(".."))) // List(File("/home"))

A better enum

// the old way!
object WeekDay extends Enumeration {
  type WeekDay = Value
  val Mon, Tue, Wed, Thu, Fri, Sat, Sun = Value
}

def isWeekend(d: WeekDay) = d match {
  case Sat | Sun => true
  // Oops! Missing case ... still compiles
}
// the new way!
sealed trait WeekDay
object WeekDay {
  val Mon, Tue, Wed, Thu, Fri, Sat, Sun = new WeekDay {}
  val values: Set[WeekDay] = Values
}

def isWeekend(d: WeekDay) = d match {
  case Sat | Sun => true
  case _         => false // compiler checks for this
}

Values is in shapeless/examples/enum.scala

Tags

import shapeless.tag, tag.@@

trait First
val First = tag[First]

trait Last
val Last = tag[Last]
def hello(first: String @@ First, last: String @@ Last) = {
  println(s"hello $first $last")
  println(s"${first.getClass} ${first.getClass}")
}
hello("Bruce", "Lee") // ZZZZZWAP!
val first = First("Bruce")
val last = Last("Lee")

hello(first, last)
// hello Bruce Lee
// class java.lang.String class java.lang.String

More…

https://github.com/milessabin/shapeless/tree/shapeless-2.2.5/examples/

Thank you!

Bottle