Announcing Shark: Smart Heap Analysis Reports for Kotlin

Recently I just released LeakCanary 2 Beta 1, and with it a new standalone library: Smart Heap Analysis Reports for Kotlin, aka Shark.

Shark is the heap analyzer that powers LeakCanary 2. It's a Kotlin standalone heap analysis library that runs at high speed with a low memory footprint.

HAHA

When I released the first version of LeakCanary in 2015, I also released HAHA (Headless Android Heap Analyzer), which was a fork of Eclipse Memory Analyzer without its UI. In HAHA 2 I replaced the Eclipse code with a repackaging of perflib, the heap analyzer used by the Android Studio memory profiler, because I knew perflib would be maintained by Google.

Perflib was not designed to run on Android devices, and loads in memory the entire graph of objects from the heap dump. The most common issue LeakCanary ran into when analyzing heap dumps was… running out of memory!

So, on a recent San Francisco - Paris trip, I used the 11h flight to prototype an idea: could we build a heap dump analyzer that could navigate a hprof file to read objects only when needed?

Shark 🦈

Turns out we can! Shark scans the Hprof file once and builds an index of each object id to the position of the object in the heap dump. After that, Shark can read any object on demand, keeping the memory footprint low and constant. To improve performance and limit IO reads, Shark holds a bit of extra metadata in the index and also uses a Least Recently Used (LRU) cache of objects.

After much tweaking and performance tuning, LeakCanary was able to parse the heap dump and find the shortest paths from GC Roots to a leaking object 6 times faster than with perflib, using 10 times less memory.

Shark is released in layers:

1. Shark Hprof: Read and write records in hprof files.
2. Shark Graph: Navigate the heap object graph.
3. Shark: Generate heap analysis reports.
4. Shark Android: Android heuristics to generate tailored heap analysis reports.
5. Shark CLI: Analyze the heap of debuggable apps installed on an Android device connected to your desktop. The output is similar to the output of LeakCanary, except you don't have to add the LeakCanary dependency to your app.
6. LeakCanary: Builds on top. It automatically watches destroyed activities and fragments, triggers a heap dump, runs Shark Android and then displays the result.

A few more things:

• Shark is built on top of Okio. Okio makes it easy to parse heap dumps efficiently.
• Shark is a 100% Kotlin library, and Kotlin is essential to its design, because Shark relies heavily on sealed classes and sequences to save memory.
• Shark has the unique ability to help narrow down the cause of memory leaks through platform specific heuristics.
• Shark is heavily tested (80% test coverage).
• Shark can run in both Java and Android VMs, with no other dependency than Okio and Kotlin.
• Shark can analyze both Java and Android VM hprof files.

Shark CLI

The Shark Command Line Interface (CLI) enables you to analyze heaps directly from your computer. It can dump the heap of an app installed on a connected Android device, analyze it, and even strip a heap dump of any sensitive data (e.g. PII, passwords or encryption keys) which is useful when sharing a heap dump.

Download it here!

Usage instructions:

$ ./bin/shark-cli

Shark CLI

                 ^`.                 .=""=.
 ^_              \  \               / _  _ \
 \ \             {   \             |  d  b  |
 {  \           /     `~~~--__     \   /\   /
 {   \___----~~'              `~~-_/'-=\/=-'\,
  \                         /// a  `~.      \ \
  / /~~~~-, ,__.    ,      ///  __,,,,)      \ |
  \/      \/    `~~~;   ,---~~-_`/ \        / \/
                   /   /            '.    .'
                  '._.'             _|`~~`|_
                                    /|\  /|\

Commands: [analyze-process, dump-process, analyze-hprof, strip-hprof]

analyze-process: Dumps the heap for the provided process name, pulls the hprof file and analyzes it.
  USAGE: analyze-process PROCESS_PACKAGE_NAME

dump-process: Dumps the heap for the provided process name and pulls the hprof file.
  USAGE: dump-process PROCESS_PACKAGE_NAME

analyze-hprof: Analyzes the provided hprof file.
  USAGE: analyze-hprof HPROF_FILE_PATH

strip-hprof: Replaces all primitive arrays from the provided hprof file with arrays of zeroes.
  USAGE: strip-hprof HPROF_FILE_PATH

Shark code examples

Reading records in a hprof file

dependencies {
  implementation 'com.squareup.leakcanary:shark-hprof:$sharkVersion'
}

// Prints all class and field names
Hprof.open(heapDumpFile)
    .use { hprof ->
      hprof.reader.readHprofRecords(
          recordTypes = setOf(StringRecord::class),
          listener = OnHprofRecordListener { position, record ->
            println((record as StringRecord).string)
          })
    }

Navigating the heap object graph

dependencies {
  implementation 'com.squareup.leakcanary:shark-graph:$sharkVersion'
}

// Prints all thread names
Hprof.open(heapDumpFile)
    .use { hprof ->
      val heapGraph = HprofHeapGraph.indexHprof(hprof)
      val threadClass = heapGraph.findClassByName("java.lang.Thread")!!
      val threadNames: Sequence<String> = threadClass.instances.map { instance ->
        val nameField = instance["java.lang.Thread", "name"]!!
        nameField.value.readAsJavaString()!!
      }
      threadNames.forEach { println(it) }
    }

Generating a heap analysis report

dependencies {
  implementation 'com.squareup.leakcanary:shark:$sharkVersion'
}

val heapAnalyzer = HeapAnalyzer(AnalyzerProgressListener.NONE)
val analysis = heapAnalyzer.checkForLeaks(
    heapDumpFile = heapDumpFile,
    leakFinders = listOf(ObjectInspector { _, reporter ->
      reporter.whenInstanceOf("com.example.ThingWithLifecycle") { instance ->
        val field = instance["com.example.ThingWithLifecycle", "destroyed"]!!
        val destroyed = field.value.asBoolean!!
        if (destroyed) {
          leakingReasons += "ThingWithLifecycle.destroyed = true"
        }
      }
    })
)
println(analysis)

Generating an Android heap analysis report

dependencies {
  implementation 'com.squareup.leakcanary:shark-android:$sharkVersion'
}

val heapAnalyzer = HeapAnalyzer(AnalyzerProgressListener.NONE)
val analysis = heapAnalyzer.checkForLeaks(
    heapDumpFile = heapDumpFile,
    referenceMatchers = AndroidReferenceMatchers.appDefaults,
    objectInspectors = AndroidObjectInspectors.appDefaults
)
println(analysis)

My hope is that with Shark, it becomes so easy to analyze heap dumps that we'll see an explosion of memory analysis tools and variants of LeakCanary for other platforms. Let's get rid of memory leaks across all platforms, try it out!