Posted on

Keeping dependencies up to date in Gradle toml files

TL;DR – This post shows you how to create a script that checks for available upgrades to third party dependencies in a Gradle libs.versions.toml file

Managing dependencies – What’s missing?

Keeping dependencies up to date is part of ongoing technical debt. It’s not the most glamorous task, but neglect it and you will find yourself in trouble sooner or later.

That’s why it’s important to make it as easy for the organization to “do the right thing” and keep those dependencies up to date with the least friction possible. A great tool for this is Dependabot, which automatically creates PRs with suggested upgrades. We use it a lot. Unfortunately, Dependabot has a limitation – currently, it does not support upgrading *.toml files. We won’t know that it’s time to upgrade libraries listed in the file!

What’s a toml file?

toml is a file format for storing configurations. It’s equivalent to a json, yaml or ini file, and is easily and clearly parseable. A full description of it’s capabilities can be found in the toml project page.

Toml & Gradle

In Gradle, we can define a libs.versions.toml file and use it as a version catalog that is shared between projects. This is great for aligning the versions of third party dependencies and making sure all projects work with the same third party versions. This prevents all sort of nasty bugs that we would get when depending on different versions of the same dependency that we really don’t want to run into.

Let’s take a look at a sample libs.versions.toml file:

[versions]
aws = "1.12.311"
commonsIo = "2.11.0"
...

[libraries]
awsSdkCore = { module = "com.amazonaws:aws-java-sdk-core", version.ref = "aws" }
awsSdkS3 = { module = "com.amazonaws:aws-java-sdk-s3", version.ref = "aws" }
awsSdkSqs = { module = "com.amazonaws:aws-java-sdk-sqs", version.ref = "aws" }
awsSdkSns = { module = "com.amazonaws:aws-java-sdk-sns", version.ref = "aws" }
commonsIo = { module = "commons-io:commons-io", version.ref = "commonsIo" }
...


As you can see, it’s fairly straightforward and very readable: The versions section defines the version number, while the libraries section defines all our dependencies and uses the version.ref property to refer to the version, enabling us to define the same version for multiple dependencies that belong in the same “family”.

We now need to add a reference to the libs.versions.toml file in our Gradle build. We do that in the settings.gradle.kts file:

dependencyResolutionManagement {
  versionCatalogs {
    create("externalLibs") {
      from(files("gradle/libs.versions.toml"))
    }
  }
}

Now, we can reference the versions using the externalLibs prefix in our build.gradle.kts file:

dependencies {
    implementation(externalLibs.awsSdkCore)
}

And that’s it!

Creating a script

OK, so back to our problem: We can’t use Dependabot to generate PRs for updating the libs.versions.toml file. So what can we do that’s the best next thing?

Reviewing the file for available updates is a grueling task. The project that I work on has over 100 different versions in the toml file. Reviewing them one by one and checking each one for the latest version in https://mvnrepository.com/ is not scalable. Our dependencies will probably fall into neglect if we use this approach.

If we had a script that parses the toml file and compares the current version to the latest available version, we’d have a tool that shows us the latest version that’s available. Moreover, if we commit this file, we would be able to see what’s changed since the last commit, next time we run it.

After searching for such a script and and not finding one, I decided to write one myself.

What are the steps that we need the script to perform?

  1. Extract a list of all the third parties.
  2. Look up the current version for each of these third parties.
  3. Look up the latest release for each of them.
  4. Compare the current version to the latest version and output the diff to a file.

With a few exceptions, all of the third parties are hosted in https://repo1.maven.org/maven2/ where each one has a maven-metadata.xml which contains a tag called <release> which holds the latest GA version. That’s our data source for the latest version.

Initially I started out trying to implement this script using bash. After re-learning bash for the n-th time (always fun…), this was the result (feel free to skip to details here):

grep -E '^.* ?= ?"[^"]*"' libs.versions.toml | \
grep -v module | \
sed -E 's/([^= ]*) ?= ?"([^"]*)".*/\1 \2/g' | \
awk 'FNR == NR { a[$1] = $2 ; next } { print $1 " " a[$2] }' - <( \
  grep module libs.versions.toml | \
  grep version | \
  sed -E 's/^.*{ ?module ?= ?"([^"]*)", ?version\.ref ?= ?"([^"]*)".*/\1 \2/g' \
) | \
sort | \
awk 'FNR == NR { a[$1] = $2 ; next } $2 != a[$1] { print $1 ": " a[$1] " -> " $2 }' - <( \
  grep module libs.versions.toml | \
  grep version | \
  sed -E 's/.*module ?= ?"([^"]*)".*/\1/g' | \
  tr .: '/' | \
  sed 's/^/https\:\/\/repo\.maven\.apache\.org\/maven2\//g' | \
  sed 's/$/\/maven-metadata.xml/g' | \
  xargs curl -s | \
  egrep '(groupId|artifactId|release)' | \
  xargs -n3 | \
  sed 's/\<groupId\>\(.*\)\<\/groupId\> \<artifactId\>\(.*\)\<\/artifactId\> \<release\>\(.*\)\<\/release\>/\1\:\2\ \3/g' | \
  sort \
) > latest_available_upgrades.txt

This script works, but… As one of my reviewers pointed out in the code review (Thanks Shay!) this is hardly maintainable. Good luck debugging this or just understanding what is does.

Now that the proof of concept was, well… proven, I rewrote this in friendlier language: Kotlin script. For those of you not familiar with Kotlin script, it’s just plain Kotlin with some added annotations to help compile the script. You can run it from the command line without any prior compilation.

Here’s the same code, but in readable Kotlin script:

#!/usr/bin/env kotlin

@file:Repository("https://repo1.maven.org/maven2/")
@file:DependsOn("org.tomlj:tomlj:1.1.0")
@file:DependsOn("org.apache.httpcomponents.core5:httpcore5:5.1.4")

import org.apache.hc.core5.http.HttpHost
import org.apache.hc.core5.http.impl.bootstrap.HttpRequester
import org.apache.hc.core5.http.impl.bootstrap.RequesterBootstrap
import org.apache.hc.core5.http.io.entity.EntityUtils
import org.apache.hc.core5.http.io.support.ClassicRequestBuilder
import org.apache.hc.core5.http.protocol.HttpCoreContext
import org.apache.hc.core5.util.Timeout
import org.tomlj.Toml
import org.xml.sax.InputSource
import java.io.FileReader
import java.io.StringReader
import java.net.URI
import javax.xml.parsers.DocumentBuilderFactory
import javax.xml.xpath.XPathConstants
import javax.xml.xpath.XPathExpression
import javax.xml.xpath.XPathFactory

val coreContext: HttpCoreContext = HttpCoreContext.create()
val httpRequester: HttpRequester = RequesterBootstrap.bootstrap().create()
val timeout: Timeout = Timeout.ofSeconds(5)

val mavenTarget: HttpHost = HttpHost.create(URI("https://repo1.maven.org"))

val dbf: DocumentBuilderFactory = DocumentBuilderFactory.newInstance()
val xpathFactory: XPathFactory = XPathFactory.newInstance()
val releasePath = xpathFactory.newXPath().compile("//metadata//versioning//release")!!

main()

fun main() {
  val toml = FileReader("./libs.versions.toml").use { file ->
    Toml.parse(file)
  }

  val versions = toml.getTable("versions")!!.toMap()
  val libraries = toml.getTable("libraries")!!

  val (currentVersionRefs, missingVersions) = libraries.keySet().associate {
    val module = libraries.getString("$it.module")!!
    val versionRef = libraries.getString("$it.version.ref")
    module to versionRef
  }.entries.partition { it.value != null }

  val currentVersions = currentVersionRefs.associate { it.key to versions.getValue(it.value) }

  val latestVersions = currentVersions.mapValues { (packageName, _) ->
    getLatestVersion(packageName, mavenTarget, httpRequester, timeout, coreContext, dbf, releasePath)
  }

  // Compare current to latest version and print if different
  currentVersions.toSortedMap().forEach { (packageName, currentVersion) ->
    val latestVersion = latestVersions.getValue(packageName)
    if (currentVersion != latestVersion) {
      println("$packageName: $currentVersion -> $latestVersion")
    }
  }

  if (missingVersions.isNotEmpty()) {
    println("\n--- Missing version.ref - Need to check manually ---\n")
    println(missingVersions.map { it.key }.sortedBy { it }.joinToString("\n"))
  }

}

fun getLatestVersion(packageName: String, mavenTarget: HttpHost?, httpRequester: HttpRequester, timeout: Timeout?, coreContext: HttpCoreContext?, dbf: DocumentBuilderFactory, releasePath: XPathExpression): String {
  val (groupId, artifactId) = packageName.split(":")
  val path = "${groupId.replace(".", "/")}/$artifactId"
  val request = ClassicRequestBuilder.get()
      .setHttpHost(mavenTarget).setPath("/maven2/$path/maven-metadata.xml").build()
  val xmlString = httpRequester.execute(mavenTarget, request, timeout, coreContext).use { response ->
    if (response.code != 200) return "Could not find value in Maven Central. Need to check manually."
    EntityUtils.toString(response.entity)
  }
  val db = dbf.newDocumentBuilder()
  val document = db.parse(InputSource(StringReader(xmlString)))
  return releasePath.evaluate(document, XPathConstants.STRING).toString()
}

Notes:

  • Third party libraries that were used: https://github.com/tomlj/tomlj & https://github.com/apache/httpcomponents-client
  • There are a few cases in the toml’s libraries section where we still manage dependencies via the dependencyManagement section of the build.gradle.kts file and the library’s entry does not have a version.ref property. These are not handled by the script and appear in a separate “Missing version.ref” section.
  • There are a few libraries that are not hosted in Maven Central’s repo but rather in Jitpack’s repo – These are also not handled and a “Could not find value in Maven Central” message is printed instead.

Running this script as 

./check_latest_versions.main.kts > latest_available_upgrades.txt

we get the following output:

com.amazonaws:aws-java-sdk-core: 1.12.311 -> 1.12.326
com.amazonaws:aws-java-sdk-s3: 1.12.311 -> 1.12.326
com.amazonaws:aws-java-sdk-sns: 1.12.311 -> 1.12.326
com.amazonaws:aws-java-sdk-sqs: 1.12.311 -> 1.12.326

We can see immediately that our AWS lib’s version is not up to date. After committing latest_available_upgrades.txt, next time we run this script, we will also know when AWS publishes a new version of the libraries.

Summary

As you can see, this script automates the chore of checking for third party updates and reduces it to running a single command. It doesn’t create a PR as Dependabot does, but for most cases, it gets us most of the way towards quickly identifying which dependencies require an upgrade.

Posted on

Kotlin code styles for stopping loop iterations

This post follows https://galler.dev/getting-background-tasks-to-play-nicely-with-deployments/ and discusses different Kotlin code styles we can use for stopping a long running task. It is a summary of a discussion we had over a PR in which we wanted to terminate an execution loop over a collection.

Iterating over a collection in Kotlin

There are many ways to iterate over collections in Kotlin. Assuming you also want to collect results as you iterate, you can run the gamut with syntax styles ranging from the very Java-esque to super streamlined functional code.

Naive iteration on a collection

Let’s take an example of a loop that receives a list of customer IDs and sends them an email. Our boilerplate will look like this:

class Iterations {

  private val customerIds = List(100) {
    Integer.toHexString(Random.nextInt(256 * 256))
  }

  private fun sendEmail(customerId: String): Boolean {
    println("Sending email to customer $customerId")
    sleep(500)
    return Random.nextInt(2) == 0
  }
}

We have a list of 100 customer ID strings and a function that calls our email provider. The function takes 500ms to complete and returns a boolean success indicator.

Our first pass at creating the send loop is a style that you might see with Java developers making the transition to Kotlin, with little experience in functional programming.

Create a list of responses, iterate with a for loop and add the response to the result list. Return the statuses.

  fun sendEmails1(): List<Boolean> {

    val statuses = mutableListOf<Boolean>()
    for (customerId in customerIds) {
      val response = sendEmail(customerId)
      statuses.add(response)
    }

    return statuses
  }

This is, of course, very verbose, and also makes unnecessary use of a mutable list. We can improve on this quite easily:

  fun sendEmails2(): List<Boolean> {

    val statuses = mutableListOf<Boolean>()
    customerIds.forEach { customerId ->
      val response = sendEmail(customerId)
      statuses.add(response)
    }

    return statuses
  }

We’re still using a mutable list, but using a forEach on the collection. Not much of an improvement. Let’s keep going:

   fun sendEmails3(): List<Boolean> {

    val statuses = customerIds.map { customerId ->
      val response = sendEmail(customerId)
      response
    }

    return statuses
  }

We’ve made the jump from forEach to map which saves us the trouble of allocating a list of results. Now to clean up and streamline the code:

  fun sendEmails4() = customerIds.map { customerId -> sendEmail(customerId) }

The function is inlined. The response variable has been removed. We’ve even removed the explicit return type for brevity. From 6 lines of code we are down to 1.

As a side-note, implicit return types and putting everything in one line are not necessarily Best Practice. I recommend watching Putting Down the Golden Hammer from KotlinConf 2019 for an interesting discussion on that. This is just an example taken to the absolute extreme for the sake of discussion.

As far as performance, all implementations will take ~50 seconds to run (100 x 500ms).

A wrench in the gears

Now comes the twist: We have to be able to stop mid-way. We are shutting down our process and we need to interrupt our run.

Let’s expand our boilerplate to include a shutdown flag:

class Iterations {

  private val customerIds = List(100) {
    Integer.toHexString(Random.nextInt(256 * 256))
  }

  private val shouldShutDown = AtomicBoolean(false)

  private fun isShuttingDown(): AtomicBoolean {
    sleep(200)
    return shouldShutDown
  }

  init {
    Thread {
      sleep(3000)
      shouldShutDown.set(true)
    }.start()
  }


  private fun sendEmail(customerId: String): Boolean {
    println("Sending email to customer $customerId")
    sleep(500)
    return Random.nextInt(2) == 0
  }

This shutdown flag comes with a cost – 200ms of performance. In the example, the flag is in-memory, but it could also be a call to the DB or an external API. The list of customers could be 100K items long, not 100. Therefore we are simulating a time penalty for the flag check. We also initialize a background thread that will raise the shutdown flag after 3 seconds. Total time for each iteration will be 700ms (500 for sending the email + 200 for checking the flag)

How can we add the check to our code?

Let’s try and make a minimal change:

  fun sendWithInterrupt1(): List<Boolean> = customerIds.mapNotNull { customerId ->
    if (isShuttingDown().get()) return@mapNotNull null

    sendEmail(customerId)
  }

We’re still using map. Sort of. We just modified it to mapNotNull. We’ve made the minimal change, and the code is still relatively clean and readable. But mapNotNull will evaluate the entire collection. After 3 seconds, we have processed 5 customers (3000ms / 700ms rounded up). We will pass through 95 more redundant iterations that will cost us about 19 seconds to pass through – 95 calls x 200ms to check the flag. This is unacceptable. We should stop immediately.

The solution that’s staring us in the face is to abandon our map and go back to using forEach:

  fun sendWithInterrupt2(): List<Boolean> {

    val statuses = mutableListOf<Boolean>()
    customerIds.forEach { customerId ->
      if (isShuttingDown().get()) return statuses

      val response = sendEmail(customerId)
      statuses.add(response)
    }

    return statuses
  }

This is ugly. We have a return in two places. We’re back to a mutable list. But we will stop after 3.5 seconds. We have traded readability and elegance in return for performance.

Is there a way to eat our cake and have it too (or is it having your cake and eating it too)? Anyway, we want to stick to a functional style, but also be able to stop mid-evaluation. Turns out that there is a way to do this in Kotlin: Sequences.

Quick reminder here: Kotlin collections are eagerly evaluated by default. Sequences are lazily evaluated.

Let’s see what we can do:

  fun sendWithInterrupt3() = customerIds
      .asSequence()
      .map { customerId ->
        sendEmail(customerId)
      }
      .takeWhile { !isShuttingDown().get() }
      .toList()

We added asSequence before map. The flag check is in the takeWhile section. This acts as our stop criteries – a While condition. We wrap it up with toList().

Now, the only terminal operation here is the toList. All other operations return an intermediate sequence that is not immediately evaluated. When we run this, total running time is about 3.5 seconds – the time it takes to process 5 email sends and flag checks.

What’s the right way to go? (A.K.A. Tradeoffs)

Looking at this final version, I can’t say wholeheartedly that it is the ideal solution. Yes, the style is very functional and looks like a pipeline, but on the other hand, another developer looking at this code may not be familiar with these lesser knows operators (takeWhile, asSequence). We are adding a slight learning curve.

We did manage to avoid mutable lists and intermediate variables, keeping the code tight and leaving less room for future modifications that might lead us down a dark path.

The code is definitely more Kotlin-y and more idiomatic, but some might argue that the KISS principle should guide us to stick with the mutable, non-functional version.

Overall, my personal preference would be to go with this last version, but my recommendation to the engineer who was responsible for the change was to make the call himself according to his preference. There are more important hills to die on ¯\_(ツ)_/¯

Posted on

Preventing software antipatterns with Detekt

Detekt is a static code analyzer for Kotlin. This post talks about how I used Detekt for non-trivial, systematic improvements to code quality, using custom Detekt rules.

Source files can be found here: https://github.com/ygaller/detekt-antipatterns

Antipatterns, antipatterns everywhere

I review a lot of pull requests on a regular basis. During reviews, antipatterns show up all the time. Many of them are repetitive and addressing them is important, but does not involve deep insight into the code. In fact, repetitive nitpicking in code reviews is an antipattern in itself. Repeating the same points over and over is exasperating, and they keep popping up in PRs coming from different developers.

Sometimes you see an antipattern that is so bad that it sends you on a gardening task to weed out all instances of it in the codebase. One such example is the instantiation of Jackson’s ObjectMapper.

Antipattern example – misusing ObjectMapper

Instantiating an ObjectMapper is intensive as far as object allocation and memory use. If you’re not careful and causally use it in code that is called frequently, it will cause memory churn – the intensive allocation and release of objects, leading to frequent garbage collection. This can directly contribute to the instability of a JVM process.

Let’s have a look at an example of bad usage of the ObjectMapper: We have a CustomerRecord class which represents a DB record. This record has a json string field. When we fetch the record we want to convert the record into a DTO. This requires us to deserialize the string so that we can access the serialized object.

data class CustomerRecord(
    val firstName: String,
    val lastName: String,
    val emailAddress: String,
    val preferences: String // json
) {
 
  fun toDto(): CustomerDto {
    val preferences = ObjectMapper().registerModule(KotlinModule()).readValue<CustomerPreferences>(preferences)
    return CustomerDto(
        firstName,
        lastName,
        emailAddress,
        preferences.language,
        preferences.currency
    )
  }
}

The problem here is that with each invocation of toDto(), we are creating a new instance of ObjectMapper.

How bad is it? Let’s create a test and compare performance. For comparison, version #2 of this code has a statically instantiated ObjectMapper:

data class CustomerRecord(
    val firstName: String,
    val lastName: String,
    val emailAddress: String,
    val preferences: String // json
) {
  companion object {
    private val objectMapper = ObjectMapper()
        .registerModule(KotlinModule()) // To enable no default constructor
  }

  fun toDtoStatic(): CustomerDto {
    val preferences = objectMapper.readValue<CustomerPreferences>(preferences)
    return CustomerDto(
        firstName,
        lastName,
        emailAddress,
        preferences.language,
        preferences.currency
    )
  }
}

Let’s compare performance:

IterationsNew Duration
(Deserializations / s)		Static Duration
(Deserializations / s)		Ratio
10K767ms (13,037)44ms (227,272)x17
100K4675ms (21,390)310ms (322,580)x15
1M37739ms (26,504)1556ms (642,673)x24
New – creating ObjectMapper() in each iteration
Static – creating ObjectMapper() once

This is an amazing result! Extracting the ObjectMapper to a static field and preventing an instance from being created each time has improved performance by a factor of over 20 in some cases. This is the ratio of time that initializing an ObjectMapper takes vs. what the actual derserialization takes.

The problem with the one-time gardener

A large codebase could have hundreds of cases of bad ObjectMapper usages. Even worse, a month after cleaning up the codebase and getting rid of all these cases, some new engineers will join the group. They are not familiar with the best practice of static ObjectMappers and inadvertently reintroduce this antipattern into the code.

The solution here is to proactively prevent engineers from committing such code. It should not be up to me, the reviewer to catch such errors. After all, I should be focusing on deeper insights into the code, not on being a human linter.

Detekt to our aid

To catch these antipatterns in Kotlin, I used a framework called Detekt. Detekt is a static code analyzer for Kotlin. It provides many code smell rules out of the box. A lesser known feature is that it can be extended with custom rules.

Let’s first try and figure out what would valid uses of ObjectMapper look like. An ObjectMapper would be valid if instantiated:

  • Statically, in a companion class
  • In a top level object which is a singleton itself
  • In a private variable in the top-level of a file, outside of a class but not accessible from outside the file

What would an ObjectMapper code smell look like? When it’s instantiated:

  • As a member of a class that may be instantiated many times
  • As a member of an object that is not a companion object within a class (highly unlikely but still…)

I created a rule to detect the bad usages and permit the good ones. In order to do this, I extended Detekt’s Rule abstract class.

Detekt uses the visitor pattern, exposing a semantic tree of the code via dozens of functions that can be overridden in Rule. Each function is invoked for a different component in the code. At first glance, it’s hard to understand which is the correct method to override. It took quite a bit of trial and error to figure out that the correct function to override is visitCallExpression. This function is invoked for all expressions in the code, including the instantiation of ObjectMapper.

An example of how we examine the expression can be found in class JsonMapperAntipatternRule. We search the expression’s parents for the first class & object instances. Depending on what we find, we examine the expression in the context of a class or of a file.

    val objectIndex = expression.parents.indexOfFirst { it is KtObjectDeclaration }
    val classIndex = expression.parents.indexOfFirst { it is KtClass }

    val inFileScope = objectIndex == -1 && classIndex == -1
    if (inFileScope) {
      validateFileScope(expression)
    } else {
      validateObjectScope(objectIndex, classIndex, expression)
    }

Test coverage can be found in JsonMapperAntipatternRuleTest. Tests include positive and negative examples and can be added as we encounter more edge cases that we may not have thought of.

The positive test cases we want to allow:

// Private ObjectMapper in class' companion object
class TestedUnit {

  companion object {
    private val objectMapper = ObjectMapper()
  }
}

// ObjectMapper in object
object TestedObject {
    private val objectMapper = ObjectMapper()
}
// Private ObjectMapper in file (top level)
private val objectMapper = ObjectMapper()

class TestedObject {
}

// Part of a more complex expression
private val someObject = ObjectMapper().readValue("{}")

class TestedObject {
}

On the other hand, cases which we want to prevent:

// ObjectMapper not in companion class
class TestedUnit {

  private val objectMapper = ObjectMapper()
}

// Within a non-companion object
class External {
  object Internal {
      val notGood = ObjectMapper()
  }
}

// Not private in top level of file
val notGood = ObjectMapper()

class External {
}

The detekt-custom-rules module is a minimal Detekt plugin module. Once built, the result is a jar that can be used interactively within IntelliJ, or as a plugin when building the rest of the project.

Using the plugin with IntelliJ

After installing the Detekt plugin in IntelliJ, we enable it and point it to the plugin jar like so:

The result is that we now have syntax highlighting for this issue:

Using the plugin with a Maven build

To use is as part of the build, we need to point the maven plugin to the jar:

            <plugin>
                <groupId>com.github.ozsie</groupId>
                <artifactId>detekt-maven-plugin</artifactId>
                <version>${detekt.version}</version>
                <executions>
                    <execution>
                        <id>detekt-check</id>
                        <phase>verify</phase>
                        <goals>
                            <goal>check</goal>
                        </goals>
                    </execution>
                </executions>

                <configuration>
                    <config>config/detekt.yml</config>
                    <plugins>
                        <plugin>detekt-custom-rules/target/detekt-custom-rules-1.0-SNAPSHOT.jar</plugin>
                    </plugins>
                </configuration>
            </plugin>

With the plugin in place we receive a warning during the build:

custom-rules - 5min debt
        JsonMapperAntipatternRule - [preferences] at detekt-antipatterns/customers/src/main/java/com/ygaller/detekt/CustomerRecord.kt:18:23

In detekt.yml it’s possible to configure the build to either fail or just warn on the custom rule.

Conclusion

Using Detekt and the custom rule, I managed to screen for a non-trivial antipattern. Adding it to IntelliJ and to the build means that it is much less likely to pop up in future code reviews.

Moreover, it decenteralizes the knowledge of this issue and enables any engineer working on our code in the future to be alerted to the fact that he is introducing an inefficiency into the system. One that can easily be avoided.