TL;DR – This post shows you how to create a script that checks for available upgrades to third party dependencies in a Gradle libs.versions.toml file
Managing dependencies – What’s missing?
Keeping dependencies up to date is part of ongoing technical debt. It’s not the most glamorous task, but neglect it and you will find yourself in trouble sooner or later.
That’s why it’s important to make it as easy for the organization to “do the right thing” and keep those dependencies up to date with the least friction possible. A great tool for this is Dependabot, which automatically creates PRs with suggested upgrades. We use it a lot. Unfortunately, Dependabot has a limitation – currently, it does not support upgrading *.toml
files. We won’t know that it’s time to upgrade libraries listed in the file!
What’s a toml file?
toml
is a file format for storing configurations. It’s equivalent to a json
, yaml
or ini
file, and is easily and clearly parseable. A full description of it’s capabilities can be found in the toml project page.
Toml & Gradle
In Gradle, we can define a libs.versions.toml
file and use it as a version catalog that is shared between projects. This is great for aligning the versions of third party dependencies and making sure all projects work with the same third party versions. This prevents all sort of nasty bugs that we would get when depending on different versions of the same dependency that we really don’t want to run into.
Let’s take a look at a sample libs.versions.toml
file:
[versions] aws = "1.12.311" commonsIo = "2.11.0" ... [libraries] awsSdkCore = { module = "com.amazonaws:aws-java-sdk-core", version.ref = "aws" } awsSdkS3 = { module = "com.amazonaws:aws-java-sdk-s3", version.ref = "aws" } awsSdkSqs = { module = "com.amazonaws:aws-java-sdk-sqs", version.ref = "aws" } awsSdkSns = { module = "com.amazonaws:aws-java-sdk-sns", version.ref = "aws" } commonsIo = { module = "commons-io:commons-io", version.ref = "commonsIo" } ...
As you can see, it’s fairly straightforward and very readable: The versions
section defines the version number, while the libraries
section defines all our dependencies and uses the version.ref
property to refer to the version, enabling us to define the same version for multiple dependencies that belong in the same “family”.
We now need to add a reference to the libs.versions.toml
file in our Gradle build. We do that in the settings.gradle.kts
file:
dependencyResolutionManagement { versionCatalogs { create("externalLibs") { from(files("gradle/libs.versions.toml")) } } }
Now, we can reference the versions using the externalLibs
prefix in our build.gradle.kts
file:
dependencies { implementation(externalLibs.awsSdkCore) }
And that’s it!
Creating a script
OK, so back to our problem: We can’t use Dependabot to generate PRs for updating the libs.versions.toml
file. So what can we do that’s the best next thing?
Reviewing the file for available updates is a grueling task. The project that I work on has over 100 different versions in the toml
file. Reviewing them one by one and checking each one for the latest version in https://mvnrepository.com/ is not scalable. Our dependencies will probably fall into neglect if we use this approach.
If we had a script that parses the toml
file and compares the current version to the latest available version, we’d have a tool that shows us the latest version that’s available. Moreover, if we commit this file, we would be able to see what’s changed since the last commit, next time we run it.
After searching for such a script and and not finding one, I decided to write one myself.
What are the steps that we need the script to perform?
- Extract a list of all the third parties.
- Look up the current version for each of these third parties.
- Look up the latest release for each of them.
- Compare the current version to the latest version and output the diff to a file.
With a few exceptions, all of the third parties are hosted in https://repo1.maven.org/maven2/ where each one has a maven-metadata.xml
which contains a tag called <release>
which holds the latest GA version. That’s our data source for the latest version.
Initially I started out trying to implement this script using bash. After re-learning bash for the n-th time (always fun…), this was the result (feel free to skip to details here):
grep -E '^.* ?= ?"[^"]*"' libs.versions.toml | \ grep -v module | \ sed -E 's/([^= ]*) ?= ?"([^"]*)".*/\1 \2/g' | \ awk 'FNR == NR { a[$1] = $2 ; next } { print $1 " " a[$2] }' - <( \ grep module libs.versions.toml | \ grep version | \ sed -E 's/^.*{ ?module ?= ?"([^"]*)", ?version\.ref ?= ?"([^"]*)".*/\1 \2/g' \ ) | \ sort | \ awk 'FNR == NR { a[$1] = $2 ; next } $2 != a[$1] { print $1 ": " a[$1] " -> " $2 }' - <( \ grep module libs.versions.toml | \ grep version | \ sed -E 's/.*module ?= ?"([^"]*)".*/\1/g' | \ tr .: '/' | \ sed 's/^/https\:\/\/repo\.maven\.apache\.org\/maven2\//g' | \ sed 's/$/\/maven-metadata.xml/g' | \ xargs curl -s | \ egrep '(groupId|artifactId|release)' | \ xargs -n3 | \ sed 's/\<groupId\>\(.*\)\<\/groupId\> \<artifactId\>\(.*\)\<\/artifactId\> \<release\>\(.*\)\<\/release\>/\1\:\2\ \3/g' | \ sort \ ) > latest_available_upgrades.txt
This script works, but… As one of my reviewers pointed out in the code review (Thanks Shay!) this is hardly maintainable. Good luck debugging this or just understanding what is does.
Now that the proof of concept was, well… proven, I rewrote this in friendlier language: Kotlin script. For those of you not familiar with Kotlin script, it’s just plain Kotlin with some added annotations to help compile the script. You can run it from the command line without any prior compilation.
Here’s the same code, but in readable Kotlin script:
#!/usr/bin/env kotlin @file:Repository("https://repo1.maven.org/maven2/") @file:DependsOn("org.tomlj:tomlj:1.1.0") @file:DependsOn("org.apache.httpcomponents.core5:httpcore5:5.1.4") import org.apache.hc.core5.http.HttpHost import org.apache.hc.core5.http.impl.bootstrap.HttpRequester import org.apache.hc.core5.http.impl.bootstrap.RequesterBootstrap import org.apache.hc.core5.http.io.entity.EntityUtils import org.apache.hc.core5.http.io.support.ClassicRequestBuilder import org.apache.hc.core5.http.protocol.HttpCoreContext import org.apache.hc.core5.util.Timeout import org.tomlj.Toml import org.xml.sax.InputSource import java.io.FileReader import java.io.StringReader import java.net.URI import javax.xml.parsers.DocumentBuilderFactory import javax.xml.xpath.XPathConstants import javax.xml.xpath.XPathExpression import javax.xml.xpath.XPathFactory val coreContext: HttpCoreContext = HttpCoreContext.create() val httpRequester: HttpRequester = RequesterBootstrap.bootstrap().create() val timeout: Timeout = Timeout.ofSeconds(5) val mavenTarget: HttpHost = HttpHost.create(URI("https://repo1.maven.org")) val dbf: DocumentBuilderFactory = DocumentBuilderFactory.newInstance() val xpathFactory: XPathFactory = XPathFactory.newInstance() val releasePath = xpathFactory.newXPath().compile("//metadata//versioning//release")!! main() fun main() { val toml = FileReader("./libs.versions.toml").use { file -> Toml.parse(file) } val versions = toml.getTable("versions")!!.toMap() val libraries = toml.getTable("libraries")!! val (currentVersionRefs, missingVersions) = libraries.keySet().associate { val module = libraries.getString("$it.module")!! val versionRef = libraries.getString("$it.version.ref") module to versionRef }.entries.partition { it.value != null } val currentVersions = currentVersionRefs.associate { it.key to versions.getValue(it.value) } val latestVersions = currentVersions.mapValues { (packageName, _) -> getLatestVersion(packageName, mavenTarget, httpRequester, timeout, coreContext, dbf, releasePath) } // Compare current to latest version and print if different currentVersions.toSortedMap().forEach { (packageName, currentVersion) -> val latestVersion = latestVersions.getValue(packageName) if (currentVersion != latestVersion) { println("$packageName: $currentVersion -> $latestVersion") } } if (missingVersions.isNotEmpty()) { println("\n--- Missing version.ref - Need to check manually ---\n") println(missingVersions.map { it.key }.sortedBy { it }.joinToString("\n")) } } fun getLatestVersion(packageName: String, mavenTarget: HttpHost?, httpRequester: HttpRequester, timeout: Timeout?, coreContext: HttpCoreContext?, dbf: DocumentBuilderFactory, releasePath: XPathExpression): String { val (groupId, artifactId) = packageName.split(":") val path = "${groupId.replace(".", "/")}/$artifactId" val request = ClassicRequestBuilder.get() .setHttpHost(mavenTarget).setPath("/maven2/$path/maven-metadata.xml").build() val xmlString = httpRequester.execute(mavenTarget, request, timeout, coreContext).use { response -> if (response.code != 200) return "Could not find value in Maven Central. Need to check manually." EntityUtils.toString(response.entity) } val db = dbf.newDocumentBuilder() val document = db.parse(InputSource(StringReader(xmlString))) return releasePath.evaluate(document, XPathConstants.STRING).toString() }
Notes:
- Third party libraries that were used: https://github.com/tomlj/tomlj & https://github.com/apache/httpcomponents-client
- There are a few cases in the toml’s
libraries
section where we still manage dependencies via thedependencyManagement
section of thebuild.gradle.kts
file and the library’s entry does not have aversion.ref
property. These are not handled by the script and appear in a separate “Missing version.ref” section. - There are a few libraries that are not hosted in Maven Central’s repo but rather in Jitpack’s repo – These are also not handled and a “Could not find value in Maven Central” message is printed instead.
Running this script as
./check_latest_versions.main.kts > latest_available_upgrades.txt
we get the following output:
com.amazonaws:aws-java-sdk-core: 1.12.311 -> 1.12.326
com.amazonaws:aws-java-sdk-s3: 1.12.311 -> 1.12.326
com.amazonaws:aws-java-sdk-sns: 1.12.311 -> 1.12.326
com.amazonaws:aws-java-sdk-sqs: 1.12.311 -> 1.12.326
We can see immediately that our AWS lib’s version is not up to date. After committing latest_available_upgrades.txt
, next time we run this script, we will also know when AWS publishes a new version of the libraries.
Summary
As you can see, this script automates the chore of checking for third party updates and reduces it to running a single command. It doesn’t create a PR as Dependabot does, but for most cases, it gets us most of the way towards quickly identifying which dependencies require an upgrade.
Nice work. I can relate the readability considerations. Indeed, debugging a shell script is not easy, and therefore converting to a Kotlin script sounds like the better path to go. I would take Python, but that’s just me 🙂
Keep it up!
That’s only because you haven’t tried Kotlin :-).
But more seriously: The whole backend is written in Kotlin, so it just makes sense to use Kotlin script which will be understood by any BE engineer. There are cases such as data processing where Python is the obvious choice, but when it comes to build-like scripts, I think both languages do a good job and there’s no real reason to prefer Python.
BTW, to make the script really impactful, the follow up versions of the script also take an “owners” toml as input and then break down the upgrades by team + mojor/minor/patch upgrades like this:
“`
###############
### The A Team ###
###############
[owned libraries]
com.googlecode.libphonenumber:libphonenumber
software.amazon.awssdk:sts
[major]
None
[minor]
software.amazon.awssdk:sts: 2.17.282 -> 2.18.4 [2022-10-26]
“`
You can pretty much copy-paste this section into a ticket (or even automate the ticket in the future), making it very easy to run this once a month / quarter. At that point, you’ve reduced the upgrade management overhead to one hour per quarter which means that it is now doable. If you make it easy for people to do the right thing, they will end up doing it 🙂