Knowing what your dependencies are doing

The problem

A Dependabot pull request arrives on a Monday. Its title says chore(deps): bump express from 4.18.2 to 4.19.2. Its diff is 1,100 lines of package-lock.json.

The reviewer has three options and all are bad. Read the lockfile diff — a serialisation format, not a review artifact, which will take forty minutes to produce a conclusion that is 90% "hashes changed". Trust the title, which describes one package out of however many actually moved. Or approve because CI is green, which is what happens.

The same team also has forty repositories, Dependabot enabled, and 400 open pull requests. Each is individually reasonable; collectively they are a wall, and the human response to a wall of undifferentiated work is to stop looking at it. Nothing in the pile indicates which four matter. Neither tool below ranks a queue across repositories. What they offer is per pull request: a report that says what changed, and opt-in gates that turn red only on the pull requests a human needs to read.

And underneath both, a ranking problem that neither scanning nor bumping addresses. Consider two packages in the same manifest:

• Package A is 4.0 libyears behind. Its maintainers ship monthly, the last commit was Tuesday, six people have commit rights, and the upgrade path is documented across three major versions.
• Package B is 0.2 libyears behind. Its sole maintainer's last commit was fourteen months ago, the repository is archived, and the issue tracker has forty open reports and no responses.

A drift-only tool ranks A as the problem, and it is right about the arithmetic. The ranking still misses the point. Upgrading A is a scheduled afternoon. B is a dependency with no future, and when its first CVE lands there will be no patch, because there is nobody to write one. The work is a replacement, and replacements take weeks of planning that has not started.

The two problems are the same problem at two zoom levels: a change you are looking at right now, and a portfolio you are not looking at at all.

The method

Explaining a change: sbom-diff

Rank by consequence. The report's sections run from most to least alarming: renames and major-version jumps first, then minor and patch updates, additions and removals, with licence changes and VEX deltas in their own flagged sections. A report in package-name order treats openssl 3.0.1 → 4.0.0 and left-pad 1.3.0 → 1.3.1 as equals, and the reader's attention is the scarce resource being spent.

Components are matched on package URL identity, so a package that changes its reported name between two SBOMs is one rename rather than an addition plus a removal. Getting this wrong inflates both counts and produces the noise that teaches people to stop reading the report.

A downgrade stays in its semver bucket (2.0.0 → 1.9.0 is still a major change) but is flagged (downgrade) in the report and counted in the headline, because an upgrade is usually intentional and a downgrade usually is not. A silent rollback is one of the few things in a lockfile diff that is nearly always worth a question.

Licences are compared independently of versions. A package can relicense without a version bump, and that event is invisible to any comparison keyed on version alone. --deny-licenses 'AGPL-3.0,GPL-3.0' turns the judgement into a gate once the team has made it.

Say which changes to read. The diff says what changed; --review says which of the changes deserve a human. Every added or updated component is scored against ten independent signals: an OSV advisory new to this version, an install script that was not there before, lost provenance, a change in who can publish, more than one major version skipped, a brand-new package, a release younger than --young-days, a licence change, a fallen OpenSSF Scorecard, and transitive fan-out. Points put each component in one of three tiers (read this, glance, routine), and the report ends with the scoring rule and every signal that did not run, with the reason. A signal that cannot be answered for an ecosystem (npm lifecycle scripts have no equivalent on PyPI) is reported as not checkable, never as clean. The cost is the network: registry, OSV and deps.dev calls, cached on disk for 24 hours, spaced per host, and capped by --max-requests (default 300). --offline scores from the two SBOMs alone.

Decline to infer what the document does not support. Direct versus transitive is read from the dependency graph and from nothing else; a document without one reports every component as transitive, and the README says so explicitly before recommending --max-added-transitive as a gate.

Nothing fails by default. Every gate is opt-in: --fail-on {any,major,license}, --max-added, --max-added-transitive, --fail-on-downgrade, --fail-on-license-change, --deny-licenses and --fail-on-tier {read,glance,routine}. A tool that breaks the build the day it is installed gets uninstalled the same day, and the adoption path that works is: read the report for a month, then decide which of its findings you are prepared to block on.

Ranking a portfolio: depwatch

Cross the two axes. Drift and viability are computed independently and plotted against each other:

                             viability
                             ▲
                    healthy  │  upgrade      behind?  → drift > --stale
                    ─────────┼────────►      fading?  → viability < --risky
                       watch │  replace      drift

Figure 1 — The quadrant. replace is far behind and has nobody left to catch up with; neither axis alone identifies it.

Drift is arithmetic on published dates. Viability is a weighted read of maintainer signals. The repository's own guidance is to gate on drift and treat viability as an argument, not a verdict: a tool that blurred the two would invite teams to fail builds on a heuristic about somebody's commit history, which is both unfair and unstable.

State the input's error bar in the output. Inputs are read in order of accuracy: an SBOM (resolved versions, every ecosystem at once), then a lockfile, then a manifest. A lockfile beside the manifest wins, because a version range gives only its floor: drift computed from a manifest is an upper bound, drift from a lockfile is a measurement. A note on stderr says which input was read, and the footer names it again, so the number is never read as more precise than it is.

Missing viability signals are dropped and the remaining weights renormalised, so a package with only a release history is not penalised for data that could not be fetched. A dependency whose registry could not be reached is marked degraded and excluded from the counts: unknown is not unhealthy, and conflating them makes every network hiccup look like a supply-chain finding.

Make the expensive tier opt-in. Pulse and cadence fall out of the version timeline already fetched for drift, so the cheap tier costs nothing extra. --deep adds maintainer count, archived status, last commit and funding at one additional request per dependency, with GITHUB_TOKEN recommended. A tool that always fetched everything would be too slow to run in CI on a large manifest.

Record what was accepted, and detect two kinds of "worse". A baseline written with --write-accepted reports more drift than was signed off, and also a worse quadrant at the same drift. A dependency whose maintainer walked away since you accepted it has not moved on the drift axis at all, so a baseline keyed only on version numbers would miss it.

The headline results

• Drift is the ICSE 2015 libyear metric
• What the sbom-diff tests assert.
• What the depwatch tests assert.
• sbom-diff has zero runtime dependencies.

The full results, with the tables and the reasoning behind each figure, are in the PDF.