Scoring a repository so the number means something

The problem

A team decides to measure code quality. Someone wires a scanner into CI. Six weeks later the dashboard says 94/100 and nobody trusts it, for a reason that takes a while to surface: the security scanner was never installed on the runner. It failed to start, the wrapper swallowed the error, and the gate recorded a pass. The repository has been scoring 94 for six weeks on the strength of checks that did not happen.

This is the characteristic failure of aggregate quality scoring, and it runs in both directions. A missing scanner props up a bad repository: nothing ran, nothing failed, the score is high. Or the opposite convention treats unavailable as failure, which punishes a clean Python repository for not having the Rust toolchain installed; teams respond by disabling gates until the number moves, which destroys the signal from the other side.

Two further failures make quality scores actively misleading.

Absent evidence scored as failure. A repository with no classes scores zero on cohesion. One with no git history scores zero on churn. The number drops for a reason that is not a quality problem, and the response (ignoring the score or gaming it) destroys the signal either way.

Uniform thresholds across languages. A 47-line function is unremarkable in Go and too long in Python. A tool applying one threshold marks idiomatic Go as bloated and idiomatic Python as fine, and its output is dismissed by whichever team it insults.

And underneath: a polyglot repository (a Go daemon, a Python analysis script, a TypeScript front end, some Terraform) needs four tools, four configurations and four report formats, and no single number covers it. So teams either score one language and call it the repository, or score nothing.

The method

Not-run is a third state, and it is not scored. Every gandalf gate reports pass, warn, fail, or not-run-with-a-reason. Not-run gates appear in the report with what was skipped and why, and their own line shows amber, but they are excluded from both the composite and the verdict (aggregate() counts only the gates that ran), so a missing scanner neither inflates nor deflates the number. When most gates land there, the run says so directly and points at the setup step instead of emitting a meaningless 100.

The cost of both designs is coverage. What gandalf checks depends on what is installed, and what gradebook checks is limited to what source and git log reveal; each score is only as wide as its list of exclusions.

Classify first, then run only what applies. The languages present decide which gates are relevant. A Python-only change never waits on the Go toolchain, and irrelevant gates are named as skipped rather than silently omitted.

Stdlib-only, and it installs nothing. gandalf itself has no dependencies. It runs other people's scanners if they are on PATH, or in the gandalf-tools image, which make tools builds rather than pulls, so the gate does not reach the network on its own initiative.

Policy lives outside the gates. A gate reports an outcome and a score; what turns those into a pass or fail lives in one place, driven by --fail-on and --min-score. Gates stay simple and the policy stays reviewable, which makes the gate set an extension point rather than a fork point.

Print what was not scored. gradebook applies the same discipline without any external tools:

not scored (weights redistributed): Cohesion (GRASP)

Calibrate per language, and print the calibration.

languages: go (22)  ·  calibrated for go: 60 lines, complexity 12, 5 params

Thresholds are weighted by how much of the repository each language is, and the calibration is printed next to the languages so the reader knows what standard was applied. A polyglot repository gets a blended threshold rather than one language's conventions imposed on the others.

Rank the fixes by points recovered. This converts a score into a plan:

Biggest wins:
  +2.0   Naming & intent — name things for what they are: a Manager or a Helper is a class nobody could describe
  +1.5   Dependency inversion — concentrate infrastructure behind a few adapters and inject it

A team with an afternoon spends it on the +2.0 item. Without the ranking, the same team spends it on whichever dimension it happens to care about.

Anchor findings at file:line. Both tools do this, and gandalf goes further: with --fix the correction cascades to the underlying tools, and on a pull request the same fixes arrive as GitHub suggestion blocks that commit in one click.

Suppress without hiding. .gandalf.toml rules and a baseline mark findings; they do not delete them. The suppressed count stays in the report, so a repository cannot quietly reach green by accumulating exceptions.

Gate on the trend. gradebook --baseline report.json --fail-on-drop fails CI when the score gets worse. Most existing repositories cannot clear an absolute bar on day one, and a gate that is red on the day it is installed teaches people to disable gates.

The headline results

• gandalf's composite is the plain mean of the sub-scores of the gates that ran
• What the tests assert.
• gradebook's model is published
• Both are read-only and execute nothing of yours.
• The LLM summary is optional.

The full results, with the tables and the reasoning behind each figure, are in the PDF.