agentic-ai-llm

11 command-line packages · 10 measurement apps · 20 products

A portfolio of small tools that each answer one question about code or data, ten apps that each measure one thing about coder models, and twenty multi-agent operator consoles. Every projects/ package installs with zero dependencies.

View on GitHub → (code private)

Interface

The command line
The command line
The run console
The run console

Real measurements

How much wrong code does a three-assert benchmark accept? Break each reference solution in one place, run the problem's own tests, and search every survivor for an input on which it and the original disagree. A survivor with such a witness is a program the benchmark marked correct and is not.

MeasurementMBPP (full)MBPP (hand-verified)
Problems whose reference passes its own tests971413
Single-point mutants run4,0551,553
Mutants that still passed all three asserts731 (18.0%)257 (16.5%)
Of those, provably different programs362 (49.5% of survivors)121 (47.1% of survivors)
Problems accepting at least one broken version326 (33.6%)117 (28.3%)
Killed by a crash rather than a failed assert14.2%13.5%

Reproduced from scratch for this page: the counts match the summary committed in the repository exactly, on both splits. The whole measurement loads no model and runs in about 36 minutes on one CPU.

a second app, with a labelled split and no model at all

{
 "app": "vulnerability baseline",
 "dataset": "Devign test split",
 "question": "can a model beat answering safe every time?"
}

what came back

{
 "split": "test",
 "rows": 2732,
 "safe": 1477,
 "vulnerable": 1255,
 "always_safe_baseline": 0.5406,
 "baseline_with_conflicting_labels_removed": 0.5407,
 "shift_in_points": 0.003,
 "rows_dropped_as_conflicting": 2
}

The majority-class score is computed first, so a model score has something to be read against. Removing the rows with conflicting labels moves it by 0.003 - which is how you know the baseline is not an artefact of the labels.

Eight of the ten measurement apps need a GPU and two model sizes, so their numbers are not shown here: nothing offline can check them. These two can be re-run on any CPU, which is why they are the two on this page.

What it does

repo-cartographer
Map an unfamiliar codebase from its AST — no embeddings, no model
test-smith
Mutation testing from the standard library: does the suite catch the change?
review-bot
Propose findings, then try to disprove each; report only survivors
migration-pilot
Modernise Python where the rewrite is provably equivalent, refuse where it is not
release-captain
Release readiness from diff statistics, not opinion
db-surgeon
Prove a migration's rollback on a throwaway copy before trusting it
compliance-auditor
Stated policy against collected evidence; no inferred compliance
csv-analyst
Compute only what validates, and never narrate an uncomputed number
log-detective
Extract log templates, and report what the extraction destroyed
contract-reader
Read a licence and cite the character span behind every claim
study-tutor
Spaced repetition where the scheduler is arithmetic

What it will not do

Kafka, Redis, Postgres and Ollama adapters all ship and no product binds one: every runtime constructs the in-memory bus and store, which is why twenty products run end to end with no broker, no database and no container.

Stack