vuln-builder – SQLite Starter (v1)

First functional, conservative builder for the Gitea organization VULN.

Scope

• Clones/fetches all named repositories in a persistent Docker volume.
• Keeps the source Git commit for every mirror in sources.
• Indexes CVEProject (cveMetadata), NVD (cve) and OSV (id/affected) into SQLite.
• Saves OSV package/version range events without guessing at version comparisons.
• Saves only simple non-negated NVD CPE disjunctions; complex AND/NOT expressions are not converted into potentially incorrect matches.
• Other vendors and distro data are not yet normalized. This is not yet a complete Trivy-compatible scanner DB.
• Writes master.db (local staging), compressed master.db.zst, and on subsequent builds delta-OLD-to-NEW.jsonl.zst, plus manifest.json.
• No automated release publishing yet: files remain on the Docker host volume at /vuln-cache/vuln-builder/releases.

Install

1. Create empty VULN/vuln-builder in Gitea, then push this folder to branch main.
2. Ensure Docker runner with label vulnerability is online and the named volume gitea-vuln-cache is mounted in job containers at /vuln-cache.
3. Configure Gitea Actions read permission for the Gitea token across all VULN repositories (a job token may be scoped to a single repo; if cross-repo reads fail, create a separate read-only organization credential named VULN_READ_TOKEN, and change only the workflow's secret reference). Do not log credentials.
4. Start .gitea/workflows/build.yml. Cron: daily 03:35 UTC. Initial download can be very large. Make sure all source mirrors finished their first import.

Manual release inspection

On Docker host:

docker run --rm -v gitea-vuln-cache:/vuln-cache:ro alpine ls -lh /vuln-cache/vuln-builder/releases

Safety and limitations

• Input source repositories need valid Git checkout and the three core sources must exist.
• A successful first-build does not imply all feeds are fully imported; check their source-specific completion markers before starting.
• SQLite v1 is an index, not a final vulnerability scanner: no Debian/RPM version comparison, no complete CPE boolean-expression evaluator, no CSAF product-tree traversal yet.
• Delta JSONL updates have {table, op, key} for deletion and {table, op, row} for upserts. A client must verify the starting version, apply within one SQLite transaction with foreign key constraints managed correctly, and only mark the new version after integrity checks. This package does not yet implement that client.
• The source data can require many GB in the persistent Git cache. Avoid resetting/deleting the volume to save time.
• master.db remains uncompressed on the server for diffing against the next build.
• The master artifact and manifest are replaced one after another; consumers must only use manifest.json after it is written and verify the published SHA-256.

Tests

python3 -m unittest discover -s tests -v