Initial Builder

README.md

@@ -1,20 +1,45 @@
-# vuln-builder – CVE-Referenzimplementierung
+# vuln-builder – SQLite Starter (v1)
 
-Erstellt `master.db.zst`, `manifest.json` und (wenn `--previous` existiert) `delta.jsonl.zst` aus dem Repository `vuln-cve`. Das SQLite-Schema hält **CVE-ID, Status, Aktualisierungsdatum und verlustfrei die originale CNA-affected-Liste** vor. Es ist ein kompakter **CVE-Index**, **kein fertiger Scanner**: CNA-affected-Angaben sind nicht allein ausreichend für genaue CPE-, PURL-, Vendor- oder Distro-Versionsabgleiche; diese benötigen quellenspezifische Parser und AND/OR/NOT-Bedingungen.
+First functional, **conservative** builder for the Gitea organization `VULN`.
 
-## Lokal
+## Scope
 
-```bash
-python3 -m pip install -r requirements.txt
-python3 scripts/build.py --source ../vuln-cve/data --output dist
-# Später mit vorheriger Datei für Delta:
-python3 scripts/build.py --source ../vuln-cve/data --previous /pfad/alte/master.db --output dist
-# Delta auf vorhandene Kopie anwenden:
-python3 scripts/apply_delta.py --database /pfad/alte/master.db --delta dist/delta.jsonl.zst
+- Clones/fetches all named repositories in a persistent Docker volume.
+- Keeps the source Git commit for **every** mirror in `sources`.
+- Indexes **CVEProject (`cveMetadata`)**, **NVD (`cve`)** and **OSV (`id`/`affected`)** into SQLite.
+- Saves OSV package/version range events without guessing at version comparisons.
+- Saves only simple non-negated NVD CPE disjunctions; complex AND/NOT expressions are **not** converted into potentially incorrect matches.
+- Other vendors and distro data are **not yet normalized**. This is not yet a complete Trivy-compatible scanner DB.
+- Writes `master.db` (local staging), compressed `master.db.zst`, and on subsequent builds `delta-OLD-to-NEW.jsonl.zst`, plus `manifest.json`.
+- **No automated release publishing yet**: files remain on the Docker host volume at `/vuln-cache/vuln-builder/releases`.
+
+## Install
+
+1. Create empty `VULN/vuln-builder` in Gitea, then push this folder to branch `main`.
+2. Ensure Docker runner with label `vulnerability` is online and the named volume `gitea-vuln-cache` is mounted in **job** containers at `/vuln-cache`.
+3. Configure Gitea Actions read permission for the Gitea token across all `VULN` repositories (a job token may be scoped to a single repo; if cross-repo reads fail, create a separate read-only organization credential named `VULN_READ_TOKEN`, and change only the workflow's secret reference). Do not log credentials.
+4. Start `.gitea/workflows/build.yml`. Cron: daily 03:35 UTC. Initial download can be *very* large. Make sure all source mirrors finished their first import.
+
+## Manual release inspection
+
+On Docker host:
+
+```sh
+docker run --rm -v gitea-vuln-cache:/vuln-cache:ro alpine ls -lh /vuln-cache/vuln-builder/releases
 ```
 
-## GitHub Actions
+## Safety and limitations
 
-Die Build-Action checkt `vuln-cve` aus und lädt die vorherige veröffentlichte `master.db.zst` per `gh release download`. Sie erstellt eine neue GitHub Release mit Master und ggf. Delta. `VULN_READ_TOKEN` muss Leserechte auf `vuln-cve` haben (Fine-grained PAT oder GitHub-App-Token); der `GITHUB_TOKEN` des Builder-Repos braucht `contents: write`. Private Quellen, Branch-Schutzregeln und Self-hosted-Runner sind entsprechend einzurichten.
+- Input source repositories need valid Git checkout and the three core sources must exist.
+- A successful first-build **does not** imply all feeds are fully imported; check their source-specific completion markers before starting.
+- SQLite v1 is an index, not a final vulnerability scanner: no Debian/RPM version comparison, no complete CPE boolean-expression evaluator, no CSAF product-tree traversal yet.
+- Delta JSONL updates have `{table, op, key}` for deletion and `{table, op, row}` for upserts. A client must verify the starting version, apply within one SQLite transaction with foreign key constraints managed correctly, and only mark the new version after integrity checks. This package does **not** yet implement that client.
+- The source data can require many GB in the persistent Git cache. Avoid resetting/deleting the volume to save time.
+- `master.db` remains uncompressed on the server for diffing against the next build.
+- The master artifact and manifest are replaced one after another; consumers must only use `manifest.json` after it is written and verify the published SHA-256.
 
-**Grenzen für den Produktivbetrieb:** Die GitHub-Release-Dateigrößen und Repo-Quotas müssen überprüft werden. Der Build erstellt bisher nur einen CVE-Datensatz-Index; NVD/OSV/Hersteller/KEV/EPSS-Adapter sowie die Zusammenführung von Advisories und Versionsbereichen sind noch nicht implementiert. Deltas müssen sequenziell und unverändert veröffentlicht werden; GitHub `latest` ersetzt keine versionierte Delta-Historie. Releases dürfen nicht überschrieben werden. Für größere Datenmengen sind ein persistenter Runner, inkrementelle Builds und Release-Retention-Strategie sinnvoll.
+## Tests
+
+```sh
+python3 -m unittest discover -s tests -v
+```