Initial Builder

.github/workflows/build.yml

@@ -0,0 +1,50 @@
+name: Build vulnerability SQLite
+on:
+  schedule:
+    - cron: '40 */6 * * *'
+  workflow_dispatch:
+permissions:
+  contents: write
+concurrency:
+  group: sqlite-publish
+  cancel-in-progress: false
+jobs:
+  publish:
+    runs-on: [self-hosted, linux, x64, vulnerability]
+    timeout-minutes: 240
+    steps:
+      - name: Checkout builder
+        uses: actions/checkout@v4
+      - name: Checkout CVE raw-data repo
+        uses: actions/checkout@v4
+        with:
+          repository: ${{ github.repository_owner }}/vuln-cve
+          path: sources/vuln-cve
+          token: ${{ secrets.VULN_READ_TOKEN }}
+          persist-credentials: false
+          fetch-depth: 1
+      - name: Verify compression tool
+        run: zstd --version
+      - name: Download latest released baseline, if available
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: |
+          mkdir -p prior
+          gh release download --repo "$GITHUB_REPOSITORY" --pattern master.db.zst --dir prior || echo 'No prior release; baseline only'
+          if test -s prior/master.db.zst; then
+            python3 scripts/unpack.py prior/master.db.zst prior/master.db
+          fi
+      - name: Build master and delta
+        run: |
+          REV=$(git -C sources/vuln-cve rev-parse HEAD)
+          if test -f prior/master.db; then
+            python3 scripts/build.py --source sources/vuln-cve/data --source-revision "$REV" --previous prior/master.db
+          else
+            python3 scripts/build.py --source sources/vuln-cve/data --source-revision "$REV"
+          fi
+      - name: Publish release
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: |
+          TAG="db-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
+          gh release create "$TAG" dist/*.zst dist/manifest.json --title "$TAG" --notes 'Automated vulnerability DB snapshot and incremental delta'

.gitignore

@@ -0,0 +1,4 @@
+dist/
+prior/
+sources/
+__pycache__/

README.md

@@ -0,0 +1,20 @@
+# vuln-builder – CVE-Referenzimplementierung
+
+Erstellt `master.db.zst`, `manifest.json` und (wenn `--previous` existiert) `delta.jsonl.zst` aus dem Repository `vuln-cve`. Das SQLite-Schema hält **CVE-ID, Status, Aktualisierungsdatum und verlustfrei die originale CNA-affected-Liste** vor. Es ist ein kompakter **CVE-Index**, **kein fertiger Scanner**: CNA-affected-Angaben sind nicht allein ausreichend für genaue CPE-, PURL-, Vendor- oder Distro-Versionsabgleiche; diese benötigen quellenspezifische Parser und AND/OR/NOT-Bedingungen.
+
+## Lokal
+
+```bash
+python3 -m pip install -r requirements.txt
+python3 scripts/build.py --source ../vuln-cve/data --output dist
+# Später mit vorheriger Datei für Delta:
+python3 scripts/build.py --source ../vuln-cve/data --previous /pfad/alte/master.db --output dist
+# Delta auf vorhandene Kopie anwenden:
+python3 scripts/apply_delta.py --database /pfad/alte/master.db --delta dist/delta.jsonl.zst
+```
+
+## GitHub Actions
+
+Die Build-Action checkt `vuln-cve` aus und lädt die vorherige veröffentlichte `master.db.zst` per `gh release download`. Sie erstellt eine neue GitHub Release mit Master und ggf. Delta. `VULN_READ_TOKEN` muss Leserechte auf `vuln-cve` haben (Fine-grained PAT oder GitHub-App-Token); der `GITHUB_TOKEN` des Builder-Repos braucht `contents: write`. Private Quellen, Branch-Schutzregeln und Self-hosted-Runner sind entsprechend einzurichten.
+
+**Grenzen für den Produktivbetrieb:** Die GitHub-Release-Dateigrößen und Repo-Quotas müssen überprüft werden. Der Build erstellt bisher nur einen CVE-Datensatz-Index; NVD/OSV/Hersteller/KEV/EPSS-Adapter sowie die Zusammenführung von Advisories und Versionsbereichen sind noch nicht implementiert. Deltas müssen sequenziell und unverändert veröffentlicht werden; GitHub `latest` ersetzt keine versionierte Delta-Historie. Releases dürfen nicht überschrieben werden. Für größere Datenmengen sind ein persistenter Runner, inkrementelle Builds und Release-Retention-Strategie sinnvoll.

requirements.txt

@@ -0,0 +1 @@
+# Keine Python-Fremdpakete erforderlich. Systempaket: zstd

scripts/apply_delta.py

@@ -0,0 +1,65 @@
+#!/usr/bin/env python3
+"""Apply delta atomically, rejecting gaps or mismatched base database."""
+import argparse
+import base64
+import hashlib
+import json
+import os
+from pathlib import Path
+import shutil
+import sqlite3
+import tempfile
+import subprocess
+
+def fingerprint(conn):
+    h = hashlib.sha256()
+    for cve_id, row_hash in conn.execute('SELECT cve_id,row_hash FROM cve ORDER BY cve_id'):
+        h.update(cve_id.encode() + b'\x00' + row_hash.encode() + b'\n')
+    return h.hexdigest()
+
+def apply(db, delta):
+    db = Path(db)
+    with tempfile.TemporaryDirectory(dir=db.parent) as tmp:
+        work = Path(tmp) / 'master.db'
+        shutil.copy2(db, work)
+        conn = sqlite3.connect(work)
+        proc = subprocess.Popen(['zstd', '-q', '-dc', str(delta)], stdout=subprocess.PIPE, text=True)
+        try:
+            lines = proc.stdout
+            header = json.loads(next(lines))
+            if header.get('format') != 1 or header.get('type') != 'header':
+                raise ValueError('Invalid delta header')
+            if fingerprint(conn) != header['base']:
+                raise ValueError('Delta base mismatch: full master or missing prior delta required')
+            conn.execute('BEGIN IMMEDIATE')
+            for line in lines:
+                op = json.loads(line)
+                if op['type'] == 'delete':
+                    conn.execute('DELETE FROM cve WHERE cve_id=?', (op['id'],))
+                elif op['type'] == 'upsert':
+                    blob = base64.b64decode(op['affected_b64'], validate=True)
+                    expected = hashlib.sha256(json.dumps([op['id'], op['state'], op['updated']], separators=(',', ':')).encode() + blob).hexdigest()
+                    if expected != op['hash']:
+                        raise ValueError('Delta row hash mismatch')
+                    conn.execute('INSERT OR REPLACE INTO cve VALUES (?,?,?,?,?)', (op['id'], op['state'], op['updated'], blob, op['hash']))
+                else:
+                    raise ValueError('Unknown delta operation')
+            if proc.wait() != 0:
+                raise ValueError('zstd decompression failed')
+            if fingerprint(conn) != header['target']:
+                raise ValueError('Delta target mismatch')
+            conn.commit()
+        finally:
+            proc.stdout.close()
+            if proc.poll() is None:
+                proc.kill()
+                proc.wait()
+        conn.close()
+        os.replace(work, db)
+
+if __name__ == '__main__':
+    p = argparse.ArgumentParser()
+    p.add_argument('--database', required=True, type=Path)
+    p.add_argument('--delta', required=True, type=Path)
+    a = p.parse_args()
+    apply(a.database, a.delta)

scripts/build.py

@@ -0,0 +1,113 @@
+#!/usr/bin/env python3
+"""Build compact CVE SQLite plus optionally incremental row-level delta (CVE slice)."""
+import argparse
+import base64
+import datetime as dt
+import hashlib
+import json
+import sqlite3
+import zlib
+from pathlib import Path
+import subprocess
+
+SCHEMA = '''CREATE TABLE cve (
+    cve_id TEXT PRIMARY KEY,
+    state TEXT NOT NULL,
+    date_updated TEXT,
+    affected BLOB NOT NULL,
+    row_hash TEXT NOT NULL
+) WITHOUT ROWID;
+CREATE INDEX idx_cve_state ON cve(state);'''
+
+def encode_record(doc):
+    meta = doc['cveMetadata']
+    cna = doc.get('containers', {}).get('cna', {})
+    # Preserve entire CNA affected array as structured JSON. Do NOT infer ranges.
+    affected = json.dumps(cna.get('affected', []), sort_keys=True, separators=(',', ':'), ensure_ascii=False).encode('utf-8')
+    blob = zlib.compress(affected, level=9)
+    row = (meta['cveId'], meta.get('state', 'UNKNOWN'), meta.get('dateUpdated'), blob)
+    digest = hashlib.sha256(json.dumps([row[0], row[1], row[2]], separators=(',', ':')).encode() + blob).hexdigest()
+    return (*row, digest)
+
+def fingerprint(conn):
+    h = hashlib.sha256()
+    for cve_id, row_hash in conn.execute('SELECT cve_id,row_hash FROM cve ORDER BY cve_id'):
+        h.update(cve_id.encode() + b'\x00' + row_hash.encode() + b'\n')
+    return h.hexdigest()
+
+def export_zstd(path, output):
+    with output.open('wb') as dst:
+        subprocess.run(['zstd', '-q', '-12', '-c', str(path)], stdout=dst, check=True)
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument('--source', type=Path, required=True, help='vuln-cve/data')
+    ap.add_argument('--output', type=Path, default=Path('dist'))
+    ap.add_argument('--previous', type=Path, help='previous SQLite master.db')
+    ap.add_argument('--source-revision', default='unknown')
+    args = ap.parse_args()
+    args.output.mkdir(parents=True, exist_ok=True)
+    db = args.output / 'master.db'
+    db.unlink(missing_ok=True)
+    current = sqlite3.connect(db)
+    current.executescript(SCHEMA)
+    count = 0
+    for path in sorted(args.source.rglob('CVE-*.json')):
+        with path.open(encoding='utf-8') as fh:
+            doc = json.load(fh)
+        if doc.get('cveMetadata', {}).get('cveId') != path.stem:
+            raise ValueError(f'CVE metadata mismatch at {path}')
+        current.execute('INSERT INTO cve VALUES (?,?,?,?,?)', encode_record(doc))
+        count += 1
+        if count % 5000 == 0:
+            current.commit()
+    if count == 0:
+        raise RuntimeError('No CVE records, refusing empty release')
+    current.commit()
+    current.execute('VACUUM')
+    current.commit()
+    current_hash = fingerprint(current)
+    previous_hash = None
+    changed = removed = 0
+    if args.previous:
+        prev = sqlite3.connect(f'file:{args.previous.resolve()}?mode=ro', uri=True)
+        previous_hash = fingerprint(prev)
+        old = {row[0]: row[1] for row in prev.execute('SELECT cve_id,row_hash FROM cve')}
+        delta_file = args.output / 'delta.jsonl.zst'
+        plain_delta = args.output / 'delta.jsonl'
+        with plain_delta.open('w', encoding='utf-8') as out:
+            header = {'type': 'header', 'format': 1, 'base': previous_hash, 'target': current_hash}
+            out.write(json.dumps(header) + '\n')
+            for id, state, date, blob, row_hash in current.execute('SELECT * FROM cve ORDER BY cve_id'):
+                old_hash = old.pop(id, None)
+                if old_hash != row_hash:
+                    change = {'type': 'upsert', 'id': id, 'state': state, 'updated': date, 'affected_b64': base64.b64encode(blob).decode(), 'hash': row_hash}
+                    out.write(json.dumps(change, separators=(',', ':')) + '\n')
+                    changed += 1
+            for id in sorted(old):
+                out.write(json.dumps({'type': 'delete', 'id': id}) + '\n')
+                removed += 1
+        export_zstd(plain_delta, delta_file)
+        plain_delta.unlink()
+        prev.close()
+    export_zstd(db, args.output / 'master.db.zst')
+    manifest = {
+        'format': 1,
+        'schema': 'cve-index-v1',
+        'built_at': dt.datetime.now(dt.timezone.utc).isoformat(),
+        'sources': {'cve': args.source_revision},
+        'records': count,
+        'content_sha256': current_hash,
+        'previous_content_sha256': previous_hash,
+        'upserts': changed,
+        'deletions': removed,
+        'artifacts': {},
+    }
+    for f in sorted(args.output.glob('*.zst')):
+        manifest['artifacts'][f.name] = {'sha256': hashlib.sha256(f.read_bytes()).hexdigest(), 'bytes': f.stat().st_size}
+    (args.output / 'manifest.json').write_text(json.dumps(manifest, indent=2) + '\n')
+    print(json.dumps(manifest, indent=2))
+    current.close()
+
+if __name__ == '__main__':
+    main()

scripts/unpack.py

@@ -0,0 +1,5 @@
+#!/usr/bin/env python3
+import subprocess
+import sys
+with open(sys.argv[2], 'wb') as dest:
+    subprocess.run(['zstd', '-q', '-dc', sys.argv[1]], stdout=dest, check=True)

sql/schema.sql

@@ -0,0 +1,9 @@
+-- CVE-Metadatenindex. Keine CPE-/Paket-Matching-Engine!
+CREATE TABLE cve (
+    cve_id TEXT PRIMARY KEY,
+    state TEXT NOT NULL,
+    date_updated TEXT,
+    affected BLOB NOT NULL, -- zlib-komprimierte JSON-CNA-affected-Liste
+    row_hash TEXT NOT NULL
+) WITHOUT ROWID;
+CREATE INDEX idx_cve_state ON cve(state);