46 lines
3.0 KiB
Markdown
46 lines
3.0 KiB
Markdown
# vuln-builder – SQLite Starter (v1)
|
||
|
||
First functional, **conservative** builder for the Gitea organization `VULN`.
|
||
|
||
## Scope
|
||
|
||
- Clones/fetches all named repositories in a persistent Docker volume.
|
||
- Keeps the source Git commit for **every** mirror in `sources`.
|
||
- Indexes **CVEProject (`cveMetadata`)**, **NVD (`cve`)** and **OSV (`id`/`affected`)** into SQLite.
|
||
- Saves OSV package/version range events without guessing at version comparisons.
|
||
- Saves only simple non-negated NVD CPE disjunctions; complex AND/NOT expressions are **not** converted into potentially incorrect matches.
|
||
- Other vendors and distro data are **not yet normalized**. This is not yet a complete Trivy-compatible scanner DB.
|
||
- Writes `master.db` (local staging), compressed `master.db.zst`, and on subsequent builds `delta-OLD-to-NEW.jsonl.zst`, plus `manifest.json`.
|
||
- **No automated release publishing yet**: files remain on the Docker host volume at `/vuln-cache/vuln-builder/releases`.
|
||
|
||
## Install
|
||
|
||
1. Create empty `VULN/vuln-builder` in Gitea, then push this folder to branch `main`.
|
||
2. Ensure Docker runner with label `vulnerability` is online and the named volume `gitea-vuln-cache` is mounted in **job** containers at `/vuln-cache`.
|
||
3. Configure Gitea Actions read permission for the Gitea token across all `VULN` repositories (a job token may be scoped to a single repo; if cross-repo reads fail, create a separate read-only organization credential named `VULN_READ_TOKEN`, and change only the workflow's secret reference). Do not log credentials.
|
||
4. Start `.gitea/workflows/build.yml`. Cron: daily 03:35 UTC. Initial download can be *very* large. Make sure all source mirrors finished their first import.
|
||
|
||
## Manual release inspection
|
||
|
||
On Docker host:
|
||
|
||
```sh
|
||
docker run --rm -v gitea-vuln-cache:/vuln-cache:ro alpine ls -lh /vuln-cache/vuln-builder/releases
|
||
```
|
||
|
||
## Safety and limitations
|
||
|
||
- Input source repositories need valid Git checkout and the three core sources must exist.
|
||
- A successful first-build **does not** imply all feeds are fully imported; check their source-specific completion markers before starting.
|
||
- SQLite v1 is an index, not a final vulnerability scanner: no Debian/RPM version comparison, no complete CPE boolean-expression evaluator, no CSAF product-tree traversal yet.
|
||
- Delta JSONL updates have `{table, op, key}` for deletion and `{table, op, row}` for upserts. A client must verify the starting version, apply within one SQLite transaction with foreign key constraints managed correctly, and only mark the new version after integrity checks. This package does **not** yet implement that client.
|
||
- The source data can require many GB in the persistent Git cache. Avoid resetting/deleting the volume to save time.
|
||
- `master.db` remains uncompressed on the server for diffing against the next build.
|
||
- The master artifact and manifest are replaced one after another; consumers must only use `manifest.json` after it is written and verify the published SHA-256.
|
||
|
||
## Tests
|
||
|
||
```sh
|
||
python3 -m unittest discover -s tests -v
|
||
```
|