31 lines
2.5 KiB
Markdown
31 lines
2.5 KiB
Markdown
# vuln-cpe – NVD Official CPE Dictionary + CPE Match Criteria
|
||
|
||
Standalone raw-data mirror for Gitea Actions, using a persistent Git checkout on the existing `vuln-runner` host.
|
||
|
||
## Sources
|
||
- `https://services.nvd.nist.gov/rest/json/cpes/2.0` — product CPE dictionary; max 10,000/page.
|
||
- `https://services.nvd.nist.gov/rest/json/cpematch/2.0` — match criteria and version ranges; max 500/page.
|
||
|
||
The mirror stores one full API record wrapper per UUID at `data/cpes/<prefix>/<uuid>.json` and `data/cpematch/<prefix>/<uuid>.json`. No flattening or interpretation is done. Deprecated records are retained as such. Source entries removed without a modification event require a periodically scheduled full reconciliation; this implementation does **not** delete absent entries automatically.
|
||
|
||
## Gitea setup
|
||
1. Create empty `VULN/vuln-cpe` repo and push these files to `main`.
|
||
2. Create `/var/lib/act_runner/cache/vuln-cpe` owned by `act_runner` (`0700`).
|
||
3. Runner label `vuln-runner`; normal Gitea `secrets.GITEA_TOKEN` needs write permission.
|
||
4. Put the same `NVD_API_KEY` as for `vuln-nvd` in the org Actions Secrets (recommended). Without key the requests are slower.
|
||
5. Manually start the workflow, then it runs every four hours (UTC cron).
|
||
|
||
## Bootstrap and updates
|
||
Both feeds are separate, checkpointed and capped at 20 pages per run. The initial full import takes multiple runs, committing progress after every run; it does **not** mean all data are available after the first successful action. Watch `state/cpes.json` and `state/cpematch.json`: both must show `"mode":"delta"` before declaring the initial mirror complete.
|
||
|
||
After a full import, changes since the bootstrap start are replayed in 110-day windows with a 2-minute safety lag, to avoid gaps. Individual pages have a durable checkpoint; a crash during a page causes harmless replay on the next run. The HTTP API window is limited to 120 days. No data is deleted merely because it wasn't in a page.
|
||
|
||
## Test
|
||
`python3 -m unittest discover -s tests -v`
|
||
|
||
## Important limitations
|
||
- The NVD API imposes request limits; repeated runs for a large bootstrap can take days. Do not run multiple jobs in parallel with the same API key.
|
||
- A shallow Git checkout with thousands of records still requires disk space and pushes can be large.
|
||
- NVD CVE applicability tree (`AND`, `OR`, negate) remains in `vuln-nvd`; this repository stores only match criteria and CPE dictionary, not the applicability tree itself.
|
||
- A standalone full reconciliation/retirement audit should be scheduled periodically. NVD deprecations are retained with their status.
|