feat(scan): add cdn provider detection pool#287
Conversation
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #287 +/- ##
=======================================
Coverage ? 65.24%
=======================================
Files ? 91
Lines ? 7982
Branches ? 0
=======================================
Hits ? 5208
Misses ? 2377
Partials ? 397 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
github-actions
Bot
added
scan
pr summary8 files changed (+721 -2)
|
vmfunc
left a comment
vmfunc
left a comment
There was a problem hiding this comment.
separate cdn pool so a cf-ray can't argmax over the real framework is the right call, and keeping every sig HeaderOnly does kill the body-badge fp. but the per-header scoping you're leaning on isn't real.
detectors/cdn.go: the Header: "Server"/Header: "Via" qualifiers do nothing, nothing reads a Signature.Header field. MatchSignatures runs through containsHeader (detector.go:126) which substring-matches the pattern against every header name and value, there's no target-header concept. so "cloudflare"/"cloudfront"/"vercel"/"netlify"/"fastly" fire on any header, not just the one the edge stamps.
concrete fp: an nginx origin with no cf-ray/x-amz-cf-id but a CSP or Link header referencing a cdn-hosted asset (cdnjs.cloudflare.com, a *.cloudfront.net img-src, vercel.app...). the brand word alone scores 0.4-0.5 -> sigmoid 0.73-0.88, past the 0.5 threshold -> you report "Amazon CloudFront" on a site that just hot-links a cloudfront bucket. referencing a cdn asset in a header is about as common as configs get.
akamai is the model here: only vendor header-name tokens (akamai-grn, x-akamai), no bare brand word, so it's clean. drop the brand-word-in-value sigs from the other five and lean on the vendor headers (cf-ray, x-fastly-request-id, x-amz-cf-id, x-vercel-id, x-nf-request-id), each clears threshold on its own at weight 1.0. or actually wire the Header field into containsHeader. right now the comment about sigs being "scoped to the specific edge-injected header" is writing a check the engine doesn't cash.
minor: body says one finding per provider but DetectCDN is single-argmax, so a cloudflare-in-front-of-vercel stack only reports one. not blocking, the fp is.
1a56925 to
6368637
Compare
|
the FP you sketched is exactly the right thing to be worried about here, and a Signature.Header does get read. MatchSignatures branches on it before falling if sig.HeaderOnly { that is internal/scan/frameworks/detector.go:112-118 on main, and it landed in I built your concrete case as a test: an nginx origin with no cf-ray or on the minor: agreed that DetectCDN being single-argmax means a |
There was a problem hiding this comment.
Pull request overview
This PR introduces a dedicated CDN/hosting provider detection pool that is isolated from the existing framework detector registry, preventing CDN matches from suppressing application framework detection. It also adds a new built-in scan module that fetches a target once and reports a CDN/hosting fingerprint from response headers.
Changes:
- Add a separate CDN detector registry and
DetectCDNAPI ininternal/scan/frameworks. - Add CDN provider detectors (Cloudflare/Fastly/Akamai/CloudFront/Vercel/Netlify) plus tests to pin true/false positives and registry isolation.
- Add a new built-in
CDNModuleand register it in the built-in module list, with module-level tests.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| internal/scan/frameworks/detectors/cdn.go | Adds CDN provider detector implementations and registers them into a dedicated CDN pool. |
| internal/scan/frameworks/detectors/cdn_test.go | Adds unit tests for CDN detector true/false positives and registration behavior. |
| internal/scan/frameworks/cdn.go | Introduces CDN registry, CDNResult, and DetectCDN argmax selection with thresholding. |
| internal/scan/frameworks/cdn_test.go | Adds integration-style tests ensuring CDN detection doesn’t suppress framework detection. |
| internal/scan/builtin/register.go | Registers the new CDNModule in the built-in module set. |
| internal/scan/builtin/cdn_module.go | Implements the CDN detection module that fetches a target and emits an info finding. |
| internal/scan/builtin/cdn_module_test.go | Adds tests verifying the module detects Cloudflare and produces no finding on plain origins. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| func (d *cloudflareDetector) Detect(body string, headers http.Header) (float32, string) { | ||
| base := fw.NewBaseDetector(d.Name(), d.Signatures()) | ||
| return sigmoidConfidence(base.MatchSignatures(body, headers)), "" | ||
| } |
| func (d *fastlyDetector) Detect(body string, headers http.Header) (float32, string) { | ||
| base := fw.NewBaseDetector(d.Name(), d.Signatures()) | ||
| return sigmoidConfidence(base.MatchSignatures(body, headers)), "" | ||
| } |
| func (d *akamaiDetector) Detect(body string, headers http.Header) (float32, string) { | ||
| base := fw.NewBaseDetector(d.Name(), d.Signatures()) | ||
| return sigmoidConfidence(base.MatchSignatures(body, headers)), "" | ||
| } |
| func (d *cloudfrontDetector) Detect(body string, headers http.Header) (float32, string) { | ||
| base := fw.NewBaseDetector(d.Name(), d.Signatures()) | ||
| return sigmoidConfidence(base.MatchSignatures(body, headers)), "" | ||
| } |
| func (d *vercelDetector) Detect(body string, headers http.Header) (float32, string) { | ||
| base := fw.NewBaseDetector(d.Name(), d.Signatures()) | ||
| return sigmoidConfidence(base.MatchSignatures(body, headers)), "" | ||
| } |
| func (d *netlifyDetector) Detect(body string, headers http.Header) (float32, string) { | ||
| base := fw.NewBaseDetector(d.Name(), d.Signatures()) | ||
| return sigmoidConfidence(base.MatchSignatures(body, headers)), "" | ||
| } |
| body, err := io.ReadAll(io.LimitReader(resp.Body, cdnMaxBodySize)) | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
|
|
||
| result := &modules.Result{ | ||
| ModuleID: m.Info().ID, | ||
| Target: target, | ||
| Findings: []modules.Finding{}, | ||
| } | ||
|
|
||
| cdn := frameworks.DetectCDN(string(body), resp.Header) |
|
following up on my last comment, because I owe you a correction. you were right the brand words (cloudflare, fastly, vercel...) are scoped with Header, which is {Pattern: "cf-ray", Weight: 0.6, HeaderOnly: true} no Header, so those did go through containsHeader and matched header values as Access-Control-Expose-Headers: CF-Ray, X-Amz-Cf-Id, X-Vercel-Id cdns themselves emit that header, so it is close to the worst case for this fixed by adding Signature.Presence: with Header set, the signature matches on covered by TestCDNIgnoresVendorHeaderNamesInValues, which fails before the |
|
CI on this branch is red for a reason unrelated to the change: main does not internal/modules/executor.go:197:57: undefined: MaxBodySize #374 fixes it. this branch builds and tests clean with that one-line fix applied |
detect cdn providers via a dedicated detection pool separate from the framework single-argmax reducer, so a cdn match cannot suppress a framework match. new cdn-detection module fetches once, runs frameworks.DetectCDN, and emits one info finding per detected provider.
the brand-word signatures carry Header: Server / Header: Via, and MatchSignatures routes those through headerValueContains rather than the all-headers containsHeader, so a scoped signature only sees the header the provider controls. nothing covered that, which makes the scoping look decorative on a read of containsHeader alone. add the case it protects: an nginx origin whose CSP and Link headers reference cdnjs.cloudflare.com, a cloudfront bucket, a vercel.app frame-ancestor and a netlify api, with no edge header anywhere. it must stay unreported. the companion table keeps every provider firing on headers a real edge stamps.
the vendor markers (cf-ray, x-amz-cf-id, x-vercel-id and friends) carried no Header, so MatchSignatures ran them through containsHeader, which substring matches every header name and every header value. an origin that merely names those headers in a value was read as sitting behind that edge: Access-Control-Expose-Headers: CF-Ray, X-Amz-Cf-Id, X-Vercel-Id -> "Cloudflare" at 0.9526 on a plain nginx origin that header is about the most common cors config there is, and cdns themselves emit it, so this was a live false positive rather than a corner. add Signature.Presence: with Header set, the signature matches on the header existing rather than on its value. that is the right primitive for an edge marker, whose value is an opaque request id with nothing to match. the brand words keep their existing Server/Via scoping.
263eeba to
af86050
Compare
detect cdn providers via a dedicated detection pool separate from the
framework single-argmax reducer, so a cdn match cannot suppress a
framework match. new cdn-detection module fetches once, runs
frameworks.DetectCDN, and emits one info finding per detected provider.
the brand-word signatures are scoped with
Header: Server/Header: Via, which MatchSignatures routes through headerValueContains rather than the all-headers containsHeader. a test pins the case that protects: an nginx origin whose CSP and Link headers reference cdnjs.cloudflare.com, a cloudfront bucket, a vercel.app frame-ancestor and a netlify api, with no edge header anywhere, stays unreported.