enhancement(remap): unify VRL source configuration - #25948
Conversation
🎉 All green!🧪 All tests passed 🔗 Commit SHA: 67d0872 | Docs | Datadog PR Page | Give us feedback! |
74f4e74 to
a2a0004
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 021ec10e8f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9287a95d99
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 872ba5dc18
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
872ba5d to
9b90bfc
Compare
The remap transform had three mutually exclusive source fields (source, file,
files) with no schema-level enforcement, so conflicts surfaced only at runtime
and the docs generator had to omit examples from the alternatives.
source is now a single field accepting an inline VRL string, a { type: vrl,
program: ... } object, or { type: disk, files: [...] } to load from disk. A
custom deserializer enforces that exactly one of source, file, or files is set
and warns on the deprecated top-level file/files, which remain functional.
source is optional at the schema layer so legacy file/files-only configs still
validate against the generated schema; the website minimal-example generator
carries a small override for that until the deprecated fields are removed.
9b90bfc to
ac23566
Compare
Summary
The
remaptransform had three mutually exclusive source fields (source,file,files)with no schema-level enforcement: supplying more than one was caught only at runtime with an
opaque error, and the docs generator had to strip
docs::examplesfrom the alternatives toavoid emitting invalid auto-generated configs. This consolidates everything under a single
sourcefield whose object form is a typed,AnyCondition-style enum, so the constraint isenforced at parse time and generated docs carry accurate examples again.
sourcestays theprimary field because nearly every remap config uses it.
Changes
sourcenow accepts an inline VRL string (unchanged, the common case) or atype-taggedobject:
{ type: vrl, program: "..." }for an inline program, or{ type: disk, files: [...] }to load from disk.
source,file, orfilesis set (also onthe compile path, for configs built directly in Rust) and preserves each field through
serialization so configs round-trip.
fileandfilesfields are deprecated: still functional, documented asdeprecated, and they emit a startup warning pointing to the new form.
Vector configuration
Inline (unchanged):
Inline, explicit form:
Load from disk:
Deprecated, still works:
How did you test this PR?
round-tripping, and end-to-end compile/execute for both
type: vrlandtype: disk(the diskcase also asserts file watching).
cargo run -- validateon configs for all forms, including deprecatedfile/files(validateand emit the warning) and a
source+fileconflict (fails at load withmust provide exactly one of source, file, or files).make validate-config-examplesfor the generated website examples.Is this a breaking change?
Does this PR include user facing changes?
no-changeloglabel to this PR.References