Skip to content

enhancement(remap): unify VRL source configuration - #25948

Open
pront wants to merge 2 commits into
masterfrom
pront-mutually-exclusive-deserializer
Open

enhancement(remap): unify VRL source configuration#25948
pront wants to merge 2 commits into
masterfrom
pront-mutually-exclusive-deserializer

Conversation

@pront

@pront pront commented Jul 27, 2026

Copy link
Copy Markdown
Member

Summary

The remap transform had three mutually exclusive source fields (source, file, files)
with no schema-level enforcement: supplying more than one was caught only at runtime with an
opaque error, and the docs generator had to strip docs::examples from the alternatives to
avoid emitting invalid auto-generated configs. This consolidates everything under a single
source field whose object form is a typed, AnyCondition-style enum, so the constraint is
enforced at parse time and generated docs carry accurate examples again. source stays the
primary field because nearly every remap config uses it.

Changes

  • source now accepts an inline VRL string (unchanged, the common case) or a type-tagged
    object: { type: vrl, program: "..." } for an inline program, or { type: disk, files: [...] }
    to load from disk.
  • A custom deserializer enforces that exactly one of source, file, or files is set (also on
    the compile path, for configs built directly in Rust) and preserves each field through
    serialization so configs round-trip.
  • The top-level file and files fields are deprecated: still functional, documented as
    deprecated, and they emit a startup warning pointing to the new form.

Vector configuration

Inline (unchanged):

source: |
  .foo = "bar"

Inline, explicit form:

source:
  type: vrl
  program: |
    .foo = "bar"

Load from disk:

source:
  type: disk
  files:
    - path/to/program_1.vrl

Deprecated, still works:

file: path/to/program_1.vrl

How did you test this PR?

  • Unit tests covering deserialization of every form, mutual-exclusivity errors, empty-source
    round-tripping, and end-to-end compile/execute for both type: vrl and type: disk (the disk
    case also asserts file watching).
  • cargo run -- validate on configs for all forms, including deprecated file/files (validate
    and emit the warning) and a source + file conflict (fails at load with must provide exactly one of source, file, or files).
  • make validate-config-examples for the generated website examples.

Is this a breaking change?

  • Yes
  • No

Does this PR include user facing changes?

  • Yes. Please add a changelog fragment based on our guidelines.
  • No. A maintainer will apply the no-changelog label to this PR.

References

@github-actions github-actions Bot added docs review on hold The documentation team reviews PRs only after a PR is approved by the COSE team. domain: transforms Anything related to Vector's transform components domain: external docs Anything related to Vector's external, public documentation and removed docs review on hold The documentation team reviews PRs only after a PR is approved by the COSE team. labels Jul 27, 2026
@datadog-vectordotdev

datadog-vectordotdev Bot commented Jul 27, 2026

Copy link
Copy Markdown

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 67d0872 | Docs | Datadog PR Page | Give us feedback!

@github-actions github-actions Bot added the docs review on hold The documentation team reviews PRs only after a PR is approved by the COSE team. label Jul 27, 2026
@pront pront added no-changelog Changes in this PR do not need user-facing explanations in the release changelog type: tech debt A code change that does not add user value. labels Jul 27, 2026
@pront pront changed the title refactor(remap, sample): enforce mutual exclusivity with custom deserializers chore(remap, sample): enforce mutual exclusivity with custom deserializers Jul 27, 2026
@pront pront changed the title chore(remap, sample): enforce mutual exclusivity with custom deserializers chore(remap): enforce mutual exclusivity of source fields with a custom deserializer Jul 27, 2026
@pront pront changed the title chore(remap): enforce mutual exclusivity of source fields with a custom deserializer chore(remap): fold file and files into VrlProgram source enum Jul 27, 2026
@pront
pront force-pushed the pront-mutually-exclusive-deserializer branch from 74f4e74 to a2a0004 Compare July 27, 2026 18:13
@pront pront removed the no-changelog Changes in this PR do not need user-facing explanations in the release changelog label Jul 27, 2026
@pront
pront marked this pull request as ready for review July 27, 2026 19:07
@pront
pront requested review from a team as code owners July 27, 2026 19:07

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 021ec10e8f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/transforms/remap.rs Outdated
Comment thread src/transforms/remap.rs Outdated
Comment thread src/transforms/remap.rs Outdated
@pront pront changed the title chore(remap): fold file and files into VrlProgram source enum enhancement(remap): unify VRL source configuration Jul 27, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9287a95d99

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/transforms/remap.rs Outdated
Comment thread src/transforms/remap.rs Outdated
Comment thread src/transforms/remap.rs Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 872ba5dc18

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/transforms/remap.rs Outdated
@pront
pront force-pushed the pront-mutually-exclusive-deserializer branch from 872ba5d to 9b90bfc Compare July 27, 2026 21:11
The remap transform had three mutually exclusive source fields (source, file,
files) with no schema-level enforcement, so conflicts surfaced only at runtime
and the docs generator had to omit examples from the alternatives.

source is now a single field accepting an inline VRL string, a { type: vrl,
program: ... } object, or { type: disk, files: [...] } to load from disk. A
custom deserializer enforces that exactly one of source, file, or files is set
and warns on the deprecated top-level file/files, which remain functional.

source is optional at the schema layer so legacy file/files-only configs still
validate against the generated schema; the website minimal-example generator
carries a small override for that until the deprecated fields are removed.
@pront
pront force-pushed the pront-mutually-exclusive-deserializer branch from 9b90bfc to ac23566 Compare July 28, 2026 01:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs review on hold The documentation team reviews PRs only after a PR is approved by the COSE team. domain: external docs Anything related to Vector's external, public documentation domain: transforms Anything related to Vector's transform components type: tech debt A code change that does not add user value.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Investigate a proper way to declare mutually exclusive config fields

1 participant