Omni legacy pipeline by dkalinowski · Pull Request #4379 · openvinotoolkit/model_server

dkalinowski · 2026-07-16T11:59:11Z

CVS-191153

Copilot

Pull request overview

Adds initial “Omni” pipeline support to OVMS GenAI serving, enabling audio input (Chat Completions + Responses) and audio output generation/streaming integration, along with supporting utilities, docs, and demo clients.

Changes:

Introduces a new OMNI pipeline type (proto + C++ enums) and routes initialization to a new legacy Omni servable/executor/initializer.
Adds audio input decoding to the LLM input-processing pipeline and extends OpenAI-compatible request parsing for input_audio and output modalities/audio.
Refactors audio_utils into a dedicated namespace and provides non-resampling decode helper used by Omni audio input.

Reviewed changes

Copilot reviewed 34 out of 36 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
src/test/audio/text2speech_test.cpp	Updates tests to use `ovms::audio_utils` namespace.
src/test/audio/audio_utils_test.cpp	Updates tests to use `ovms::audio_utils` namespace.
src/llm/servable_initializer.hpp	Adds `PipelineType::OMNI`.
src/llm/servable_initializer.cpp	Detects OMNI models and initializes Omni legacy servable.
src/llm/omni_model/legacy/servable.hpp	Declares Omni legacy servable/context/properties.
src/llm/omni_model/legacy/servable.cpp	Implements Omni request lifecycle, audio response injection, streaming handling.
src/llm/omni_model/legacy/servable_initializer.hpp	Declares Omni legacy servable initializer.
src/llm/omni_model/legacy/servable_initializer.cpp	Builds `ov::genai::OmniPipeline` + tokenizer and config.
src/llm/omni_model/legacy/legacy_executor.hpp	Declares Omni legacy executor + wrapper thread.
src/llm/omni_model/legacy/legacy_executor.cpp	Implements Omni generation execution + speech streaming callback.
src/llm/llm_calculator.proto	Adds OMNI to pipeline enum for graph options.
src/llm/io_processing/input_processors/image_decoding_processor.cpp	Preserves multimodal part ordering when non-image parts exist (e.g., audio).
src/llm/io_processing/input_processors/audio_decoding_processor.hpp	Adds new processor to extract `input_audio` into tensors.
src/llm/io_processing/input_processors/audio_decoding_processor.cpp	Implements base64 decode + audio parsing to `ov::Tensor`.
src/llm/io_processing/input_processor.cpp	Wires AudioDecodingProcessor into pipeline for Omni chat requests.
src/llm/io_processing/input_processing_config.hpp	Adds `isOmni` config flag.
src/llm/BUILD	Adds Omni + audio decoding sources and deps to Bazel targets.
src/llm/apis/openai_responses.cpp	Supports `input_audio` items and serializes streamed audio delta events.
src/llm/apis/openai_request.hpp	Adds audio output request fields to request model.
src/llm/apis/openai_completions.cpp	Validates `input_audio` content parts for chat completions.
src/llm/apis/openai_api_handler.cpp	Parses `modalities` + `audio` output config.
src/audio/text_to_speech/t2s_calculator.cc	Qualifies audio utils calls with `ovms::audio_utils`.
src/audio/speech_to_text/s2t_calculator.cc	Qualifies audio utils calls with `ovms::audio_utils`.
src/audio/audio_utils.hpp	Moves audio helpers into `ovms::audio_utils` and adds `readWithoutResample`.
src/audio/audio_utils.cpp	Implements `readWithoutResample` and wraps implementation in namespace.
OMNI_PLAN.md	Documents design/plan and API mapping for Omni integration.
OMNI_AUDIO_OUTPUT.md	Documents audio output implementation and streaming event format.
demos/omni/omni_voice_chat.py	Adds a voice-to-voice demo client (record + stream + playback).
demos/omni/omni_stream_playback.py	Adds streaming playback demo (text then audio deltas).
demos/omni/omni_responses_client.py	Adds minimal Responses API audio-input demo client.
demos/omni/omni_full_client.py	Adds full multimodal demo (audio+image → text+audio).
demos/omni/omni_client.py	Adds minimal Chat Completions audio-input demo client.
demos/omni/omni_audio_output_client.py	Adds Chat Completions audio-output demo client.
demos/omni/.gitignore	Ignores generated WAVs in demos.

dkalinowski · 2026-07-21T14:04:48Z

+            if (responseDoc.HasMember("choices") && responseDoc["choices"].IsArray() && responseDoc["choices"].Size() > 0) {
+                // Chat Completions format: inject into choices[0].message.audio
+                auto& message = responseDoc["choices"][0]["message"];
+                rapidjson::Value audioObj(rapidjson::kObjectType);
+                audioObj.AddMember("data", rapidjson::Value(audioBase64.c_str(), alloc), alloc);
+                std::string transcript = message.HasMember("content") && message["content"].IsString() ? message["content"].GetString() : "";
+                audioObj.AddMember("transcript", rapidjson::Value(transcript.c_str(), alloc), alloc);
+                message.AddMember("audio", audioObj, alloc);
+            } else if (responseDoc.HasMember("output") && responseDoc["output"].IsArray()) {


it will base on executionContext->apiHandler->getEndpoint()

+    // Existence of talker model indicates omni pipeline
+    bool isOmni = std::filesystem::exists(parsedModelsPathFs / "openvino_talker_model.xml");
    // Existence of embeddings models indicates we are dealing with VLM pipeline
    bool isVLM = (std::filesystem::exists(parsedModelsPathFs / "openvino_text_embeddings_model.xml") &&
-                  std::filesystem::exists(parsedModelsPathFs / "openvino_vision_embeddings_model.bin"));
+                  std::filesystem::exists(parsedModelsPathFs / "openvino_vision_embeddings_model.bin")) && !isOmni;


dkalinowski · 2026-07-20T10:19:45Z

+                const auto part = content[j];
+                const auto type = part["type"].as_string().value_or("");
+                if (type == "image_url") {
+                    ov::genai::JsonContainer textEntry({{"type", "text"}, {"text", imageTagByPart[j]}});


it will work only for omni i think, needs to be changed

mzegla · 2026-07-20T14:35:15Z

+├── openvino_model.xml          # thinker (VLM)
+├── openvino_talker_model.xml   # talker (speech)


Why the comments? If we add comments to those two models, why not for other XMLs?

I wanted to showcase whats new, it is not really user facing documentation yet I would say, updated to this for now:

but after we merge it we should add sphinx doc and actual demo similar to our other demos

mzegla · 2026-07-20T15:29:13Z

+    // Copy audio output config from parsed request to execution context
+    const auto& req = omniExecutionContext->apiHandler->getRequest();
+    omniExecutionContext->audioOutputRequested = req.audioOutputRequested;
+    omniExecutionContext->audioVoice = req.audioVoice;
+    omniExecutionContext->audioFormat = req.audioFormat;


Is that the only omni specific part in this method? If so, could we call base class parseRequest on this and then cast context to omni and fill remaining fields?

I think we should do it in next PR - it is related to VLMPipeline & OmniPipeline cleanup

mzegla · 2026-07-20T15:31:48Z

+absl::Status OmniModelLegacyServable::readCompleteExecutionResults(std::shared_ptr<GenAiServableExecutionContext>& executionContext) {
+    auto omniExecutionContext = std::static_pointer_cast<OmniModelLegacyServableExecutionContext>(executionContext);
+    if (omniExecutionContext->payload.client->isDisconnected()) {
+        return absl::CancelledError();
+    }
+    omniExecutionContext->finished.wait();
+    if (!omniExecutionContext->success) {
+        return absl::InvalidArgumentError("Request processing failed, check its correctness.");
+    }
+    return absl::OkStatus();
+}


Do we need implementation of that? Looks kind of generic - wouldn't base class be able to handle this?

I think we should do it in next PR - it is related to VLMPipeline & OmniPipeline cleanup

mzegla · 2026-07-20T15:32:45Z

+absl::Status OmniModelLegacyServable::readPartialExecutionResults(std::shared_ptr<GenAiServableExecutionContext>& executionContext) {
+    executionContext->deltaChannel.waitForData();
+    return absl::OkStatus();
+}


Same, looks very generic, not sure if we need implementation here

I think we should do it in next PR - it is related to VLMPipeline & OmniPipeline cleanup

mzegla · 2026-07-20T15:34:37Z

+
+namespace ovms {
+
+struct OmniModelLegacyServableExecutionContext : public GenAiServableExecutionContext {


How about VLM servable execution context as base? Have you considered that?
Same for all other classes. Maybe it would allow us to reuse some more generic methods?

dkalinowski · 2026-07-21T12:08:14Z

+                std::string transcript = message.HasMember("content") && message["content"].IsString() ? message["content"].GetString() : "";
+                audioObj.AddMember("transcript", rapidjson::Value(transcript.c_str(), alloc), alloc);
+                message.AddMember("audio", audioObj, alloc);
+            } else if (responseDoc.HasMember("output") && responseDoc["output"].IsArray()) {


probably base on apihandler->api type or something like that

dkalinowski · 2026-07-21T12:10:18Z

    // Shape: {"delta":{...}} for content/reasoning/tool_calls, or an empty Document{}
-    // for finish-only chunks. Inspect the delta directly — no parsing needed here.
-    if (parsedDelta.HasMember("delta") && parsedDelta["delta"].IsObject()) {
+    // for finish-only chunks, or {"_audio_delta":"<base64>"} for audio chunks.


the comment should be moved below

it still applies in general, so i extended it

mzegla · 2026-07-23T08:34:07Z


+namespace ovms::audio_utils {
+
+static constexpr uint32_t PIPELINE_SUPPORTED_SAMPLE_RATE = 16000;


Will read methods work with custom sampling rate?
This name is also quite generic - what pipeline?

mzegla · 2026-07-23T08:35:47Z

+std::vector<float> readWithoutResample(const std::string_view& audioData, const std::string& format) {
+    if (format == "wav") {
+        return readWav(audioData, 0);
+    } else if (format == "mp3") {
+        return readMp3(audioData, 0);
+    } else {
+        throw std::runtime_error("Unsupported audio format: " + format);
+    }
+}


If 0 is a special value to disable resampling, maybe we should make it a const and somehow document (like explicitly in the comment)?

It is documented in code:

// Decode WAV data into mono float32 PCM samples. // If targetSampleRate > 0, resamples to that rate. Otherwise returns at native sample rate. std::vector<float> readWav(const std::string_view& wavData, uint32_t targetSampleRate = PIPELINE_SUPPORTED_SAMPLE_RATE); // Decode MP3 data into mono float32 PCM samples. // If targetSampleRate > 0, resamples to that rate. Otherwise returns at native sample rate. std::vector<float> readMp3(const std::string_view& mp3Data, uint32_t targetSampleRate = PIPELINE_SUPPORTED_SAMPLE_RATE);

but I will make 0 a constexpr RESAMPLE_DISABLED

mzegla · 2026-07-23T08:39:35Z

+            }
+        }
+        request.audioOutputRequested = hasAudio;
+        // When modalities is explicitly provided and "text" is absent, suppress text in response


Suggested change

// When modalities is explicitly provided and "text" is absent, suppress text in response

// When "modalities" is explicitly provided and "text" is absent, suppress text in response

mzegla · 2026-07-23T08:40:06Z

+            if (modality == "audio") {
+                hasAudio = true;
+            } else if (modality == "text") {
+                hasText = true;
+            }


No images/video modalities?

These modalities determine what model is capable of generating. We dont know any llm model that is capable of generating images. And it is not supported by OpenAI API anyway

mzegla · 2026-07-23T09:29:24Z

    bool isVLM = false;
+    // True for Omni servables. Enables AudioDecodingProcessor in addition to
+    // ImageDecodingProcessor (implies isVLM-like behavior for images).
+    bool isOmni = false;


I can see possibility that we will serve models that will accept audio input and will not be Omni. Like text, image, audio on the input and just text on the output. That's not Omni I suppose, but still requires audio processing. I'm thinking about renaming isVLM, isOmni to something explicit like supportsVisualInput, supportsAudioInput.
That naming would remove pipeline type awareness from the scope and focus on input data types.

This is good direction, but lets do it once we have actual models that are capable of that

mzegla · 2026-07-23T09:31:22Z

+        if (context.config.isOmni) {
+            processors.emplace_back(std::make_unique<AudioDecodingProcessor>());
+        }
+
+        processors.emplace_back(std::make_unique<TextContentNormalizationProcessor>());


We have a comment in input processor header I believe where we mention input processor ordering. Could you update it as well to reflect the new ordering?

mzegla · 2026-07-23T09:39:03Z

      // Default option. If selected, pipeline type will be inferred by the serving
      AUTO = 4;
+      // Omni model (text, image, audio input; text + speech output) based on OmniPipeline
+      OMNI = 5;


I would prefer the default option to be first or last so it stands out among others. Would it be possible to for example move AUTO to the top with 0 identifier? Or would we experience some compatibility issues?

I dont think it will introduce any compatibility issues - I will change it

mzegla · 2026-07-23T09:40:07Z

-                  std::filesystem::exists(parsedModelsPathFs / "openvino_vision_embeddings_model.bin"));
+    bool hasEmbeddingsModels = (std::filesystem::exists(parsedModelsPathFs / "openvino_text_embeddings_model.xml") &&
+                                   std::filesystem::exists(parsedModelsPathFs / "openvino_vision_embeddings_model.bin")) &&
+                               !hasTalkerModel;


Does talker model existence automatically means model cannot have embedding models?

No, I will change the comment to:

// Existence of embeddings models indicates we are dealing with VLM pipeline // But if it has talker model, it means it is Omni pipeline which is built out of VLM Pipeline and Talker

Still, the comment does not resolve the way hasEmbeddingsModels means. With that change it is dependent of hasTalkerModel which blurs the meaning.
Maybe we need another bool here? Or change conditions to explicitly if(hasEmbeddingsModels && !hasTalkerModel) then VLM

mzegla · 2026-07-23T09:41:40Z

                SPDLOG_LOGGER_ERROR(modelmanager_logger, "Error during LLM node resources initialization: {}", status.string());
                return status;
            }
+        } else if (pipelineType == PipelineType::OMNI) {


Don't we need similar validation as for VLM to match behavior? For example if someone sets Omni explicitly, but there is no talker model in the catalog.

When someone forces --pipeline_type VLM but the model is LLM:

Models directory content indicates non-VLM pipeline, but pipeline type is set to VLM type.

When someone forces --pipeline_type OMNI but the model is VLM:

Models directory content indicates VLM pipeline, but pipeline type is set to non-VLM type.

When someone forces --pipeline_type VLM but the model is OMNI:

Models directory content indicates non-VLM pipeline, but pipeline type is set to VLM type.

I think it works as expected

mzegla · 2026-07-23T11:31:18Z

            const auto& exportSettings = this->serverSettings.hfSettings.exportSettings;
            auto textGenSettings = std::get<TextGenGraphSettingsImpl>(this->serverSettings.hfSettings.graphSettings);
-            std::vector allowedPipelineTypes = {"LM", "LM_CB", "VLM", "VLM_CB", "AUTO"};
+            std::vector allowedPipelineTypes = {"LM", "LM_CB", "VLM", "VLM_CB", "AUTO", "OMNI"};


Can we keep the ordering as in the graph? So AUTO at the end?

mzegla · 2026-07-23T11:39:12Z

+    const auto status = processor.process(req);
+
+    EXPECT_TRUE(status.ok());
+    EXPECT_TRUE(req.inputAudios.empty());


can we also check that processor did not remove anything, so the history is intact

mzegla · 2026-07-23T11:43:24Z

Have you considered a test with image, audio and normalization processor working together?

dkalinowski added 9 commits July 8, 2026 16:29

basic implementation, no audio support

8698743

audio input

1aab3d3

save

971d2b7

save

54c3c24

save

3f2ddda

save

8ca465a

save

194eb6b

save

ad35cda

save

78f743c

Copilot AI review requested due to automatic review settings July 16, 2026 11:59

Copilot started reviewing on behalf of dkalinowski July 16, 2026 12:00 View session

Copilot AI reviewed Jul 16, 2026

View reviewed changes

dkalinowski and others added 9 commits July 16, 2026 16:00

Merge remote-tracking branch 'origin/main' into omni

b0f7029

save

4a80e10

fix copilot issues

fe13f20

save

d46c27e

adjustments

7cb565e

save

10372bb

ov 2026.3 rc2

3b90c92

ov 2026.3 rc2

c87c3ac

Merge remote-tracking branch 'origin/ov0717' into omni

4273af9

dkalinowski force-pushed the omni branch from 0873ba1 to 4273af9 Compare July 17, 2026 15:31

fix windows

198de1d

dkalinowski commented Jul 20, 2026

View reviewed changes

Comment thread src/audio/audio_utils.cpp

dkalinowski commented Jul 20, 2026

View reviewed changes

Comment thread src/llm/apis/openai_api_handler.cpp

dkalinowski commented Jul 20, 2026

View reviewed changes

dkalinowski requested review from michalkulakowski and mzegla July 20, 2026 11:01

pre review cleanup

dcfaa12

dkalinowski force-pushed the omni branch from 2354470 to dcfaa12 Compare July 20, 2026 13:06

mzegla reviewed Jul 20, 2026

View reviewed changes

Merge remote-tracking branch 'origin/main' into omni

6b8239a

dkalinowski commented Jul 21, 2026

View reviewed changes

dkalinowski added 5 commits July 21, 2026 17:30

post-review changes

bcd7e60

better streaming client

b077ceb

audio chunk frame configurable

a3b8bf5

Merge remote-tracking branch 'origin/main' into omni

f424c48

change tag preservation order logic

164aed0

mzegla reviewed Jul 23, 2026

View reviewed changes

dkalinowski added 2 commits July 23, 2026 11:54

unit tests

2112ddf

Merge remote-tracking branch 'origin/main' into omni

261d9b8

michalkulakowski approved these changes Jul 23, 2026

View reviewed changes

post re-review

fbf724a

dkalinowski changed the title ~~Omni~~ Omni legacy pipeline Jul 23, 2026

mzegla reviewed Jul 23, 2026

View reviewed changes

dkalinowski added 2 commits July 23, 2026 14:39

post-post re-review

60717c0

move processRequest code to prepareRequest

07b91a0

dkalinowski force-pushed the omni branch from 40c5168 to 07b91a0 Compare July 23, 2026 13:38

fix static analysis

5bf0bf4

dkalinowski force-pushed the omni branch from 969ac64 to 5bf0bf4 Compare July 23, 2026 13:58

		├── openvino_model.xml # thinker (VLM)
		├── openvino_talker_model.xml # talker (speech)


		namespace ovms {

		struct OmniModelLegacyServableExecutionContext : public GenAiServableExecutionContext {


		namespace ovms::audio_utils {

		static constexpr uint32_t PIPELINE_SUPPORTED_SAMPLE_RATE = 16000;

	// When modalities is explicitly provided and "text" is absent, suppress text in response
	// When "modalities" is explicitly provided and "text" is absent, suppress text in response

Uh oh!

Conversation

dkalinowski commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dkalinowski Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dkalinowski commented Jul 16, 2026 •

edited

Loading

dkalinowski Jul 22, 2026 •

edited

Loading

dkalinowski Jul 23, 2026 •

edited

Loading