Is GGUF over?

[m8rr]

Native formats are definitely faster, though my hard drive would beg to differ.

"If you use gguf we recommend keeping dynamic vram enabled and using native ComfyUI model formats instead."

[m8rr]

reader = gguf.GGUFReader(path)

Currently with GGUF, executing just a single line, takes 20 seconds for Gemma4 E2B. On top of that, it executes once more in the tokenizer loader.

m8rr@736fc9c
Optimized GGUFReader tokenizer array parsing, delivering a noticeable speedup in initial loading time.
(Further optimization is possible by skip list processing.
Lists are handled by the tokenizer generator, and the generated tokenizer is saved as a file cache.
If a tokenizer file already exists, it is retrieved and reused.)

m8rr@7187555
Performance is further enhanced by reusing the existing GGUFReader, preventing unnecessary reloads.

m8rr@1c113ea
Improves inference speed by torch-compiling dequantize_functions. In some cases, it delivers up to a 30% speedup.

[m8rr]

https://github.com/m8rr/ComfyUI-GGUF/tree/Dynamic-VRAM
Qwen3-VL Vision Support, gemma4-E2B Vision Support.
Qwen3-VL-4B is correctly recognized as Krea2TEModel_, and it will probably work without mmproj.
https://huggingface.co/realrebelai/KREA-2_GGUFs/tree/main/TURBO

D:\AI\ComfyUI_windows_portable>.\python_embeded\python.exe -s ComfyUI\main.py --windows-standalone-build --fast fp16_accumulation --use-sage-attention --disable-api-nodes -cache-ram 0 --fast-disk
[INFO] setup plugin alembic.autogenerate......
[INFO] Found comfy_kitchen backend.....
[INFO] Checkpoint files will always be loaded safely.
[INFO] Total VRAM 12282 MB, total RAM 32085 MB
[INFO] pytorch version: 2.12.1+cu130
[INFO] Enabled fp16 accumulation.
[INFO] Set vram state to: NORMAL_VRAM
[INFO] Device: cuda:0 NVIDIA GeForce RTX 4070 SUPER : cudaMallocAsync
[INFO] Using async weight offloading with 2 streams
[INFO] Enabled pinned memory 12834.0
[INFO] Using sage attention
aimdo: .....
[INFO] DynamicVRAM support detected and enabled
[INFO] Python version: 3.13.12 (tags/v3.13.12:1cbe481, Feb  3 2026, 18:22:25) [MSC v.1944 64 bit (AMD64)]
[INFO] ComfyUI version: 0.26.0
[INFO] comfy-aimdo version: 0.4.10
[INFO] comfy-kitchen version: 0.2.10
[INFO] comfyui-frontend-package version: 1.45.19
[INFO] comfyui-workflow-templates version: 0.10.3
[INFO] comfyui-embedded-docs version: 0.5.5

[INFO] got prompt
[INFO] Using pytorch attention in VAE
[INFO] Using pytorch attention in VAE
[INFO] VAE load device: cuda:0, offload device: cpu, dtype: torch.bfloat16
[INFO] got prompt
[INFO] gguf qtypes: F32 (145), Q6_K (37), Q4_K (216)
[WARNING] Dequantizing token_embd.weight to prevent runtime OOM.
[INFO] Attenpting to find mmproj file for text encoder...
[INFO] Using mmproj 'Qwen3-VL-4B-Instruct.mmproj-Q8_0.gguf' for text encoder 'Qwen3-VL-4B-Instruct-Q4_K_M.gguf'.
[INFO] gguf qtypes: F32 (212), Q8_0 (104)
[INFO] CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16
[INFO] Requested to load Krea2TEModel_
[INFO] Model Krea2TEModel_ prepared for dynamic VRAM loading. 2808MB Staged. 0 patches attached. Force pre-loaded 243 weights: 1158 KB.
[INFO] gguf qtypes: Q4_K (262), F32 (166), F16 (2)
[INFO] model weight dtype torch.float16, manual cast: None
[INFO] model_type FLUX
[INFO] Requested to load Krea2
[INFO] Model Krea2 prepared for dynamic VRAM loading. 6877MB Staged. 0 patches attached. Force pre-loaded 160 weights: 2824 KB.
100%|████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:11<00:00,  1.49s/it]
[INFO] Requested to load WanVAE
[INFO] 0 models unloaded.
[INFO] Model WanVAE prepared for dynamic VRAM loading. 241MB Staged. 0 patches attached. Force pre-loaded 60 weights: 61 KB.
[INFO] Prompt executed in 25.12 seconds
[INFO] Model Krea2 prepared for dynamic VRAM loading. 6877MB Staged. 0 patches attached. Force pre-loaded 160 weights: 2824 KB.
100%|████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:12<00:00,  1.52s/it]
[INFO] 0 models unloaded.
[INFO] Model WanVAE prepared for dynamic VRAM loading. 241MB Staged. 0 patches attached. Force pre-loaded 60 weights: 61 KB.
[INFO] Prompt executed in 12.60 seconds

[m8rr]

ComfyUI now supports INT8. Is GGUF really dead?

https://huggingface.co/silveroxides/models

4070s, windows
I tested Ideogram4 INT8 with Turbo LoRA, generating 768x1280 images in 6 steps.
GGUF Q4K, 5.7G, 12s
FP8, 9G, 6s
INT8 9.3G 5s

krea2
Q4KM, 7G, 10s
INT8 14G, 8s

Is the gap narrowing because INT8 exceeds the VRAM, or is the BF16 of Ideogram 4 GGUF being handled incorrectly?

[SRStwo]

The "comfyanonymous" dev always talks about performance when making changes like this but not about quality. I have yet to see anyone produce data that shows any version of FP8 beating or equaling Q8_0 in perplexity. Performance is meaningless if the quality is inferior.

[m8rr]

@SRStwo I always use Turbo models and treat it like a slot machine, but I totally agree with you. There definitely should be an option for higher quality.

[Ph0rk0z]

I don't think it's over. GGUF was always slower but fidelity and compatibility is better. FP8 loses on quality time after time. FP4 is better but you better have $$blackwell$$. Int8 is good if you only want that size. GGUF can be sped up a few % with cublas too.. but I gotta make it a checkbox. Especially for stuff like text encoders, there really is no other sane option.

[kingp0dd]

I tried int8 ltx2.3 but all the models won't fit in my 32gb ram 12gb vram so it reloads from disk, making GGUF faster. Pretty sure there are other similar setups like mine, which still benefit from gguf.

Unless I'm doing something wrong with int8.