Knowledge Stack
Ingestion
Turn raw data into AI‑ready knowledge. A python‑first pipeline for load → chunk → embed → transform.Runs on Ignite today • Gateway integration plannedA pipeline you can read
“One Engine” approach to ingestion, chunking, embeddings, and multimodal transforms, VNG keeps the pipeline explicit and python‑first while scaling to large volumes.Load
Pull from files, object storage, streams, and databases. Normalize encodings, detect formats, and attach source metadata so downstream steps keep provenance.vng.load()Chunk
Split content into retrieval‑ready units with structure awareness—pages, headings, tables, and code blocks—so context survives splitting.vng.chunk()Embed
Generate embeddings for text and images, capture model version, dimensionality, and field schema to keep vectors consistent across updates.vng.embed()Transform
Normalize fields, enrich metadata, and map to target schemas. Output can stream to VBase, warehouses, or downstream services.vng.transform()Pipeline ViewSources → chunking → embeddings → transforms → sinks. This frames of read/transform/write for multimodal AI workloads.For embeddings, Daft exposes built‑in text and image embedding functions; VNG follows the same model‑first design by making embedding a first‑class step in the pipeline.
