GitHub - ProjectArgus-cc/libargus.cc: A unified, unmanaged multimodal runtime. Zero-allocation native AI inference for Java 22+.

An unmanaged, zero-allocation native AI execution runtime consolidating Vision, Speech, and LLM compute pipelines behind a single Project Panama FFM boundary. Notev1.0.0 Stable — Unified Hardware Orchestration & Zero-Allocation ABI

libargus is an ultra-lean, high-performance, model-agnostic inference wrapper engineered to consolidate LLM text generation, Whisper-based speech-to-text (ASR), Speech-LLM text-to-speech (TTS), and bleeding-edge Multimodal (Vision, Audio, and Video) encoding and evaluation pipelines into a single process-global native execution runtime. Built directly on top of the modular GGML and llama.cpp (libmtmd) compute engines, libargus provides a unified, thread-safe C API designed explicitly for frictionless, zero-copy compilation alongside modern unmanaged orchestration frameworks, featuring out-of-the-box structural alignment for the JDK 22+ Project Panama Foreign Function & Memory (FFM) API.

Core Architectural Pillars

Process-Global Backend Singularity: Eliminates VRAM fragmentation and multi-context driver race conditions by orchestrating a singular, shared initialization pathway (ggml_backend_load_all()) across text, audio, speech, and multimodal subsystems. Decoupled Weights & Execution: Separates model weight loading (argus_model_t) from evaluation context memory states (argus_context_t), allowing model reuse across multiple concurrent sessions. Bleeding-Edge Multimodal Projectors: Integrates the new libmtmd C++ engine to ingest raw bitmaps, audio PCM arrays, and video files/streams. Tokenizes prompts and media into a unified chunk sequence, executes projection on the GPU, and automatically configures M-RoPE position grids and non-causal attention matrices. Unmanaged Video Iteration Pipe: Decodes and streams video files frame-by-frame using internal FFmpeg subprocess pipes, yielding raw RGB frames or localized timestamp text chunks (e.g., [12m34s]) at a specified target frame rate. Pointers-Only FFM Alignment: Replaces pass-by-value and volatile C++ polymorphic boundaries with strictly aligned, flat C functions accepting pointers. Structure padding is manually packed to prevent compilers from injecting alignment gaps. Absolute Zero-Copy Memory Boundaries: Eliminates JVM heap primitive arrays (int[], float[]) across hot paths.

Integrates Project Panama MemorySegment parameters directly, allowing token tapes, audio waves, and video frames to generate speech and text with zero GC footprint. Selective Concurrency Locking: Integrates context-level mutex synchronization to allow thread-safe decoding and context operations while enabling fully lock-free, concurrent tokenizer accesses on read-only models. Speculative & MTP Acceleration: Incorporates native verification loops for traditional speculative drafting and Multi-Token Prediction (draft-mtp) directly inside the C++ execution layer. KV Cache Quantization: Supports native configurations (type_k and type_v cache enums) to offload memory footprints to Q8_0, Q4_0, or other optimized formats. Zero-Allocation Vocab & GGUF Metadata Introspection: Exposes safe, unmanaged boundaries to lookup special vocab tokens (BOS, EOS, EOT, PAD), verify End-Of-Generation (EOG) conditions, and dynamically enumerate GGUF dictionary entries.

Codebase Topology libargus/ ├── CMakeLists.txt # Layer-0 dependency isolation & optimization matrix ├── include/ │ └── libargus.h # Master C ABI stable layout definitions ├── src/ │ ├── argus_internal.h # Shared private structures (model & context) │ ├── argus_common.cc # Global backend lifecycles & hardware registries │ ├── argus_text.cc # Llama model/context handling, speculative loops & TTS │ ├── argus_audio.cc #

Load an image or audio file into unmanaged memory try (ArgusBitmap image = ArgusBitmap.loadFile(arena, mctx, Path.of("media/cat.png"), false)) { // 2. Tokenize prompt text replacing the media marker String prompt = "<__media__>\nDescribe what you see in this image."; try (ArgusInputChunks chunks = mctx.tokenize(arena, prompt, true, List.of(image))) { // 3. Evaluate chunks (handles image projection on GPU & M-RoPE position grids) int newNPast = context.evalMultimodalChunks(mctx, chunks, 0, 0, 1024, true); System.out.println("Prompt evaluated. Ready to sample output tokens! New position: " + newNPast); } } } finally { ArgusBackend.free(); } } } Frame-by-Frame Video Stream Processing // Iterate and read frames/timestamps from a video file sequentially try (ArgusVideo video = ArgusVideo.loadFile(arena, mctx, Path.of("media/video.mp4"), 4.0f, 5000); ArgusVideoItem item = new ArgusVideoItem()) { while (video.readNext(item)) { if (item.bitmap() != null) { // Process the extracted RGB frame bitmap (ownership is managed by ArgusVideoItem) ArgusBitmap frame = item.bitmap(); // ... } else if (item.text() != null) { // Received a video timestamp chunk (e.g. "[00m05s]") System.out.println("At timestamp: " + item.text()); } } } Extracting Semantic Text Embeddings Retrieve float embedding vectors from dedicated models (e.g., jina-embeddings-v3): import cc.projectargus.libargus.*; import java.lang.foreign.Arena; import java.lang.foreign.MemorySegment; import java.lang.foreign.ValueLayout; import java.nio.file.

Path;

public class EmbeddingsApp { public static void main(String[] args) { ArgusBackend.init();

try (Arena arena = Arena.ofConfined(); ArgusModel model = ArgusModel.load(arena, Path.of("models/jina-embeddings-v3-Q4_K_M.gguf"), 99, true)) {

// Initialize context configuration with embeddings enabled ArgusContextConfig config = new ArgusContextConfig.Builder(512) .cpuThreads(4) .embeddings(true) .build();

try (ArgusContext context = ArgusContext.init(arena, model, config)) { // Tokenize and evaluate prompt String text = "text-matching: Retrieve semantic vector for this sentence."; MemorySegment textSeg = arena.allocateFrom(text); MemorySegment tokenBuf = arena.allocate(ValueLayout.JAVA_INT, 512); int nTokens = context.tokenize(textSeg, tokenBuf, true); context.decodeBatch(tokenBuf, nTokens, 0, 0, false);

// Retrieve embedding vector (e.g., 1024 dimensions) int expectedDim = 1024; MemorySegment embeddingsBuf = arena.allocate(ValueLayout.JAVA_FLOAT, expectedDim); int nFloats = context.getEmbeddings(0, embeddingsBuf, expectedDim); System.out.println("Retrieved embeddings containing " + nFloats + " floats."); } } finally { ArgusBackend.free(); } } } Model Metadata & Vocabulary Introspection Query special tokens and traverse GGUF model configurations directly from unmanaged memory: // Access model vocabulary metadata int bos = model.vocabBos(); int eos = model.vocabEos(); int eot = model.vocabEot(); // End-Of-Turn token for chat models int nTokens = model.vocabNTokens(); // Vocabulary capacity boolean isEog = model.vocabIsEog(sampledToken); // Native End-Of-Generation verification

// Query model architecture dimensions & parameters int nEmbd