GitHub - KevinKenya/nairobi-connector-open-source

English | 简体中文 | Español | Deutsch

Overview Nairobi OS is a high-performance, distributed data science infrastructure designed for extreme resource efficiency. It enables processing of massive datasets in constrained environments (Edge, IoT, Serverless) by leveraging a specialized Rust-based refinery daemon. By utilizing kernel-level features such as io_uring, memfd, and Huge Pages, Nairobi OS achieves sub-millisecond IPC overhead and zero-copy data pipelines. Key Features

Zero-Copy Ingestion: Hardware-accelerated data loading using io_uring and 1GB Huge Pages. Hardware-Accelerated Visualization: Interactive Jupyter plotting via the Lagos Vision engine (wgpu and egui). Fused Analytics Pipeline: Ingest, crunch, and correlate data in a single D-Bus round trip. Kernel-Bypass Performance: Vectorized analytics leveraging Polars and Rayon for maximum hardware saturation. Sovereign Interface: A fluent Python API that hides the complexity of low-level IPC and memory management.

Architecture Nairobi OS is built on a triad of specialized components connected via D-Bus and shared memory:

Nairobi Axum Refinery: The high-performance Rust core. Manages raw data ingestion and parallelized analytics. Nairobi Hub: The IPC orchestrator. Coordinates file descriptors and signals between the refinery and clients. Lagos Vision: The visual cortex. A headless rendering engine that maps memfd handles directly into the GPU pipeline. Nairobi Python: The high-level bridge. Provides a Pythonic interface to the Rust ecosystem.

[ Data Source ] -> (io_uring/Huge Pages) -> [ Axum Refinery ] | (D-Bus / memfd / iceoryx2) | [ Nairobi Hub ] / \ [ Nairobi Python ] [ Lagos Vision ] | | [ Jupyter Notebook ] <-> [ Visual Output ]

Installation Prerequisites

Operating System: Linux or WSL2 (Kernel 5.10+ required for io_uring and memfd).

Rust: 1.70+ Python: 3.10+ System Libraries: sudo apt-get update && sudo apt-get install -y \ build-essential \ pkg-config \ libdbus-1-dev \ python3-dev \ dbus-x11 \ libosmesa6-dev \ mesa-utils

Build from Source

Clone the Repository: git clone https://github.com/KevinKenya/nairobi-connector-open-source cd nairobi-connector-open-source

Setup Virtual Environment: python3 -m venv .venv source .venv/bin/activate pip install maturin pyo3-build-config zbus anywidget traitlets

Build the Entire Stack: ./build_wheel.sh

Install the Wheel: pip install target/wheels/nairobi_os-0.3.1-py3-none-any.whl

System Configuration (Contributor Guide) Huge Pages The refinery engine prioritizes 1GB Huge Pages for zero-copy buffers. To enable these on your host: echo 1 | sudo tee /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages Note: If 1GB pages are unavailable, the engine will automatically fall back to Transparent Huge Pages (THP). io_uring and SQPOLL The DiracEngine uses io_uring with SQPOLL for maximum I/O throughput. SQPOLL typically requires elevated privileges (CAP_SYS_ADMIN) or a kernel configured with IORING_SETUP_SQPOLL. If the engine cannot initialize SQPOLL, it will fall back to standard io_uring mode. Usage import nairobi_os

# Ignite the refinery nairobi_os.connect()

# Ingest data into a SovereignFrame df = nairobi_os.read_csv("dataset.csv")

# Perform vectorized analytics print(f"Mean: {df.column_name.mean()}")

# Spawn interactive visualization df.plot() Testing Nairobi OS includes a comprehensive test suite covering Rust units, IPC integration, and Python bindings.

Running All Tests # Run Rust tests cargo test --workspace

# Run Python integration tests python3 test_nairobi.py Benchmarking Detailed performance benchmarks can be run from the nairobi-benchmarks directory: cd nairobi-benchmarks pip install -r requirements.txt python orchestration/benchmark_runner.py --workload workloads/workload_nba_pipeline.yaml Troubleshooting

D-Bus Connection Refused: Ensure dbus-daemon is running. In headless environments, use dbus-launch. Lagos Rendering Issues: Lagos requires a valid GPU driver or OSMesa for software fallback. Verify with glxinfo. Huge Page Allocation Failed: Check /proc/meminfo to ensure enough huge pages are reserved by the kernel.

License This project is licensed under the Apache License 2.0. Licensed under the Apache License, Version 2.0.

© 2026 Kevin Chege. All Rights Reserved.