GitHub - peterkelly/rex: The Rex Programming Language

Rex (short for Rush Expressions) is a statically typed, pure functional workflow language. It is designed for scientific computing and data processing: work in Rex is expressed as pure transformations over immutable values, while typed tool modules delegate work to external programs for compute-intensive tasks. The workflow runtime connects four ideas that are particularly useful when processing scientific data: A real functional language makes control flow, data flow, reuse, and error handling part of one small, expressive language rather than a mixture of YAML, shell, and application-specific configuration. Content-addressable storage identifies every stored input and output artifact by its BLAKE3 hash. Files and directory trees are immutable values, so intermediate artifacts can be passed between tools without shared filenames or mutable working directories. Typed tool APIs expose domain concepts such as video codecs, PDF structure, image operations, and output formats. Rex programs construct valid tool requests rather than assembling shell command strings. Isolated Docker execution can run each tool invocation in a fresh, locked-down container containing only its declared inputs. The same workflow can also use locally installed tools for a faster development loop. Together these properties make workflow definitions concise, inspectable, and amenable to parallel execution. They also create a clean boundary between the logic of an analysis and the operating-system processes that carry it out. Rex is also useful as a target for LLM-generated workflows: static types give fast, high-signal feedback, pure code is easier to inspect, and the closed tool boundary sharply limits what generated programs can ask the host to execute. See LLM guidance for syntax and validation advice. Project status: the main branch contains the work in progress toward Rex v4 and is currently versioned as 3.9.x. rex-workflow is new and under active development. The older production release of the core Rex language is available at talo/rex. Why a functional language for workflows? Many workflow systems begin with a directed acyclic graph and gradually grow their own expression syntax, templates, conditionals, loops, and plugin model. Rex starts with a small general-purpose language instead. It provides Hindley–Milner type inference, algebraic data types, records, pattern matching, parametric polymorphism, type classes, higher-order functions, recursion, and modules. That matters for scientific and data-processing work because real pipelines rarely remain a static sequence of commands. They need to map an analysis over a cohort, group observations, branch on metadata, preserve domain-specific failure information, combine several tools, and package reusable methods. Those operations are natural in a functional program: let observations = [3.0, -1.0, 12.0, 7.0, 20.0], selected = filter (\value -> value >= 0.0) observations, normalized = map (\value -> value / 20.0) selected in foldl (\total value -> total + value) 0.0 normalized Rex uses strict evaluation, but expressions and functions are pure: their meaning does not depend on hidden mutable state in the language. This gives the evaluator freedom to run independent asynchronous calls concurrently without making users manage threads, futures, locks, async/await syntax, or callback graphs. Sequential dependencies are expressed by passing one result into the next; independent work remains independent in the source. Purity also improves reviewability. A function's arguments describe the data it can use, its result type describes what it can produce, and an algebraic data type can enumerate every expected outcome. Tool modules preserve this model by returning ordinary typed values such as: Result Media FF.FfmpegError Result Q.PdfOutput Q.QpdfError Result P.TextFile P.PopplerError Expected invalid requests and tool-process failures can therefore be matched and handled inside the workflow. Storage failures, executor failures, and other infrastructure problems remain evaluation errors, keeping domain failures distinct from failures of the runtime itself. Static types catch workflow wiring mistakes early Tool options are represented by records and algebraic data types rather than unstructured maps. Once a hash is wrapped in a semantic artifact type, an Image cannot accidentally be supplied where a Media is expected; a codec option cannot be confused with an image operation; and a multi-file result must be handled as such. Raw imported hashes still have to be classified correctly by the workflow, and a tool reports an error if the stored bytes are not valid input. The compiler catches structural wiring errors before launching an expensive external process. Types are especially valuable when workflows are generated or modified by software. An LLM or another program can propose a Rex workflow, run the parser and type checker, and use precise diagnostics to repair it before any tool is executed. Functional composition scales beyond a DAG file Rex workflows can factor repeated logic into functions, define domain types, transform collections with map and folds, use recursion for hierarchical data, and preserve structured errors across tool boundaries. The result is a program that can grow with an analysis instead of a configuration file that eventually needs an external templating language. Content-addressable data Scientific workflows are easier to reason about when artifacts are values, not mutable locations. rex-workflow includes a content-addressable store in which every object is named by the BLAKE3 hash of its bytes. The data model has two object kinds: A blob is an opaque byte sequence: an image, video, PDF, table, model, log, or any other file. A tree is a deterministically encoded map from names to blob or tree entries. Trees represent directories and multi-file datasets, extracted image collections, and nested results. Each tree entry records its kind, hash, and size. Trees can contain other trees, so one root hash identifies a complete immutable directory hierarchy. The sizes are cumulative, meaning that an entry that refers to another tree includes total size of everything it contains. host file or directory | | rex store import v BLAKE3 blob/tree hash | | typed Rex values and tool calls v new blob/tree hashes | | rex store export v host file or directory This model provides several useful properties: Stable identity. The same bytes always produce the same hash, regardless of their original filename or machine. Immutability. Existing inputs cannot be overwritten. A transformation creates a new object and returns a new hash. Deduplication. Writing content that is already present resolves to the existing address instead of creating a second logical object. Unambiguous handoff. A tool consumes an exact object and returns the exact identities of its outputs. There is no question about which revision of a path was read. Natural composition. An output hash from one tool is immediately usable by another tool without exporting and re-importing an intermediate file. Portable storage. The same API can use a local filesystem store, an in-memory store, or a cloud-hosted S3 bucket. The CAS is deliberately aligned with the functional language: creating a new artifact does not mutate an old artifact, and calling put with the same content returns the same value. A directory update is represented by creating new trees along the changed path, much like Git's object model. Content addressing is an important foundation for caching and provenance, but it is not magic. A hash identifies bytes; it does not by itself record which workflow, parameters, tool version, or container produced them. Likewise, an external tool may be nondeterministic even when its inputs are immutable. Rex makes the artifact boundary explicit so that caching and provenance can be built rigorously instead of inferred from mutable paths. Store operations available to Rex programs The built-in std.storage module exposes immutable data directly: import std.storage (*); let report = put_string "analysis complete", files = dict_from_entries [("report.txt", (Blob, report))], result_tree = put_tree files in result_tree Programs can use put_string, put_bytes, put_tree, get_string, get_bytes, and get_tree. The shared std.artifacts module wraps hashes with semantic meaning through Media, Image, Pdf, and JsonFile; tool-specific result types add operation metadata where needed. import std.artifacts (Pdf); fn as_pdf (content: Hash) -> Pdf = Pdf { content = content }; Constructing an artifact classifies a CAS blob but does not inspect its bytes. The consuming tool remains responsible for validating that the stored content has a supported representation. Tools are typed capabilities Rex does not expose a general shell command to workflow programs. Instead, the host registers modules whose functions and types describe supported operations. The current workflow catalog contains: Rex module Runtime programs Selected capabilities tools.ffmpeg FFmpeg, FFprobe Transcode and remux media, extract audio or frames, create thumbnails, concatenate, mux, segment, package HLS/DASH, probe metadata, inspect packets and frames, and query capabilities tools.gnuplot Gnuplot Render typed figures from inline curves, error bars, bands, bars, histograms, heatmaps, vectors, labels, point clouds, paths, surfaces, and annotations tools.graphviz Graphviz dot Render semantic directed or undirected graphs with typed attributes, declared nodes, binary edges, ports, labels, defaults, and nested subgraphs tools.imagemagick ImageMagick Generate and transform images, batch-convert, identify, compare, composite, montage, extract pixels, and query formats and capabilities tools.qpdf QPDF Check PDFs, count pages, export structured JSON, transform or linearize, merge/split pages, and apply overlays or underlays tools.poppler pdfinfo, pdftotext, pdftocairo, pdfimages