Mojibake — Unicode 17 for C

Unicode 17 - C11 - zero dependencies Unicode text processing,without the baggage. Mojibake is a small, fast, self-contained Unicode library for C11 and C++17. Ship standards-compliant text handling without a runtime or dependency tree. StandardUnicode 17.0 RuntimeNone LicenseMIT Start here A practical Unicode toolkit Mojibake is a low-level Unicode 17 text-processing library written in C11 and compatible with C++17. It is released under the MIT License. Usage You don't need to install anything. There are two files (mojibake.c, mojibake.h) to add to your C/C++ project. Download it here mojibake-amalgamation-027.zip Examples of normalization, characters count and NFKC casefold. #include <stdio.h> #include <string.h> #include "mojibake.h" void print_string(const char *input, size_t length); int main(int argc, char *const argv[]) { const char *input = "Cafe\xCC\x81"; size_t length = strlen(input); mjb_result result; // Normalize example: in NFC e + ◌́ -> é (U+00E9) if(mjb_normalize(input, length, MJB_ENC_UTF_8, MJB_NORMALIZATION_NFC, MJB_ENC_UTF_8, &result) != MJB_STATUS_OK) { return 1; } // Cafe + ◌́ (U+0301, COMBINING ACUTE ACCENT) -> Café print_string(input, length); // Caf + é (U+00E9, LATIN SMALL LETTER E WITH ACUTE) -> Café print_string(result.output, result.output_size); const char *mojibake = "文字化け"; length = strlen(mojibake); // String length example: mjb_string_length counts the number of characters in a // string, not the number of bytes.

printf("\"%s\" encoded in UTF-8 is %zu bytes long, and %zu characters long\n", mojibake, length, mjb_string_length(mojibake, length, MJB_ENC_UTF_8)); mjb_result_free(&result); const char *case_input = "Straße"; // NFKC casefold example: in NFKC casefold, ß -> ss if(mjb_nfkc_casefold(case_input, strlen(case_input), MJB_ENC_UTF_8, MJB_ENC_UTF_8, &result) != MJB_STATUS_OK) { return 1; } printf("%s -> %.*s\n", case_input, (int)result.output_size, result.output); mjb_result_free(&result); return 0; } void print_string(const char *input, size_t length) { for(size_t i = 0; i < length; ++i) { unsigned char byte = (unsigned char)input[i]; if(byte >= 0x21 && byte <= 0x7E) { printf("%c", byte); } else { printf("<%02X>", byte); } } printf("\n"); } This output: Cafe<CC><81> Caf<C3><A9> "文字化け" encoded in UTF-8 is 12 bytes long, and 4 characters long Straße -> strasse Mojibake aims to be: Small Easy to use Fast Self-contained Mojibake do: Run in all modern OSes (Linux, macOS, FreeBSD, OpenBSD, NetBSD, Windows 10/11) Pass the official Unicode test suites for supported algorithms Implement all Unicode standard algorithms Satisfy all Unicode Conformance Requirements Feature highlights All the C files, together with the Unicode data tables, are concatenated into a single large file and header: mojibake.c and mojibake.h.

Zero dependencies. Text transformation Normalization: NFC/NFD/NFKC/NFKD (mjb_normalize), identifier-oriented NFKC case folding (mjb_nfkc_casefold), plus a fast quick-check (mjb_string_is_normalized) (UAX #15, Unicode 17.0.0) Case conversion: uppercase, lowercase, titlecase, and case folding with full special-casing and conditional mappings (mjb_case) Filtering: strip controls, spaces, or numeric characters while normalizing (mjb_string_filter) Text analysis Character database: every Unicode Character Database property: category, script and Script_Extensions, block, plane, numeric value, name (mjb_codepoint_character, mjb_codepoint_script_extensions) Segmentation: grapheme clusters, words, sentences, and line-break opportunities (UAX #29, Unicode 17.0.0, UAX #14, Unicode 17.0.0) Bidirectional text: full Unicode Bidirectional Algorithm: paragraph resolution, line reordering, runs (UAX #9, Unicode 17.0.0) Emoji: codepoint properties, sequence analysis, RGI emoji detection Display width: East Asian width and terminal display width, with width-aware truncation (mjb_display_width, mjb_truncate_width) Sorting and comparison Collation: Unicode Collation Algorithm string comparison and sort keys, in shifted and non-ignorable modes (mjb_string_compare, mjb_collation_key, UTS #10, Unicode 17.0.0) Security Confusable detection: generate reusable skeletons and check if strings are visually confusable (mjb_confusable_skeleton, mjb_string_is_confusable, UTS #39, Unicode 17.0.0) Identifier validation: XID/ID checks for parser and compiler authors (mjb_string_is_identifier, UAX #31, Unicode

17.0.0) Integration Encodings: the API accepts and outputs UTF-8, UTF-16LE, UTF-16BE, UTF-32LE, UTF-32BE strings, with encoding detection and conversion (mjb_string_encoding, mjb_string_convert_encoding) Parsing and string functions: character-by-character iteration (mjb_next_character) and standard C string.h-style helpers (mjb_string_length, and others) Locales: strict BCP 47 language tag parsing (mjb_locale_parse) Embeddable: custom allocators (mjb_set_memory_functions), build-time feature flags to trim table size, a C++17 wrapper (src/cpp/mojibake.hpp), a CLI tool (src/shell), and a WASM + TypeScript API (src/api) Tested: Mojibake uses Attractor as test suite and run 1.5M+ assertions including the official Unicode conformance suites for supported algorithms Fuzz Mojibake is fuzzed with libFuzzer over untrusted byte input AddressSanitizer and UBSan clean Build-time features Mojibake can compile out optional feature tables to reduce binary size.

Feature macros default to enabled. #define MJB_FEATURE_CHARACTER_NAMES controls the Unicode character-name tables used by mjb_codepoint_character(...) to fill mjb_character.name. When disabled, the tables are not compiled and mjb_character.name is reported as Codepoint U+XXXX. This will redude the output of ~30%. With CMake: cmake -S . -B build-no-name -DMJB_FEATURE_CHARACTER_NAMES=OFF cmake --build build-no-name With the provided Makefile: make build BUILD_DIR=build-no-name FEATURE_CHARACTER_NAMES=OFF make test-no-names API documentation See API.md or the site for the detailed documentation. CLI The src/shell directory builds the mojibake CLI used to test the library. Example usage: # This outputs "NFC: Café", e + ◌́ -> é mojibake nfc $'Cafe\u0301' # The output an emoji sequence [1] Basic, [2] Fully-qualified of two characters U+263A U+FE0F mojibake emoji "☺" Building from source and contributing See CONTRIBUTING.md for instructions. Licenses Mojibake is released under the MIT License (see LICENSE). Legalese Here you can find the very detailed and boring informations needed to have this library conformant to the Unicode standard, or at least what I got, at CONFORMANCE_REQUIREMENTS.md Thanks Mojibake is built using the work of extraordinary individuals and teams. Unicode Character Database - Copyright © 1991-2026 Unicode, Inc. (see license.txt) Unicode CLDR Project - Copyright © 2004-2026 Unicode, Inc. (see LICENSE) No installation required Try every function in your browser Each API reference below includes a live form backed by the WASM build. Expand a function, enter its arguments, and inspect the result immediately. Download mojibake-wasm-027.zip Reference C API Functions are organized by their metadata section. Select the plus button to see detailed behavior, examples, specifications, and the live WASM form.

mjb_normalize mjb_status mjb_normalize( const char *buffer, size_t byte_length, mjb_encoding encoding, mjb_normalization form, mjb_encoding output_encoding, mjb_result *result ); Normalize a string to the requested Unicode normalization form. If the input is already normalized and no encoding conversion is needed, the input buffer is returned as-is in result->output with result->transformed set to false, without allocating. Returns MJB_STATUS_OK — The string was normalized (or already normal) MJB_STATUS_INVALID_ARGUMENT — result is NULL, or buffer is NULL with a non-zero size MJB_STATUS_INVALID_FORM — form is not NFC, NFD, NFKC, or NFKD MJB_STATUS_OVERFLOW — The output size would overflow MJB_STATUS_NO_MEMORY — Allocation failed Example const char *input = "Cafe\xCC\x81"; // "Cafe" + U+0301 COMBINING ACUTE ACCENT mjb_result result; if(mjb_normalize(input, strlen(input), MJB_ENC_UTF_8, MJB_NORMALIZATION_NFC, MJB_ENC_UTF_8, &result) != MJB_STATUS_OK) { return 1; } // NFC: Café printf("NFC: %.*s", (int)result.output_size, result.output); if(result.transformed) { mjb_free(result.output); } Related mjb_string_is_normalized mjb_string_filter Specifications UAX #15: Unicode Normalization Forms, Unicode 17.0.0 mjb_string_filter mjb_status mjb_string_filter( const char *buffer, size_t byte_length, mjb_encoding encoding, mjb_filter filters, mjb_encoding output_encoding, mjb_result *result ); MJB_FILTER_LIMIT_COMBINING removes combining marks after the first MJB_FILTER_MAX_COMBINING_MARKS consecutive marks in an emitted run. This is useful for reducing Zalgo-style text while keeping ordinary accents and stacked marks.

Example const char *mixed_whitespace = "Hello\t\t\n\nworld"; mjb_result result; if(mjb_string_filter(mixed_whitespace, strlen(mixed_whitespace), MJB_ENC_UTF_8, MJB_FILTER_COLLAPSE_SPACES, MJB_ENC_UTF_8, &result) != MJB_STATUS_OK) { return 1; } // Filtered: Hello world printf("Filtered: %.*s", (int)result.output_size, result.output); if(result.transformed) { mjb_free(result.output); } const char *controls = "\x1\x2\t\n\v\f\r\x1f"; if(mjb_string_filter(controls, strlen(controls), MJB_ENC_UTF_8, MJB_FILTER_CONTROLS, MJB_ENC_UTF_8, &result) != MJB_STATUS_OK) { return 1; } // Filtered: \t\n\v\f\r printf("Filtered: %.*s", (int)result.output_size, result.output); if(result.transformed) { mjb_free(result.output); } Related mjb_normalize mjb_nfkc_casefold mjb_status mjb_nfkc_casefold( const char *buffer, size_t byte_length, mjb_encoding encoding, mjb_encoding output_encoding, mjb_result *result ); Apply the normative NFKC_Casefold mapping and normalize the result to NFC. This transform performs compatibility folding, full default case folding, and removal of default-ignorable codepoints. It is intended for identifier comparison and is not locale-sensitive.