std.unicode

std.unicode is a pure dusk decode and encode layer over the string’s existing byte view, added in 0.5.2. It changes nothing about how a string is stored. A string is still a NUL terminated UTF-8 byte buffer, s[i] still reads one byte, and this module is what walks that buffer scalar by scalar. Every function is total: no fault and no unbounded read. A malformed byte or an invalid scalar resyncs to the replacement character U+FFFD rather than stopping the walk. For a task oriented walkthrough see Unicode and runes, and for the rune type itself see the type system page.

@import std.unicode

The module is written in Dusk over the string’s raw byte view and adds no runtime C. It carries six functions:

func decode_rune(s: string, i: int64) -> (rune, int64)
func encode_rune(r: rune, buf: *raw char) -> int64
func rune_len(r: rune) -> int64
func rune_count(s: string) -> int64
func utf8_valid(s: string) -> bool
func sb_push_rune(sb: *StringBuilder, r: rune) -> void

• decode_rune(s, i) decodes one scalar at byte offset i and returns it paired with its encoded width. It is total. The NUL terminator decodes to (0, 0), and any malformed byte, a stray continuation, an overlong lead, a truncated tail, a surrogate, or a scalar above the maximum, resyncs to exactly (0xFFFD, 1), so a caller stepping by the width it gets back always makes forward progress one byte at a time. Its one precondition is that i lies in [0, str_len(s)]; a string is a raw NUL terminated view, so an out of range i is an unchecked read, the same honor system str_len carries.
• encode_rune(r, buf) writes a scalar’s 1 to 4 UTF-8 bytes into a caller sized buffer and returns the count. Size buf to at least 4 bytes. An invalid scalar writes the 3 byte U+FFFD encoding instead.
• rune_len(r) reports the width encode_rune would use, 1 to 4, without writing anything.
• rune_count(s) walks a string end to end and counts scalars, each malformed byte counting as exactly one, so the count never desyncs from decode_rune’s resync.
• utf8_valid(s) runs the identical decode loop and reports whether s is strict, well formed UTF-8. It is invalid only on the resync signature, width 1 paired with the U+FFFD scalar, so it can never drift from what decode_rune itself accepts.
• sb_push_rune(sb, r) appends one scalar’s encoded bytes to a StringBuilder.

The decoder is strict throughout. An overlong encoding, a surrogate, and a scalar above 0x10FFFF are all rejected the same as a truncated or malformed sequence, never silently accepted as some other valid scalar.

Decoding a string

The consumer’s idiom is a decode walk: read the scalar at the current byte, use it, then step forward by the width decode_rune returned.

count.dusk

@paradigm procedural
@import std.unicode

func main() -> int32 {
    s: string = "a中😀"
    println(rune_count(s))     // 3

    mut i: int64 = 0
    while s[i] != 0 {
        r, w := decode_rune(s, i)
        println(r)             // 97, then 20013, then 128512
        i = i + w
    }
    return 0
}

rune_count reports the same three scalars the loop visits, since both run one decode pass. s[i] != 0 stops the loop at the NUL, so decode_rune never reads past the terminator.

Building a string from runes

To go from scalars to bytes, append them to a StringBuilder with sb_push_rune. It encodes each rune’s UTF-8 bytes and pushes them, so the built buffer is ordinary UTF-8 you can view as a string.

build.dusk

@paradigm procedural
@import std.string
@import std.unicode

func main() -> int32 {
    g: *StringBuilder = alloc(sb_new())
    sb_push_rune(g, r'中')
    sb_push_rune(g, r'\u{1F600}')
    println(sb_cstr(g))               // 中😀
    println(rune_count(sb_cstr(g)))   // 2
    sb_free(g)
    free(g)
    return 0
}

For the lower level encode_rune into a *raw char buffer, and the whole rune and \u{...} story, see Unicode and runes.

Out of scope

std.unicode is a decode and encode layer only. Case folding, normalization, and grapheme clustering are not part of it. A rune is a single scalar value, so a glyph a terminal draws from several joined scalars still counts as several runes under rune_count.