# encodings/utf16

> Measure, index, order and case-map strings the way JavaScript does, in UTF-16 code units with 1-based positions.

This is the Markdown version of https://osl-lang.org/packages/encodings-utf16. Every page of the OSL docs has one: add .md to its path. https://osl-lang.org/llms.txt lists them all.

```osl
import "std:encodings/utf16"
```

OSL strings count characters: `"😀".len` is `1`. JavaScript, Java, C# and the Windows API count UTF-16 code units, where that emoji takes two. When you port code from those languages or must match offsets and orderings produced by them, this package gives you the same answers.

Use it when you read positions from JavaScript (an editor, a language server, a browser), must match JavaScript's `<`, or need its exact casing. Otherwise the built-in string methods are the right tool.

```osl
import "std:encodings/utf16"
```

The import name is `utf16`.

## Length in code units

Characters in the basic plane take one code unit. Characters outside it, such as emoji, take two (a surrogate pair).

```osl
import "std:encodings/utf16"

string s = "a😀b"
log s.len
log utf16.length(s)
log utf16.length("") utf16.length("héllo")
```

Output:

```text
3
4
0 5
```

Try it: measure `"héllo"` and `"日本語"`. Which are one unit per character?

## Positions

Positions are counted in code units and start at 1, like the rest of OSL. `index` finds where a part first occurs and returns `0` when it doesn't. `at` returns the code unit at a position, and `trim` returns the code units from one position to another, both ends included.

```osl
import "std:encodings/utf16"

string s = "a😀b"
log utf16.index(s, "b") utf16.index(s, "z")
log utf16.at(s, 1) utf16.at(s, 4)
log utf16.trim(s, 2, 3)
log utf16.trim(s, 1, 1)
```

Output:

```text
4 0
a b
😀
a
```

`"b"` is at position 4, not 3, because the emoji before it takes two units. A position that falls in the middle of a pair gives half a character:

```osl
import "std:encodings/utf16"

string half = utf16.at("😀", 1)
log utf16.length(half)
log utf16.join(half, utf16.at("😀", 2)) == "😀"
```

Output:

```text
1
true
```

`join` glues strings together like JavaScript's `+`, so two halves of a pair become the whole character again.

## Ordering

`compare(a, b)` returns `-1`, `0` or `1` by comparing code units, as JavaScript's `<` does. This differs from OSL's own ordering for characters beyond the basic plane: a high surrogate (the first half of an emoji) sorts before characters in the range U+E000 to U+FFFF.

```osl
import "std:encodings/utf16"

log utf16.compare("apple", "banana") utf16.compare("b", "a") utf16.compare("a", "a")

string wide = "\u{FF5E}"
string emoji = "\u{1F600}"
log "OSL:" (wide < emoji)
log "UTF-16:" utf16.compare(wide, emoji)
```

Output:

```text
-1 1 0
OSL: true
UTF-16: 1
```

## Case

`upper` and `lower` apply full Unicode case mapping, like `toUpperCase` and `toLowerCase`. Note that `ß` becomes `SS`, which changes the length.

```osl
import "std:encodings/utf16"

log utf16.upper("straße") utf16.lower("ÀLVARO")
log utf16.length("straße") utf16.length(utf16.upper("straße"))
```

Output:

```text
STRASSE àlvaro
6 7
```

## Case-insensitive checks

Three helpers compare as if both sides were lower-cased: `compareLower`, `containsLower` and `isSameLower`. 

```osl
import "std:encodings/utf16"

log utf16.containsLower("Hello World", "WORLD")
log utf16.compareLower("Apple", "banana")
log utf16.isSameLower("HeLLo", "hello")
```

Output:

```text
true
-1
true
```

`isSameLower(text, lowered)` expects the second argument to be lower-case already, so you can lower-case a search term once and test many strings against it.

**Quiz.** What is `utf16.length("😀")`?

- [ ] 1
- [x] 2
- [ ] 4

Answer: The emoji is outside the basic plane, so it takes two UTF-16 code units. OSL's own `.len` says 1.

### Exercise: Port a JavaScript offset

An editor reports position 4 in `"😀 hi"`. Print the letter there, then the OSL length and the UTF-16 length.

Starting code:

```osl
import "std:encodings/utf16"

string text = "😀 hi"
log text.len
```

It should print:

```text
h
4 5
```

Hint 1: `utf16.at(text, 4)`

Hint 2: `utf16.length(text)`

A solution:

```osl
import "std:encodings/utf16"

string text = "😀 hi"
log utf16.at(text, 4)
log text.len utf16.length(text)
```

## API reference

## API reference

Every member of `std:encodings/utf16`, generated from the source of OSL.

### Functions

#### `utf16.at(string text, int index) string`

at returns the UTF-16 code unit at a 1-based index as a string, like
text[index] for code units: half of a surrogate pair comes back as a lone
surrogate. Returns "" when index is outside the string, as text[index]
does.

#### `utf16.compare(string a, string b) int`

compare orders a and b by UTF-16 code units, as JavaScript's < does, and
returns -1 when a sorts first, 0 when they are equal and 1 when b sorts
first.

#### `utf16.compareLower(string a, string b) int`

compareLower orders a and b as compare(lower(a), lower(b)) does, which is
JavaScript's a.toLowerCase() < b.toLowerCase(), without building either
lowered copy when both are ASCII.

#### `utf16.containsLower(string text, string part) boolean`

containsLower reports whether lower(text) contains lower(part), as
JavaScript's text.toLowerCase().includes(part.toLowerCase()) does, without
building either lowered copy when both are ASCII.

#### `utf16.index(string text, string part) int`

index returns the 1-based code unit position where part first occurs in
text, or 0 when it doesn't occur, as text.index does. An empty part is
found at 1.

#### `utf16.isSameLower(string text, string lowered) boolean`

isSameLower reports whether utf16.lower(text) equals lowered, which is
JavaScript's text.toLowerCase() === lowered, without building the lowered
copy for ASCII text.

#### `utf16.join(string a, string b) string`

join concatenates a and b like JavaScript's +: a lone high surrogate at
the end of a and a lone low surrogate at the start of b become one
character again.

#### `utf16.length(string text) int`

length returns the length of text in UTF-16 code units, which is
JavaScript's string.length: an emoji outside the Basic Multilingual Plane
counts as 2.

#### `utf16.lower(string text) string`

lower returns text in lower case like JavaScript's toLowerCase, with full
Unicode case mapping (final sigma included). Lone surrogates are kept.

#### `utf16.sameLower(string s, string lowered) boolean`

**Deprecated.** Deprecated: use utf16.isSameLower(text, lowered) instead.

sameLower reports whether utf16.lower(s) equals lowered.

#### `utf16.trim(string text, int start, int end) string`

trim returns the code units of text from start to end, both 1-based and
inclusive, like string.trim(start, end) counted in UTF-16 code units. A
negative position counts back from the end (-1 is the last code unit),
positions past either end are clamped, and reversed positions are
swapped, so trim never fails.

#### `utf16.upper(string text) string`

upper returns text in upper case like JavaScript's toUpperCase, with full
Unicode case mapping ("ß" becomes "SS"). Lone surrogates are kept.