cache

Keep recent results in memory with a size limit and an expiry time, so slow work is done once.

import "std:cache"

std:cache is an in-memory key-value store that forgets on purpose. It holds at most a set number of entries and drops the least recently used one when it is full, and each entry can expire after a time to live. Use it to avoid repeating slow work: a network lookup, a rendered page, a heavy calculation. A cache lives only as long as the program, so use save or db for data that must survive.

Set and get#

cache.new() makes a cache. set(key, value) stores any value under a string key, and get(key) returns it, or null when there is nothing live under that key:

main.osl

import "std:cache"

auto items = cache.new()
items.set("a", 1)
items.set("b", "two")
items.set("list", [3, 4])

log items.get("a") items.get("b") items.get("list")
log items.get("missing")
log items.size() items.has("a") items.has("missing")
log items.delete("a") items.delete("a")
log items.size()

Click the code to edit it

has tells a missing key from one that holds null. delete returns whether it removed something, and clear() empties the cache.

Limiting the size#

The capacity option caps how many entries the cache holds. Reading or writing an entry marks it as recently used, and a new entry in a full cache pushes out the one used longest ago:

main.osl

import "std:cache"

auto recent = cache.new({capacity: 2})
recent.set("a", 1)
recent.set("b", 2)
recent.get("a")
recent.set("c", 3)

log recent.keys().sort()
log recent.has("b")
log recent.info()

Click the code to edit it

Reading a made b the oldest, so b was the one dropped. The ttl in info() is the default lifetime of an entry: 300000 milliseconds, five minutes. The default capacity is 100.

Expiry#

The ttl option, in milliseconds, makes entries expire. set also takes a ttl of its own for one entry. ttl(key) says how long is left, or null for a key that isn't there. Expired entries behave as if they were never stored:

main.osl

import "std:cache"
import "std:thread"

auto c = cache.new({ttl: 100})
c.set("short", "soon gone")
c.set("long", "stays", {ttl: 60000})

log c.has("short") (c.ttl("short") ?? 0) > 0
thread.sleep(200)
log c.has("short") c.get("short")
log c.get("long")
log c.ttl("missing")

c.expire("long", 50)
thread.sleep(100)
log c.has("long")

Click the code to edit it

expire(key, ms) gives a live entry a new lifetime. Expired entries are removed when next touched, or all at once with cleanupExpired().

Fetch: compute once#

fetch(key, loader) is the call you will use most. It returns the live value under key, and when there is none it calls loader(), stores the answer and returns it. The loader runs only on a miss:

main.osl

import "std:cache"

auto squares = cache.new()
int runs = 0

def slowSquare() int (
  runs = runs + 1
  return 12 * 12
)

log squares.fetch("12", slowSquare)
log squares.fetch("12", slowSquare)
log squares.fetch("12", slowSquare)
log "computed" runs "time"

Click the code to edit it

Caching a server's answers#

Here a handler counts how often it does real work, and a cache with a short life sits in front of it. The first request is a miss, the next is served from the cache, and an expired entry is computed again:

main.osl

import "std:serve"
import "std:requests"
import "std:cache"
import "std:thread"

auto app = serve.new()
auto pages = cache.new({capacity: 100, ttl: 300})
int renders = 0

def render() string (
  renders = renders + 1
  return "report #" ++ renders
)

app.get("/report", def(*serve.Context c) (
  c.text(200, pages.fetch("report", render))
))
app.start("127.0.0.1:8777").unwrap()

for _ in 1 to 2 (
  log requests.get("http://127.0.0.1:8777/report").unwrap().body
)
thread.sleep(400)
log requests.get("http://127.0.0.1:8777/report").unwrap().body

app.stop().unwrap()

Click the code to edit it

It prints report #1 twice and then report #2. The cache is safe to use from several threads at once, so handlers can share one.

Looking inside#

A cache can be read as a whole: keys(), values(), entries() (an object), toArray() (with each entry's expiry time), and forEach, filter and mapValues, which call your function with (key, value), and reduce, which calls it with (result, value):

main.osl

import "std:cache"

auto stock = cache.new()
stock.setMany({apples: 3, pears: 0, plums: 7})

log stock.entries()
log stock.pick(["apples", "kiwis"])
log stock.filter(def(string key, any value) boolean -> value > 0)
log stock.reduce(0, def(any total, any value) any -> total + value)

Click the code to edit it

Quick check

A cache has capacity: 2 and holds a and b. You read a, then store c. Which key is dropped?

Your turn

Count the misses

Look up each key with fetch. The loader should count how many times it had to run. Repeated keys must come from the cache.

main.osl

import "std:cache"

auto lookups = cache.new()
int misses = 0

for key in ["a", "b", "a", "c", "a", "b"] (
  log key
)
log "misses" misses

import "std:cache"

auto lookups = cache.new()
int misses = 0

for key in ["a", "b", "a", "c", "a", "b"] (
  log key
)
log "misses" misses

Ctrl Enter runs

Expected output

a
b
a
c
a
b
misses 3

API reference#

31 members, generated from the std:cache source of OSL 9.3.0.

Functions

#cache.create(any capacity, any ttl) *cache.Cache

Deprecated. Deprecated: use cache.new({capacity: capacity, ttl: milliseconds}) instead.

create makes a cache holding capacity entries that live ttl seconds; invalid values fall back to 100 entries and 300 seconds.

#cache.createDefault() *cache.Cache

Deprecated. Deprecated: use cache.new() instead.

createDefault makes a cache of 100 entries that live 300 seconds.

#cache.new(object? options) *cache.Cache

new creates an empty cache. Options: capacity (the most entries it holds before evicting the least recently used one, default 100) and ttl (how long an entry lives after it is set, in milliseconds, default 300000, five minutes). Throws ValueError for a capacity below 1 or a ttl that isn't above 0.

`*cache.Cache`

#cache.cleanupExpired() int

cleanupExpired removes every expired entry now instead of when it is next read, and returns how many it removed.

#cache.clear()

clear removes every entry from the cache.

#cache.delete(string key) boolean

delete removes key from the cache, returning true when it held a live entry and false when there was nothing to remove.

#cache.deleteMany(string[] keys) int

deleteMany removes every key in keys, an array of strings, and returns how many held a live entry.

#cache.entries() object

entries returns the live entries as an object of key to value, without changing their recency.

#cache.expire(string key, number ttl) boolean

expire gives the live entry under key a new lifetime of ttl milliseconds from now; a ttl of 0 removes it at once. Returns true when key held a live entry and false when there was nothing to change. Throws ValueError for a negative ttl.

#cache.fetch(string key, any loader) any

fetch returns the live value under key, or calls loader() and stores and returns what it gives back when key is missing or expired. Concurrent fetches of one key wait for a single loader call. Throws StateError when the loader fetches its own key, which would otherwise wait forever, and lets an error the loader throws propagate without storing anything.

#cache.filter(any fn) object

filter returns an object of the live entries for which fn(key, value) returns true. The cache is unchanged.

#cache.forEach(any fn)

forEach calls fn(key, value) for each live entry, from most to least recently used.

#cache.get(string key) any

get returns the value stored under key and marks it most recently used, or null when key is missing or has expired. A stored null also reads as null; use has to tell them apart.

#cache.getMany(array keys) object

Deprecated. Deprecated: use cache.Cache.pick(keys) instead.

getMany returns an object holding the live entries among keys.

#cache.getOrSet(any key, any value) any

Deprecated. Deprecated: use cache.Cache.fetch(key, loader) instead, with a loader that returns value.

getOrSet returns the value under key, storing value first when key is missing.

#cache.getOrSetFunc(any key, any fn) any

Deprecated. Deprecated: use cache.Cache.fetch(key, loader) instead.

getOrSetFunc returns the value under key, storing fn() first when key is missing.

#cache.getTTL(any key) int

Deprecated. Deprecated: use cache.Cache.ttl(key) instead, which returns milliseconds or null.

getTTL returns the seconds the entry under key has left, 0 when it has just expired and -1 when it is missing.

#cache.has(string key) boolean

has reports whether key holds a live (unexpired) entry, even one holding null. It doesn't change the entry's recency.

#cache.info() {capacity: int, size: int, ttl: number}

info describes the cache as a record with the keys size (live entries), capacity and ttl (the default lifetime in milliseconds).

#cache.keys() string[]

keys returns the keys of the live entries, in no particular order.

#cache.mapValues(any fn) object

mapValues returns an object of every live key with the value fn(key, value) returns for it. The cache is unchanged.

#cache.pick(string[] keys) object

pick returns an object holding the live entries among keys, an array of strings; missing and expired keys are left out. Each entry read becomes most recently used.

#cache.reduce(any initial, any fn) any

reduce folds the live values, from most to least recently used, into one result: it starts from initial and replaces it with fn(result, value) for each value.

#cache.set(string key, any value, object? options)

set stores value under key, replacing any earlier value, and marks it most recently used. When the cache is full the least recently used entry is evicted. Option: ttl, this entry's lifetime in milliseconds (default: the cache's ttl). Mutates the cache; throws ValueError for a negative ttl.

#cache.setMany(object entries)

setMany stores every key and value of entries, as set would one at a time.

#cache.setTTL(any key, any ttl) boolean

Deprecated. Deprecated: use cache.Cache.expire(key, milliseconds) instead.

setTTL gives the entry under key a new lifetime in seconds.

#cache.size() int

size returns the number of live entries.

#cache.stats() object

Deprecated. Deprecated: use cache.Cache.info() instead, whose ttl is in milliseconds, and cache.Cache.keys() for the keys.

stats describes the cache with the keys size, capacity, ttl (seconds) and keys.

#cache.toArray() {expiresAt: int64, key: string, value: any}[]

toArray returns the live entries, from most to least recently used, as records with the keys key, value and expiresAt (unix milliseconds).

#cache.ttl(string key) int?

ttl returns how long the entry under key has left to live, in whole milliseconds, or null when key is missing or has expired.

#cache.values() array

values returns the values of the live entries, from most to least recently used, without changing their recency.

Edit this page View as Markdown