Caching #

Memoize a single effect with Effect.cached, add expiry and invalidation, and build a keyed Cache that shares in-flight lookups.

The problem. Every app gets a hand-made cache at some point. It usually starts as a Map and a timestamp:

const cache = new Map<string, { value: User; expiresAt: number }>()

async function getUser(id: string) {
  const hit = cache.get(id)
  if (hit && hit.expiresAt > Date.now()) return hit.value
  const user = await db.load(id)                       // two callers miss at once -> two loads
  cache.set(id, { value: user, expiresAt: Date.now() + 60_000 })
  return user
}

This code looks correct until 2 requests miss at the same moment. Both see an empty slot, both call db.load, and the second write replaces the first. Under load, 1 expired key becomes 100 identical database calls. Then you add a "pending" map to deduplicate the active loads. Then you add a way to invalidate, then a size limit. Now the cache is the most fragile concurrency code in the service.

The shift

Today you think of a cache as a data structure that you check before you do the work. Effect asks you to think of it as an effect that knows how to do the work. Cache.get(cache, key) returns an Effect. When the key is missing, it starts the lookup on its own fiber and stores that fiber, not the finished value. A second caller that arrives during the lookup gets the same fiber and waits for it. 2 callers can never both decide to load, because the decision and the load are 1 atomic step inside the runtime.

Effect.cached is the same idea without a key. It memoizes any effect: it stores the result of the first run and returns that result to all later callers. The effect runs at most 1 time, also when concurrent callers use it. Expiry and invalidation are variants of the same function. They are not extra state that you manage.

	Plain memoize / `Map`	`Effect.cached` family	`Cache` module
Shape	Mutable variables	1 effect, memoized	Key to value, with a lookup function
Concurrent misses	Race condition, duplicate work	Shared, runs once	Shared per key, runs once
Expiry	Manual timestamps	`cachedWithTTL`	`timeToLive` option
Invalidation	`map.delete`	`cachedInvalidateWithTTL`	`Cache.invalidate` / `refresh`
Size limit	Manual eviction	Not needed, 1 value	`capacity`, removes the oldest entry
Failures	Usually not cached	Cached until expiry	Cached until expiry

In this section you will:

Memoize 1 effect.
Give it a lifetime and a manual invalidation.
Build a keyed cache.
Watch a counter to see when the lookup runs.

Learn #

Lesson 1. Effect.cached: run at most once #

The plain TypeScript memoize that you wrote before looks like this:

let memo: Promise<Config> | undefined
function loadConfigOnce() {
  if (!memo) memo = loadConfig()      // stores the Promise so concurrent callers share it
  return memo
}

Effect.cached(effect) does the same job, with 1 difference that causes mistakes: it returns an Effect<Effect<A>>. The outer effect creates the memoized version (it allocates the storage). You yield* it 1 time to get the inner effect. The inner effect is the one that you run many times. In the program below, calls counts how often the expensive body runs. 3 concurrent runs plus 1 later run cause only 1 run of the body.

import { Effect, Ref } from "effect"

const program = Effect.gen(function* () {
  const calls = yield* Ref.make(0)

  const loadConfig = Effect.gen(function* () {
    yield* Ref.update(calls, (n) => n + 1)   // count real executions
    yield* Effect.sleep("5 millis")          // pretend this is slow
    return { port: 8080 }
  })

  // Plain effect: every run does the work again
  yield* loadConfig
  yield* loadConfig
  console.log("plain: ran", yield* Ref.get(calls), "times")

  // cached: the outer yield* builds the memoized effect once
  yield* Ref.set(calls, 0)
  const cachedConfig = yield* Effect.cached(loadConfig)

  const results = yield* Effect.all([cachedConfig, cachedConfig, cachedConfig], { concurrency: "unbounded" })
  const again = yield* cachedConfig
  console.log("cached: ran", yield* Ref.get(calls), "time, ports:", results.map((c) => c.port).join(","), again.port)
})

Effect.runPromise(program)

⌘/Ctrl + Enter

Press Run (or ⌘/Ctrl+Enter in the editor).

Notice

None of the 3 concurrent callers ran the body a second time. The first run was in progress, and the other callers waited for it. Try to remove yield* before Effect.cached. The compiler reports an error where you use the result, because you then hold the builder effect instead of the memoized effect.

Lesson 2. Lifetimes and a kill switch: cachedWithTTL, cachedInvalidateWithTTL #

A value that is cached forever is only correct for data that never changes. A TTL (time to live) is the time that a cached value stays valid. The family has 3 members:

Function	Returns	Recomputes when
`Effect.cached(e)`	`Effect<Effect<A>>`	Never
`Effect.cachedWithTTL(e, ttl)`	`Effect<Effect<A>>`	After `ttl` has passed since the last computation
`Effect.cachedInvalidateWithTTL(e, ttl)`	`Effect<[Effect<A>, Effect<void>]>`	After `ttl`, or when you run the second effect

You write durations as strings such as "20 millis" or "1 hour". The program below uses a very short TTL and a sleep that is longer than the TTL. Then it counts the calls. It never prints times, so the output is stable.

import { Effect, Ref } from "effect"

const program = Effect.gen(function* () {
  const calls = yield* Ref.make(0)
  const version = yield* Ref.make(0)
  const load = Effect.gen(function* () {
    yield* Ref.update(calls, (n) => n + 1)
    return "v" + (yield* Ref.updateAndGet(version, (n) => n + 1))
  })

  // TTL: fresh for 20ms, then recomputed on the next read
  const withTtl = yield* Effect.cachedWithTTL(load, "20 millis")
  console.log(yield* withTtl, yield* withTtl, "calls:", yield* Ref.get(calls))
  yield* Effect.sleep("40 millis")
  console.log(yield* withTtl, "calls after expiry:", yield* Ref.get(calls))

  // Invalidate: a long TTL plus a manual reset
  yield* Ref.set(calls, 0)
  const [cfg, invalidate] = yield* Effect.cachedInvalidateWithTTL(load, "1 hour")
  console.log(yield* cfg, yield* cfg, "calls:", yield* Ref.get(calls))
  yield* invalidate                       // next read recomputes even though 1 hour has not passed
  console.log(yield* cfg, "calls after invalidate:", yield* Ref.get(calls))
})

Effect.runPromise(program)

⌘/Ctrl + Enter

Press Run (or ⌘/Ctrl+Enter in the editor).

Notice

invalidate does not recompute the value. It only removes the stored value. The next read recomputes the value, so nothing runs if nobody reads. Try to call invalidate 2 times in a row. The count still shows 1 extra call.

Lesson 3. Cache: many keys, one lookup function #

When the values depend on a key, use the Cache module. Cache.make takes a lookup function that produces a value for a key, a capacity, and an optional timeToLive. A miss is a read of a key that the cache does not hold. A hit is a read that the cache answers from a stored entry. Then:

Function	Does
`Cache.get(cache, key)`	Returns the cached value, or runs `lookup` on a miss
`Cache.has(cache, key)`	Returns `true` if an entry exists and has not expired. It does not run a lookup
`Cache.invalidate(cache, key)`	Removes 1 entry
`Cache.size(cache)`	Returns the number of stored entries

There is no built-in hit counter. The program counts the lookups in a Ref and computes the hits as reads minus lookups. Use this pattern each time you must prove that a cache works.

import { Cache, Effect, Ref } from "effect"

const usersTable = new Map([[1, "Ada"], [2, "Lin"], [3, "Sam"]])

const program = Effect.gen(function* () {
  const lookups = yield* Ref.make(0)

  const cache = yield* Cache.make({
    capacity: 100,
    lookup: (id: number) =>
      Effect.gen(function* () {
        yield* Ref.update(lookups, (n) => n + 1)   // a lookup means a cache miss
        return usersTable.get(id) ?? "?"
      })
  })

  const reads = [1, 2, 1, 1, 3, 2]
  const names: Array<string> = []
  for (const id of reads) names.push(yield* Cache.get(cache, id))

  const misses = yield* Ref.get(lookups)
  console.log(names.join(","))
  console.log("reads:", reads.length, "misses:", misses, "hits:", reads.length - misses)
  console.log("has 1:", yield* Cache.has(cache, 1), "has 9:", yield* Cache.has(cache, 9), "size:", yield* Cache.size(cache))

  yield* Cache.invalidate(cache, 1)
  yield* Cache.get(cache, 1)                     // miss again after invalidation
  console.log("after invalidate, misses:", yield* Ref.get(lookups))
})

Effect.runPromise(program)

⌘/Ctrl + Enter

Press Run (or ⌘/Ctrl+Enter in the editor).

Notice

Cache.has never starts a lookup, so it is safe for a check of the type "is this key stored?". Cache.get always produces a value, and it runs the lookup when necessary. Use the function that matches your intent.

Lesson 4. Concurrent misses share one lookup #

This is the case that the Map version got wrong. 3 fibers ask for the same key at the same instant, and the cache is empty. With a plain Map:

// all three run this before any of them has stored a value
if (!map.has(key)) map.set(key, await load(key))   // three loads

With Cache, the first get creates an entry that holds the lookup fiber. It stores the entry immediately, before the lookup finishes. The second and the third get find that entry and wait for the same fiber. 1 load, 3 results. The cache stores failures in the same way. If the lookup fails, every caller that waits gets the same failure. Later reads also see the failure until it expires or you invalidate it.

import { Cache, Effect, Ref } from "effect"

const program = Effect.gen(function* () {
  const lookups = yield* Ref.make(0)

  const cache = yield* Cache.make({
    capacity: 10,
    lookup: (id: number) =>
      Effect.gen(function* () {
        yield* Ref.update(lookups, (n) => n + 1)
        yield* Effect.sleep("10 millis")     // slow enough that all three callers arrive during it
        return "user-" + id
      })
  })

  // Three concurrent readers of the same missing key
  const results = yield* Effect.all(
    [Cache.get(cache, 7), Cache.get(cache, 7), Cache.get(cache, 7)],
    { concurrency: "unbounded" }
  )
  console.log(results.join(","))
  console.log("lookups for 3 concurrent gets:", yield* Ref.get(lookups))

  // Different keys still run their own lookups, concurrently
  yield* Effect.all([Cache.get(cache, 8), Cache.get(cache, 9)], { concurrency: "unbounded" })
  console.log("lookups after two new keys:", yield* Ref.get(lookups))
})

Effect.runPromise(program)

⌘/Ctrl + Enter

Press Run (or ⌘/Ctrl+Enter in the editor).

Notice

Change the sleep to "0 millis". The count is still 1. The shared lookup does not depend on the exact time of the calls. It comes from the fact that the cache stores the fiber before the lookup starts.

Lesson 5. Capacity, time to live, and refresh #

2 more options and 1 more operation remain.

capacity is the maximum number of entries. When a new entry pushes the cache over the limit, the cache removes the oldest entries. A read moves an entry to the "newest" end, so this is a least-recently-used policy.

timeToLive makes entries expire. An expired entry counts as missing: has returns false, and get runs the lookup again.

Cache.refresh(cache, key) always runs the lookup and replaces the entry. Use it when you know that the data changed and you want the new value now. invalidate waits for the next reader.

import { Cache, Effect, Ref } from "effect"

const program = Effect.gen(function* () {
  const lookups = yield* Ref.make(0)
  const cache = yield* Cache.make({
    capacity: 2,                    // only two entries fit
    timeToLive: "20 millis",
    lookup: (id: number) =>
      Effect.gen(function* () {
        const n = yield* Ref.updateAndGet(lookups, (c) => c + 1)
        return "user-" + id + "#" + n   // the suffix shows which lookup produced it
      })
  })

  yield* Cache.get(cache, 1)
  yield* Cache.get(cache, 2)
  yield* Cache.get(cache, 3)      // capacity 2: entry 1 is the oldest and gets evicted
  console.log("has 1:", yield* Cache.has(cache, 1), "has 3:", yield* Cache.has(cache, 3), "size:", yield* Cache.size(cache))

  console.log(yield* Cache.get(cache, 3), "(cached)")
  yield* Effect.sleep("40 millis")
  console.log(yield* Cache.get(cache, 3), "(expired, looked up again)")

  console.log(yield* Cache.refresh(cache, 3), "(refresh forces a lookup)")
  console.log(yield* Cache.get(cache, 3), "(and the refreshed value is what get returns)")
})

Effect.runPromise(program)

⌘/Ctrl + Enter

Press Run (or ⌘/Ctrl+Enter in the editor).

Notice

The #n suffix is a useful technique. Put the call number in the value, and you can tell hits from misses without a clock. Try capacity: 3. has 1 then becomes true.

Do and don't #

Do	Don't	Why
`yield*` `Effect.cached(effect)` once, outside loops and functions, and run the inner effect many times.	Do not call `yield* Effect.cached(effect)` inside a loop.	The outer effect allocates new empty storage each time it runs, so the expensive body runs on every iteration.
Use `Effect.cachedInvalidateWithTTL` when your code knows the moment that the data changes.	Do not depend on a TTL alone to see an update that your own code made.	A TTL is an estimate, and the cache returns old data until the TTL passes.
Use the `Cache` module when the values depend on a key.	Do not check a `Map` and then load the value in 2 separate steps.	2 concurrent misses both load the value, and under load 1 expired key becomes many identical calls.
Use `Cache.makeWith` with a `timeToLive` function that returns `Duration.zero` for failures.	Do not store a transient failure for the full TTL.	The cache returns the stored failure to every reader until it expires, so 1 outage becomes a cache that stays failed.
Set a capacity that holds the number of keys that the program reads often.	Do not use a capacity that is smaller than the number of keys in rotation.	Each new key removes an older key, so every read is a miss, and the cache only adds overhead.
Use `Cache.has` for a check that must not start a lookup.	Do not use `Cache.get` to check whether a key is stored.	`Cache.get` runs the lookup on a miss, so the check does work and stores a value.
Prove that a cache works with a lookup counter in a `Ref`, or with the call number in the value.	Do not print times or depend on timestamps in a check.	Checks that depend on the clock are unreliable, and counts are deterministic.

Fix it #

Each program below is broken or incomplete. Make it print the expected output with zero type errors. Use hints before the solution.

1. The cache that thrashes #

The program reads 2 keys in turn, but every read is a miss. Change 1 number so that the program prints lookups: 2.

expected output: lookups: 2

import { Cache, Effect, Ref } from "effect"

const program = Effect.gen(function* () {
  const lookups = yield* Ref.make(0)
  const cache = yield* Cache.make({
    capacity: 1,
    lookup: (id: number) => Ref.update(lookups, (n) => n + 1).pipe(Effect.as("user-" + id))
  })

  for (const id of [1, 2, 1, 2, 1, 2]) yield* Cache.get(cache, id)
  console.log("lookups:", yield* Ref.get(lookups))
})

Effect.runPromise(program)

⌘/Ctrl + Enter

Press Run (or ⌘/Ctrl+Enter in the editor).

2. Memoized three times #

The program uses Effect.cached, but the expensive effect runs once per iteration. Fix it so that the loop prints 42,42,42 and runs: 1.

expected output: 42,42,42 runs: 1

import { Effect, Ref } from "effect"

const program = Effect.gen(function* () {
  const runs = yield* Ref.make(0)
  const expensive = Ref.update(runs, (n) => n + 1).pipe(Effect.as(42))

  const values: Array<number> = []
  for (let i = 0; i < 3; i++) {
    const memo = yield* Effect.cached(expensive)
    values.push(yield* memo)
  }
  console.log(values.join(","))
  console.log("runs:", yield* Ref.get(runs))
})

Effect.runPromise(program)

⌘/Ctrl + Enter

Press Run (or ⌘/Ctrl+Enter in the editor).

3. Holding the builder #

The program does not compile. Fix the 1 line that is wrong so that the program prints total: 84 and runs: 1. Do not change the console.log lines.

expected output: total: 84 runs: 1

import { Effect, Ref } from "effect"

const program = Effect.gen(function* () {
  const runs = yield* Ref.make(0)
  const expensive = Ref.update(runs, (n) => n + 1).pipe(Effect.as(42))

  const memo = Effect.cached(expensive)
  const a = yield* memo
  const b = yield* memo
  console.log("total:", a + b)
  console.log("runs:", yield* Ref.get(runs))
})

Effect.runPromise(program)

⌘/Ctrl + Enter

Press Run (or ⌘/Ctrl+Enter in the editor).

4. Stale after update #

The config is cached for 1 hour. But after bumpVersion runs, the next read must return the new version. Change the cache setup so that the program prints config v1, config v1, config v2 on 3 lines.

expected output: config v1 config v1 config v2

import { Effect, Ref } from "effect"

const program = Effect.gen(function* () {
  const version = yield* Ref.make(1)
  const loadConfig = Ref.get(version).pipe(Effect.map((v) => "config v" + v))
  const bumpVersion = Ref.update(version, (v) => v + 1)

  const config = yield* Effect.cachedWithTTL(loadConfig, "1 hour")

  console.log(yield* config)
  console.log(yield* config)
  yield* bumpVersion
  console.log(yield* config)
})

Effect.runPromise(program)

⌘/Ctrl + Enter

Press Run (or ⌘/Ctrl+Enter in the editor).

5. The failure that stuck #

The database is down for the first lookup only. The cache stores that failure, so the second read also fails. Make failed lookups expire immediately, and keep the successes. The program must print attempt 1: db down, attempt 2: user-1, lookups: 2. Keep the Cache module.

expected output: attempt 1: db down attempt 2: user-1 lookups: 2

import { Cache, Duration, Effect, Exit, Ref, Result } from "effect"

const program = Effect.gen(function* () {
  const lookups = yield* Ref.make(0)

  const cache = yield* Cache.make({
    capacity: 10,
    lookup: (id: number) =>
      Effect.gen(function* () {
        const n = yield* Ref.updateAndGet(lookups, (c) => c + 1)
        if (n === 1) return yield* Effect.fail("db down")
        return "user-" + id
      })
  })

  for (const attempt of [1, 2]) {
    const r = yield* Effect.result(Cache.get(cache, 1))
    console.log("attempt " + attempt + ":", Result.isSuccess(r) ? r.success : r.failure)
  }
  console.log("lookups:", yield* Ref.get(lookups))
})

Effect.runPromise(program)

⌘/Ctrl + Enter

Press Run (or ⌘/Ctrl+Enter in the editor).

6. A cache that can fail #

safeName declares that it cannot fail, but the cache lookup can fail. The program does not compile. Fix safeName so that it keeps its declared type and returns "unknown" for failed lookups. The program must print Ada and then unknown.

expected output: Ada unknown

import { Cache, Effect } from "effect"

const usersTable = new Map([[1, "Ada"]])

const program = Effect.gen(function* () {
  const cache = yield* Cache.make({
    capacity: 10,
    lookup: (id: number) => {
      const name = usersTable.get(id)
      return name === undefined ? Effect.fail("no user " + id) : Effect.succeed(name)
    }
  })

  const safeName = (id: number): Effect.Effect<string> => Cache.get(cache, id)

  console.log(yield* safeName(1))
  console.log(yield* safeName(2))
})

Effect.runPromise(program)

⌘/Ctrl + Enter

Press Run (or ⌘/Ctrl+Enter in the editor).

Build it #

Write the program from the spec. The output must match exactly.

1. Feature flags with manual refresh #

A service reads feature flags from a slow source. The cache must keep the flags for 1 hour, but an admin endpoint can force a reload. Build it with Effect.cachedInvalidateWithTTL.

Requirements:

loadFlags increments loads (a Ref<number>) and returns "flags v<n>", where n is the number of loads so far. Use Ref.updateAndGet.
program creates the cached pair with a TTL of "1 hour". Then it reads 2 times, runs the invalidate effect, prints invalidated, reads 1 more time, and prints the load count.
Each read prints read: <value>.

Exact output:

read: flags v1
read: flags v1
invalidated
read: flags v2
loads: 2

expected output: read: flags v1 read: flags v1 invalidated read: flags v2 loads: 2

import { Effect, Ref } from "effect"

const program = Effect.gen(function* () {
  const loads = yield* Ref.make(0)

  // TODO: loadFlags: bump loads and return "flags v<n>"

  // TODO: const [flags, invalidate] = yield* Effect.cachedInvalidateWithTTL(...)

  // TODO: read twice, invalidate (print "invalidated"), read once, print "loads: N"
})

Effect.runPromise(program)

⌘/Ctrl + Enter

Press Run (or ⌘/Ctrl+Enter in the editor).

2. Product price cache #

Build a keyed price cache in front of a price table. Prove that duplicate concurrent reads share 1 lookup. Then apply a price change with invalidation.

Requirements:

priceTable is a mutable Map<string, number> with A: 10, B: 20, C: 30.
Create a Cache with capacity 100. Its lookup increments lookups and reads the table (all keys exist).
Read ["A", "B", "A", "C"] concurrently. Print the prices joined by , , then lookups: 3 and size: 3.
Change A to 11 in the table, invalidate A, and print price update for A.
Read ["A", "B"] again and print the prices and the lookups.

Exact output:

prices: 10, 20, 10, 30
lookups: 3
size: 3
price update for A
prices: 11, 20
lookups: 4

expected output: prices: 10, 20, 10, 30 lookups: 3 size: 3 price update for A prices: 11, 20 lookups: 4

import { Cache, Effect, Ref } from "effect"

const priceTable = new Map([["A", 10], ["B", 20], ["C", 30]])

const program = Effect.gen(function* () {
  const lookups = yield* Ref.make(0)

  // TODO: cache with capacity 100; lookup bumps lookups and reads priceTable

  // TODO: read A, B, A, C concurrently; print "prices: ...", "lookups: N", "size: N"

  // TODO: priceTable.set("A", 11), invalidate "A", print "price update for A"

  // TODO: read A, B; print "prices: ...", "lookups: N"
})

Effect.runPromise(program)

⌘/Ctrl + Enter

Press Run (or ⌘/Ctrl+Enter in the editor).

Recall #

Answer in your head first, then reveal. Come back to these tomorrow.

Why does `Effect.cached(effect)` return `Effect<Effect<A>>` and not `Effect<A>`? #

The outer effect allocates the memo storage. When you run it once, you get the inner, memoized effect. If you run the outer effect in a loop or inside a function, you get a new empty cache each time. This is the most common mistake with Effect.cached.

What is the type of `yield* Effect.cachedInvalidateWithTTL(load, "1 hour")` if `load: Effect<Config, LoadError>`? #

[Effect<Config, LoadError>, Effect<void>]: the cached effect, which keeps the error type of load, and an invalidate effect that cannot fail.

3 fibers call `Cache.get` for the same missing key at the same time. How many lookups run? #

The first get stores the lookup fiber in the entry before the lookup finishes. The other 2 find the entry and wait for that fiber. A plain Map cache cannot do this without extra 'pending' state.

The data changed, and the cache must hold the new value immediately, not on the next read. Which function do you use? #

Cache.refresh(cache, key). It runs the lookup immediately and replaces the entry. Cache.invalidate only removes the entry, so the next reader pays for the lookup.

Does `Cache` store failed lookups? #

Yes. The cache stores the Exit, success or failure, until it expires or you invalidate it. To expire failures faster, use Cache.makeWith with a timeToLive function. The function inspects the Exit and returns Duration.zero for failures.

How do you prove that a cache works without timestamps? #

Count the lookups in a Ref inside the lookup function, or put the call number in the returned value ("user-3#4"). Reads minus lookups gives the hits. Checks that depend on the clock are unreliable. Counts are deterministic.

← Batching Concurrency →