The cache that buried people for a week

Prism enriches every meeting attendee from a stack of paid B2B data providers, and those lookups cost real money per call. So we cache hard. A person’s provider docs are good for 30 days, and a person we couldn’t find at all gets negative-cached for 7, because there’s no point paying again to rediscover that a ghost is still a ghost.

Two bugs lived in that caching layer. Both were invisible in tests, both only showed up when we re-prepped a meeting with someone already enriched, and both had the same symptom: a person we successfully enriched last week silently vanished this week.

In this article we’ll go through both, because they chain. The first one is what triggers the second, and the second is what actually does the damage.

Bug one: the fetch dict and the id extractor disagreed

Every enrichment source implements two functions the engine depends on. fetch() returns a dict of provider docs, and extractIds() pulls the provider’s stable IDs out of that dict, so we can write them onto the person row and cache-hit by ID next time.

There’s a contract between those two functions. It was written down absolutely nowhere: the keys of the fetch() dict have to equal what extractIds() returns.

Both the RocketReach and MixRank sources broke it. fetch() keyed its docs by email:

return { [email]: doc }   // wrong

while extractIds() returned the provider’s numeric person ID.

so watch what that does over two preps. On the first prep everything works. We fetch, file the doc under the email key, write the numeric ID onto person.rocketreachIds, everything green, rep gets a brief.

On the next prep the engine reads that ID back off the person row and does an ID-keyed cache lookup. Which misses, because the doc isn’t filed under the ID — it’s filed under the email. A miss sets needsFetch, and a refetch walks straight into bug two.

The fix is a single line, keying by the provider ID so the doc ID lines up with everything else that references it:

// src/enrichment/sources/rocketreach/source.ts
const doc = await deps.runLookup(ctx)
if (!doc || !meetsMinimumDoc(doc)) return {} // Hard-empty
// Key by the provider id so the docId aligns with extractIds + person.rocketreachIds + source_cache
// (else the id-keyed cache lookup misses on re-prep and the negative cache suppresses the person).
const key = doc.id || email || `linkedin:${linkedin}`
return { [key]: doc } // Found

One line, but you can only see it once the contract has a name. It’s now a comment at both source implementations and the first thing I check when writing a new one.

and it’s worth knowing why that’s the safe default rather than just a tidy one. A source that doesn’t implement extractIds falls back to Object.keys(data) for its IDs, which means the dict key literally is the ID. Keying by doc.id only makes explicit what the fallback already assumes.

Bug two: a transient 429 poisoned the seven-day cache

The negative cache runs off a single timestamp on the person row. If lastExternalSearchAt is inside the last 7 days, we skip the provider call entirely and save the money:

// src/enrichment/data-source.ts
export const NEGATIVE_CACHE_DAYS = 7

if (needsFetch && ctx.lastExternalSearchAt) {
  const cutoff = new Date(Date.now() - NEGATIVE_CACHE_DAYS * DAY_MS)
  if (ctx.lastExternalSearchAt > cutoff) needsFetch = false
}

The original code stamped that timestamp whenever we attempted an external fetch. Which sounds completely reasonable, and it is, right up until you think about what an attempt can do other than succeed. A 429 from the provider. A timeout. A blip.

Any of those stamped the timestamp. And from that moment the negative cache believes we looked and found nothing, and refuses to look again for a week. A rate limit on Tuesday meant no data on that prospect until the following Tuesday.

and because it’s a cache doing precisely what a cache is supposed to do, nothing anywhere logs an error. There’s no stack trace, no failed job, no alert. The person is simply not in the brief.

The fix is to move the flag to after fetch() returns, so only a call that actually completed is ever eligible to stamp anything:

// src/enrichment/data-source.ts
let attemptedExternalFetch = false
if (needsFetch) {
  try {
    fetched = await source.fetch(ctx)
    attemptedExternalFetch = true // completed → a definitive answer (found or hard-empty); eligible to stamp neg-cache
    if (Object.keys(fetched).length) await storeCached(source.name, fetched)
  } catch (err) {
    // Transient/operational failure: leave attemptedExternalFetch=false so the negative cache is NOT stamped.
    if (Object.keys(cached.data).length === 0) {
      return {
        sourceName: source.name,
        data: {},
        success: false,
        errorMessage: err instanceof Error ? err.message : String(err),
        attemptedExternalFetch,
      }
    }
    // otherwise fall through and serve the cached data we already have
  }
}

A thrown error now leaves the flag false, the timestamp is never written, and the provider gets retried on the very next prep.

That pushed the three outcomes into the source contract itself, which is the part I’d actually keep from this whole episode. A source has to say which of three things happened, and they are genuinely different things:

/**
 * Three-outcome contract (design §7):
 *   Found       → { [key]: doc }  (cached 30d)
 *   Hard-empty  → {}              (engine stamps the 7-day negative cache)
 *   Operational → throw           (engine does NOT stamp; retried next prep)
 */

Why tests didn’t catch either one

Both bugs need two runs to exist. Bug one is fine on the first prep and only misfires when something reads an ID back off a row written by a previous prep. Bug two needs the first run to have failed transiently. Every test we had ran the path once, cleanly, and every one of them passed.

A negative cache is a claim about knowledge. An exception is the absence of one.

That’s the thing I’d actually take from this. A stamped timestamp says we looked, and there was nothing there. A thrown error says we don’t know yet. Treating the second as the first means every transient blip hardens into a week-long lie, and in a caching layer you will not find out until somebody runs the exact same path seven days later and wonders where the attendee went.

If you want the other half of what these providers put us through, the identity side of it is over in same email, two different people.

the cache that buried people for a week.

Bug one: the fetch dict and the id extractor disagreed

Bug two: a transient 429 poisoned the seven-day cache

Why tests didn’t catch either one

Further reading