SKS Video Diary
backendworkflowsinngesttemporal

Why durable workflows exist

Sharing a resource with 300 people is a chain of side effects across services. A database transaction can't protect that chain. A durable workflow can.

Open in teleprompter

Some software concepts only click once you have run something in production and watched it fail. Durable workflows are one of them. On a laptop, a request handler that does five things in a row works every time. In production, the interesting question is what happens when it stops halfway.

The chain

Take a small feature: share a private resource with 300 people. That is not one operation. It is a chain that all has to succeed: write the share rows, grant 300 permissions in your permission system, invalidate the cache, and send 300 emails.

One request, four side effects, one crash
Request handlerPOST /share · 300 users
Write sharesdatabase
Permissions× 300 users
Cacheinvalidate
never runs
Emails× 300
never runs
Server crashed at step 2
0.0s/0.0s

Put that chain inside a request handler and it runs fine, right up until the process dies partway through. A deploy, an OOM kill, a node getting recycled. Now the shares are in the database, half the permissions are granted, and nobody gets an email. Nothing is set up to notice, let alone resume.

The naive handler
app/api/share/route.tsts
export async function POST(req: Request) {
  const { resourceId, userIds } = await req.json()
 
  await db.transaction((tx) =>
    tx.insert(shares).values(rows(userIds))
  )
  for (const id of userIds) {
    await permissions.grant(id, resourceId)
  }
  await cache.invalidate(resourceId)
  for (const id of userIds) {
    await email.send(id, "You got access")
  }
  return Response.json({ ok: true })
}
0.0s/0.0s

A transaction is not enough

The first instinct is "wrap it in a transaction". A database transaction protects the changes inside it: either all the share rows land or none do. It says nothing about the permission system, the cache, or the email provider. Those are other services with their own state, and your rollback never reaches them.

What the transaction actually covers
db.transaction
Write sharesinsert 300 rows
✓
all-or-nothing
Permission system
outside
Cache provider
outside
Email provider
outside
0.0s/0.0s
Protected vs. not protected
app/api/share/route.tsts
export async function POST(req: Request) {
  const { resourceId, userIds } = await req.json()
 
  await db.transaction((tx) =>
    tx.insert(shares).values(rows(userIds))
  )
  for (const id of userIds) {
    await permissions.grant(id, resourceId)
  }
  await cache.invalidate(resourceId)
  for (const id of userIds) {
    await email.send(id, "You got access")
  }
  return Response.json({ ok: true })
}
inside the transaction
external systems, no rollback
0.0s/0.0s

Durable workflow

This is where a durable workflow comes in. You get one from a workflow engine such as Inngest or Temporal. The idea is simple: the workflow remembers how far it got, and it stores that progress durably, outside your application process. If the process dies, the progress does not.

The request no longer does the work. It hands the job to the engine and returns an acknowledgement right away, with an ID the client can use to check progress.

The API only accepts the job
app/api/share/route.tsts
export async function POST(req: Request) {
  const body = await req.json()
 
  const { ids } = await inngest.send({
    name: "resource/share",
    data: body,
  })
 
  return Response.json(
    { workflowId: ids[0] },
    { status: 202 }
  )
}
0.0s/0.0s

The chain itself becomes a function made of steps. Each step is checkpointed once it finishes.

From handler to steps
inngest/share-resource.tsts
export async function POST(req: Request) {
  const { resourceId, userIds } = await req.json()
export const shareResource = inngest.createFunction(
  { id: "share-resource" },
  { event: "resource/share" },
  async ({ event, step }) => {
    const { resourceId, userIds } = event.data
 
  await db.transaction((tx) =>
    tx.insert(shares).values(rows(userIds))
  )
  for (const id of userIds) {
    await permissions.grant(id, resourceId)
    await step.run("write-db", () =>
      db.insert(shares).values(rows(userIds))
    )
    await step.run("grant-permissions", () =>
      grantAll(userIds, resourceId)
    )
    await step.run("warm-cache", () =>
      cache.invalidate(resourceId)
    )
    await step.run("send-emails", () =>
      emailAll(userIds, "You got access")
    )
  }
  await cache.invalidate(resourceId)
  for (const id of userIds) {
    await email.send(id, "You got access")
  }
  return Response.json({ ok: true })
}
)
0.0s/0.0s

Now assume the server crashes during the permission updates. When it comes back, the engine schedules a retry. It does not start from zero. The database step is already recorded as done, so only the unfinished permission step runs again, then the rest of the chain continues.

Crash, restart, resume from the saved step
APIPOST /share
202 · id: wf_8f3a
Server crashed
Server back · retry scheduled
resumed from step 2
Workflow engine · Inngest / Temporal
write-db
grant-permissions
retry this step only
warm-cache
send-emails
progress store
outside the process
write-db ✓
grant-permissions ✓
warm-cache ✓
send-emails ✓
0.0s/0.0s

Don't make the client wait for emails

The work is now recoverable, but there is still a problem. If you share with thousands of people, why should the client wait for every email? Once the database, permissions and cache are done, access is ready. Notifications can keep going in the background.

So you durably hand that part to a separate notification workflow. The handoff is itself a step, so it cannot get lost either.

Hand off instead of looping
inngest/share-resource.tsts
export const shareResource = inngest.createFunction(
  { id: "share-resource" },
  { event: "resource/share" },
  async ({ event, step }) => {
    const { resourceId, userIds } = event.data
 
    await step.run("write-db", () =>
      db.insert(shares).values(rows(userIds))
    )
    await step.run("grant-permissions", () =>
      grantAll(userIds, resourceId)
    )
    await step.run("warm-cache", () =>
      cache.invalidate(resourceId)
    )
    await step.run("send-emails", () =>
      emailAll(userIds, "You got access")
    )
    // access is ready — emails are another workflow
    await step.sendEvent("notify", {
      name: "share/notify",
      data: { resourceId, userIds },
    })
  }
)
0.0s/0.0s
Two workflows, two timelines
share workflow
write-db
permissions
cache
access ready → respond
client unblocked
step.sendEvent("share/notify")
notification workflow
provider rate limit · waiting
temporary failure · retrying
0.0s/0.0s

The email workflow owns its own problems: sending, waiting out the provider's rate limits, and retrying temporary failures. None of that touches the request that started it.

The notification workflow
inngest/notify-shared.tsts
export const notifyShared = inngest.createFunction(
  {
    id: "notify-shared",
    retries: 5,
    throttle: { limit: 100, period: "1m" },
  },
  { event: "share/notify" },
  async ({ event, step }) => {
    for (const id of event.data.userIds) {
      await step.run(`email-${id}`, () =>
        email.send(id, "You got access")
      )
    }
  }
)
0.0s/0.0s

Next time: across microservices

Everything above assumed one application talking to a few providers. What changes when the sharing, permission and notification logic live in different services, each with its own database and deploys? That is the next one.

Teaser
Sharingown db · own deploys
?
Permissionsown db · own deploys
?
Notificationsown db · own deploys
next video
0.0s/0.0s