Build-along: give Apple Notes an AI summary + action-item pipeline

You're an engineer on the Apple Notes team, and you want to ship one small button: Summarize. A user opens a long, messy note — a meeting, a brain-dump, a phone call scrawled in real time — taps it, and gets a tidy summary pinned to the top, the points that actually matter, and a short list of things the app can do for them. That's the feature. Let's build the AI half of it in Noukai.

We'll keep the scope honest. Today it's text notes only — no audio yet — and the app suggests actions rather than firing them. Voice-memo transcription and the agentic 'save it to my calendar' version are each their own post. What you'll have at the end is a pipeline you could point a real Summarize button at.

The naive version, and why it cracks

The tempting first move is one big prompt: 'Here's a note — summarize it, pull the key points, and list any actions.' It works in the demo. Then it meets real notes. When the actions come out wrong, you can't tell which instruction failed, because all three are tangled in one call. You can't test 'find the actions' without re-running the summary. And every time you tweak one line, you re-roll all three jobs at once. One prompt is one thing you can't take apart.

Decompose it into a pipeline

So you split the one job into three, each a block you can read, test, and price on its own: compile the note into clean prose with a summary, pull the important points from that, then extract the actions. In Noukai this is a lane that runs left to right — tap any block below to see the exact prompt it runs, the input it receives, and what it returns.

Read left to right, that's the whole feature. 'Compile notes' does the unglamorous cleanup so every block after it reads tidy prose. 'Important points' turns that into the short list a busy person scans first. 'Extract actions' turns those points into things the app can offer to do. Three small, legible steps instead of one opaque one — and each is a block you can open, test against real notes, and fix in isolation.

The token trap — and the passthrough that springs it

There's a cost bug hiding in that lane. The summary has to reach the final result — it's half the feature. The obvious way is to feed it into every block so it falls out the far end. But 'Extract actions' doesn't read the summary; it reads the points. Pipe the summary through it anyway and you pay input tokens at that block for text it never looks at, then pay again to echo it back out.

That dashed block in the strip is the fix. A Passthrough carries a value straight to the result without ever sending it into a prompt. The summary rides around 'Extract actions' instead of through it — present at the end, never billed in the middle. On a four-line note it's pennies. Across every note every user ever summarizes, it's the difference between a feature you ship and one you have to throttle.

Wiring it to the app

On the app side there's almost nothing left to do. When a user taps Summarize, your server runs the published pipeline through the Noukai Python SDK — one call — and hands the result straight back to the UI, which already knows how to render it: summary at the top, points below, actions as tappable chips.

# pip install noukai-sdk
from noukai_sdk import Noukai

# Your server handles the Summarize tap. NOUKAI_API_KEY is read from the env,
# and org + project default the short slug to apple/notes/notes-to-actions.
with Noukai(org="apple", project="notes") as client:
    result = client.flow("notes-to-actions").execute(message=note_text)

data = result.result  # {"summary": ..., "points": [...], "actions": [...]}
# data["summary"] -> pinned to the top of the note
# data["points"]  -> the "worth noticing" list
# data["actions"] -> tappable chips: add to calendar, email, remind me

Notice what your handler doesn't contain: no prompts, no model names, no branching. Those live in the pipeline you published and tested. The SDK just calls the version you shipped — so tuning the summary prompt or swapping a model never touches this code. You edit the pipeline in Noukai and republish, and the same handler keeps working. (Prefer async? `AsyncNoukai` mirrors this call for an async server, and `execute_async` submits to the job queue for long notes.)

Where you'd take it next

Two doors open from here. The first is audio: the moment Notes lets you attach a voice memo, you add a transcription lane in front of this one and everything downstream is unchanged. The second is agency — letting the app not just suggest 'add to calendar' but do it on a tap, then on a sentence. That's the semi-agentic version, and it's the next build-along. If you want the discipline underneath all of this — versioning prompts, testing every change against real cases, shipping the exact pipeline you tested — that's what the AI engineering guide covers.