Streaming LLM responses over SSE in Node.js

How server-sent events carry a model's tokens to the browser, with a small Node server and client that work with any streaming chat completions API.

A language model produces its answer one token at a time, and a good interface shows each token as it arrives instead of waiting for the whole reply. Server-sent events (SSE) are the simplest way to do that over HTTP: the server keeps one response open and writes small text events into it, and the browser's built-in EventSource reads them.

This version replaces the original post, which called an API endpoint that no longer exists. It uses the chat completions streaming format, which is also what most compatible providers speak.

The server

Node 20 or newer needs no dependencies: the built-in http module serves the page and the stream, and the built-in fetch calls the model.

// server.mjs
import { createServer } from 'node:http'
 
const BASE_URL = process.env.OPENAI_BASE_URL ?? 'https://api.openai.com/v1'
const MODEL = process.env.OPENAI_MODEL // pick a current chat model from your provider
 
const PAGE = `<pre id="out"></pre>
<script>
	const out = document.getElementById('out')
	const source = new EventSource('/stream?q=' + encodeURIComponent('Say hello in three languages'))
	source.onmessage = (event) => { out.textContent += JSON.parse(event.data) }
	source.addEventListener('done', () => source.close())
	source.onerror = () => source.close()
</script>`
 
createServer(async (req, res) => {
	const url = new URL(req.url, 'http://localhost')
 
	if (url.pathname === '/') {
		res.writeHead(200, { 'Content-Type': 'text/html' }).end(PAGE)
		return
	}
	if (url.pathname !== '/stream') {
		res.writeHead(404).end()
		return
	}
 
	res.writeHead(200, {
		'Content-Type': 'text/event-stream',
		'Cache-Control': 'no-cache',
		Connection: 'keep-alive',
	})
 
	// Stop paying for tokens nobody will read when the browser goes away.
	const controller = new AbortController()
	res.on('close', () => controller.abort())
 
	try {
		const upstream = await fetch(`${BASE_URL}/chat/completions`, {
			method: 'POST',
			signal: controller.signal,
			headers: {
				'Content-Type': 'application/json',
				Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
			},
			body: JSON.stringify({
				model: MODEL,
				stream: true,
				messages: [{ role: 'user', content: url.searchParams.get('q') ?? 'Hello' }],
			}),
		})
 
		if (!upstream.ok) {
			res.write(`event: error\ndata: ${JSON.stringify(upstream.status)}\n\n`)
			return
		}
 
		const decoder = new TextDecoder()
		let buffer = ''
		for await (const chunk of upstream.body) {
			buffer += decoder.decode(chunk, { stream: true })
			const lines = buffer.split('\n')
			buffer = lines.pop() // the last line may be unfinished: keep it for the next chunk
 
			for (const line of lines) {
				if (!line.startsWith('data: ')) continue
				const data = line.slice(6).trim()
				if (data === '[DONE]') continue
				const text = JSON.parse(data).choices?.[0]?.delta?.content
				if (text) res.write(`data: ${JSON.stringify(text)}\n\n`)
			}
		}
 
		res.write('event: done\ndata: end\n\n')
	} catch (error) {
		// An abort is the browser leaving; anything else is worth telling it about.
		if (!controller.signal.aborted) {
			res.write(`event: error\ndata: ${JSON.stringify(String(error))}\n\n`)
		}
	} finally {
		res.end()
	}
}).listen(3000, () => console.log('http://localhost:3000'))

Run it with your key and a model name:

OPENAI_API_KEY=sk-... OPENAI_MODEL=your-model node server.mjs

How it works

The model's API streams its own server-sent events. Each line that starts with data: holds a JSON chunk, and the new text is at choices[0].delta.content. The stream ends with data: [DONE].

A network read does not line up with those lines. A chunk can end halfway through one, which is why the server keeps the unfinished tail in a buffer and prepends it to the next chunk. Skipping that step works in a quick test and breaks under real traffic.

The server forwards only the text, one event per piece, with the text JSON-encoded so a newline inside a token cannot break the event format. The browser appends each piece to the page. When the model finishes, the server sends a done event so the client can close the connection.

Why the client closes the connection

EventSource reconnects automatically when a connection drops, which is what you want for a live feed and not what you want here: a finished stream would immediately start a new, billable request. Closing on done and on error prevents the loop.

When to reach for something else

SSE is one-way, from server to browser. For a chat where the user can also interrupt, or for audio, you want a bidirectional channel such as a WebSocket. For plain token streaming, SSE is less code, passes through proxies cleanly and reconnects on its own.