Building a Video Intelligence Platform: VLM Analysis, Real-Time Alerts, and Natural Language Search

Architecture notes from Predact, an in-progress build. Go, RabbitMQ, pgvector, FFmpeg, and a vision-language model doing the seeing.

After almost every incident, the footage turns up. It's on tape. The camera was pointed the right way, the disk had space, the retention policy held. Someone pulls the clip and there it is: the bag left by the bench, the weapon under the jacket, the person going over the fence.

That's the part that should bother you. Nothing failed. The camera worked, the recorder worked, and the system still got the only thing that mattered wrong — it didn't say anything at the time.

A control room has two hundred cameras and one operator on shift. There is no version of that person who is watching camera 47 at 3am. The cameras aren't a monitoring system; they're an archive we've agreed to call monitoring.

So the thing I'm building isn't a better search over recordings. It's a system that can say something in the next few seconds, to someone who can still act on it — and then, separately, let that person go back and ask questions in plain language about what it saw.

This is live work. Below is where the design stands right now, what's running, what isn't, and the parts I expect to change.

---

What it is

Five services, orchestrated with Docker Compose:

service	stack	responsibility
frontend	Next.js 16, React 19, shadcn/ui, hls.js	operations UI — monitoring, playback, analysis, chat
user-service	Hono, better-auth, PostgreSQL	identity and organizations
core	Go (stdlib http, pgx), PostgreSQL + pgvector	product API, router → handler → service → repo
ingest	Go, FFmpeg	recording, retention, uploads, job production
vision-worker	Go, FFmpeg, VLM via OpenRouter	frame extraction, structured analysis, embeddings

Plus RabbitMQ for the analysis queue, MediaMTX for RTSP→HLS restreaming, and two isolated Postgres instances — identity and product data never share a database.

Figure 1. The line that matters: recording writes to disk without waiting for anything downstream. No model sits in that path.

Monitoring. Nine public border-crossing cameras restreamed through MediaMTX and played as HLS, so a browser tab holds the whole wall without a plugin.

Why a VLM and not a detector

The obvious objection is that weapon detection is a solved problem. You can train a detector for it — I have, and wrote up what that actually involves. But look at what actually needs to raise an alarm:

• a backpack sitting alone next to a bench for ten minutes
• someone handing a bag to another person and walking off in the opposite direction
• a person going over a wall instead of through the gate
• a hi-vis vest on a site where no work is scheduled today

Each of those is either a new class, a new labelled dataset and a new training run, or a hand-written rule stacked on top of detector output — and rules on top of bounding boxes get brittle the moment reality varies. Meanwhile the actual list of things worth an alarm is written by whoever runs the site this month, and it changes.

A vision-language model changes the unit of work. Instead of car, 0.94, the frame comes back described — and the description is stored as structured fields, not prose:

Analysis, one camera selected. Each sampled frame carries a scene description and counts — ten objects and two pieces of text in the frame on the left.

Opening that frame. Ten objects, each with its own description: "silver BMW X3 SUV — silver metallic, headlights on, front grille visible, no visible damage · center-left lane, middle ground". Make, model, colour, state, and where in the frame — in one string.

Three things there a detector can't give me. Attributes nobody trained for — make, model, colour, lane position — which is what a real question is made of. Absence: no visible damage is a claim about what isn't there, and no class label can express it. And when the model is unsure it says so in the same string, which is how I know how far to trust the row.

What I'm buying is an open vocabulary. What I'm paying is latency and per-frame cost, and most of the design below is a consequence of that bill.

Turning a sentence into a watcher

The part I'm building now is the one I care about most: an operator writes a scenario in plain language, and the system watches for it.

Notify me if a black Ford Kuga enters the north gate.

Notify me if someone puts a bag down and walks away from it.

The first is nearly free — it's a description matched against descriptions the pipeline already produces. The second is the hard one, and it's where a per-frame model stops being enough. "Put a bag down and walked away" doesn't exist in any single frame. It's a state change across frames: an object appears associated with a person, then the person's track leaves and the object doesn't. Detecting it means the agent needs a short memory of the scene, not just the current image.

That's the open engineering problem in this project right now, and it's why the frame analysis output is structured rather than free text — people, objects, text and signals as separate fields, so something downstream can reason over a sequence of them instead of re-reading prose.

The alert here isn't a class label. "Person lying motionless on hallway floor, potentially indicating medical emergency or fall" is a state, with a severity, attached to the frame that produced it.

The deadline is doing most of the design work

Forensic search has no deadline. A query that takes eight seconds is fine. Alerting has a hard one: an alert that arrives after the person has left the frame isn't degraded, it's zero.

So the services split along the deadline rather than along features:

• recording never blocks and is never allowed to fail
• live analysis is deadline-bound — late output is worthless output
• indexing and backfill is throughput-bound — late is fine as long as it lands

The queue between recording and analysis exists because the model provider will rate-limit me, time out, and get slow at the worst possible moment. If analysis sits inside the recording path, each of those becomes lost footage — and lost footage is the one failure nobody forgives. With a queue in between, an outage means backlog, and recording never notices.

The split, made concrete: recording is a per-source switch that has nothing to do with whether analysis is running. A source can be online and unrecorded, recorded and unanalysed, or both.

Why the queue stays even as I chase real-time

My first instinct about making this real-time was to remove the queue. That instinct was wrong, and the arithmetic is what changed my mind.

The end-to-end latency is roughly:

sampling interval + decode + model inference (network + forward pass) + delivery

A RabbitMQ publish and consume is on the order of a millisecond. It isn't the bottleneck; it's a rounding error inside a budget dominated by the sampling interval and the model call. Removing it wouldn't buy real-time — it would only remove backpressure. When input rate exceeds processing rate and there's no queue, you get two options: block (which puts the recording path at risk, the exact thing this architecture exists to prevent) or silently drop frames (a missed alert with no record that it was missed). A queue makes overflow visible and bounded: depth is a metric, TTL is a policy.

There's a scaling argument too. A bigger GPU shortens a single inference and helps latency, but nine cameras become ninety and vertical scaling hits a wall. Past that you add workers — and adding workers is exactly what the queue is for. Removing it doesn't just fail to buy real-time, it closes the road to scale.

So the real-time work is going somewhere else: running the model locally to kill network round-trips and rate limits, keeping a decode loop open per camera instead of reopening streams, making the cheap first stage cheap enough to run often, and putting a TTL on the live lane — a frame older than the alert deadline is worthless for alerting, so it drops out of the live lane and goes to the index lane instead.

Real-time doesn't mean there's no queue. It means end-to-end latency is bounded, and you know what happens when it can't be.

The model is a verifier, never a tripwire

Both halves of the product converge on the same shape: a cheap, wide, recall-oriented first stage, then the model as the second.

Search runs it like this. Every analysed frame carries a structured description and an embedding. A query pulls a deliberately wide candidate set from pgvector, tuned for recall, because embeddings happily place "black backpack" next to "dark duffel bag". Then the model looks at those candidate frames and answers the question that was actually asked.

Eight candidates from vector search, eight checked by the model, two verified — with timestamps and playable footage. The count is deliberately visible in the UI.

Alerting runs the same shape with a different cheap stage: motion and scene change over-trigger on purpose, and the model decides whether this is worth waking someone for.

The reason to build it this way isn't only cost. Recall is the metric everyone optimises and the wrong one to build this product around. An alerting system that fires forty times a shift gets muted by the end of the week, and a muted system has a recall of zero regardless of what the benchmark said. The verification stage is what keeps the thing from being switched off.

You can see the same principle in what gets stored versus what gets shown. The extraction pass is exhaustive — it will describe the trees, the road markings and the cornfield, because a describe-everything prompt describes everything. That exhaustive pass is what makes the index useful later. The filtered view is what an operator should ever see.

Getting evidence back out

Search that ends at "here's a frame" isn't much use to anyone who has to hand the result to someone else. Playback is archive-aware — it knows where the gaps are rather than pretending the timeline is continuous — and every result links to its own moment in it.

Playback. The bar underneath is the archive itself — recorded segments as blocks, uncovered time as empty space. Nothing is stretched to make the day look continuous.

Clips are produced from recordings rather than cut on demand, so the moment is already sitting there when someone goes looking — sixty seconds, titled from the analysis, ready to open.

Export runs through FFmpeg: clips concatenated without re-encoding where possible, boxes and a legend burned into the frame, and a SHA-256 checksum written alongside the file so the artifact can be shown to be unmodified.

An export, not a screen: the analysis is burned into the image file itself. Eleven objects here, including the ones nobody asked about — a cemetery, a cornfield, the road markings.

Another export, this one produced against a specific question. Same pipeline, filtered to what was asked.

What runs today, and what doesn't

Working end to end: RTSP/RTMP/HLS ingest and restreaming, continuous recording with retention, uploads with resumable chunking, frame extraction and structured VLM analysis, embeddings in pgvector, two-stage natural language search with verified evidence cards, gap-aware playback, automatic clips, and FFmpeg export with burned-in boxes and a SHA-256 checksum for the audit trail.

Not working yet, or not honestly claimable:

• Alerts surface in the app, not off it. The severity, the reasoning and the frame are all there; delivery to a phone or a webhook isn't wired.
• Cross-camera identity. One person walking through four cameras is currently four unrelated observations.
• No real noise numbers. The metric I want on a dashboard is alerts per operator per shift and the share of them a human agreed with. That only means something after a long run against live cameras, not test clips.

The limit I keep running into

At close range in decent light, dangerous-object detection holds up. At forty metres in low light it doesn't, and no amount of prompt work moves it. When the object occupies a dozen pixels, there's nothing in the image to read. The model isn't being stubborn; the information was never captured.

The reflex with LLM-shaped products is to treat every failure as a prompting failure. Some of them are optics, placement and lighting — hardware problems wearing a software costume. I don't have a software answer for this one and I don't expect to find one, so it's becoming a coverage problem instead: which cameras are trusted for which kinds of alert, and saying so in the interface. "No alerts" doesn't mean "nothing happened", the same way an empty search result doesn't, and someone will read the silence as an all-clear unless the UI is explicit about where the system can't see.

One bug worth writing down

Recording started with a rolling segment pattern like seg_%06d.ts. The counter is per-process, so a restart went back to seg_000000 and began overwriting yesterday. Not a crash — silent data loss that surfaces only when someone asks for footage that no longer exists. The fix was to stop using a counter as a name:

ffmpeg -i "$SRC" -c copy -f segment -strftime 1 \
  -segment_time 60 -segment_format mp4 \
  "$ARCHIVE/%Y/%m/%d/%H-%M-%S.mp4"

Timestamps are unique, sort correctly, and turn retention into a path comparison. If a filename is going to act as a primary key, treat it like one.

Next

• A local model as the cheap first stage. Motion is a blunt trigger; a small detector is a better one, and it removes a network round-trip from the critical path. I fine-tuned a YOLO model on a weapon dataset recently and measured the part that matters here: about 4ms per image at 640px, 15ms at 1280. On a 30fps stream that's the whole budget — the writeup is here, including why the dataset was 45% broken and what a model trained on bad labels looks like from the inside.
• Scene memory for the agent, so "put a bag down and walked away" is expressible at all.
• Cost per camera-hour at a sampling rate that's actually useful — the number that decides how much of the pipeline can stay on a hosted model.
• Compose now, Kubernetes later. Services are stateless and env-configured so the move should be a packaging job rather than a rewrite. Untested claim until I make it.

I'll write these up as they land.