Documentation/How it works

How it works

Field Voice is a hands-free assistant for technicians. They speak, and it transcribes the audio, extracts a structured work order, answers questions from the equipment manuals, and keeps a live record for supervisors. Every part runs on free APIs, with secrets kept on the server and a fallback for every model so a demo never dead-ends.

Next.jsTypeScriptGroq Whispergpt-oss + LlamaGemini retrievalSupabaseWeb Speech
Architecture

The browser only records audio and renders results. All keys and model calls live in server-side API routes, so nothing sensitive ships to the client.

Data flow
Field worker
MediaRecorder, webm clip
Next.js API routes
process, ask, agent, knowledge
Groq, Gemini failover
Whisper, gpt-oss, Llama
Gemini embeddings
768-dim, cosine search
Supabase
work_orders, kb_chunks, Realtime
Supervisor dashboard
live work orders
Voice pipelines
1 · Log inspection
Speak
Transcribe
Whisper
Extract
Zod
Save
Supabase
  1. 01The recorder captures a clip with MediaRecorder and posts it to /api/process.
  2. 02Whisper transcribes the audio, primed by an HVAC domain prompt so codes like AHU-12 survive.
  3. 03The transcript goes to openai/gpt-oss-120b through generateObject, which fills the schema field by field.
  4. 04The validated order is inserted into work_orders and streamed to the dashboard. The reply is read aloud.
2 · Ask a question, retrieval
Ask
Transcribe
Retrieve
Gemini, cosine
Answer
grounded
  1. 01The question is transcribed, then embedded with gemini-embedding-001 into a 768-dimension vector.
  2. 02Cosine similarity ranks every chunk, seed and added, and keeps the top six.
  3. 03Those chunks become the only context for llama-3.1-8b-instant, which is told to answer from context alone.
  4. 04The answer, its sources, and the chunks it used return to the worker and are spoken back.
3 · Voice command, agent
Command
Transcribe
Plan
tool calling
Write
Supabase
  1. 01The spoken command is transcribed and handed to llama-3.3-70b-versatile with three tools bound.
  2. 02The agent picks one of createWorkOrder, updateWorkOrder, or closeWorkOrder.
  3. 03Update and close first locate the order by equipment or fault code with a case-insensitive match.
  4. 04After acting it returns one short confirmation, capped at five steps so it cannot loop.
Chunking and retrieval

Good answers depend on good chunks. A naive fixed-size split would cut a torque table or a procedure in half, so a single embedding would blur two unrelated ideas. Instead the knowledge base uses a heading-anchored, size-bounded, with sentence overlap strategy, so each vector represents one coherent passage. The same chunker runs at build time on the seed manuals and at runtime on anything you add.

Strategy
gemini-embedding-001 · 768 dimensions
Max chunk
900 chars
Overlap
150 chars
Min chunk
80 chars
Why this strategy
  1. 1Split on Markdown headings so each chunk covers one coherent topic. Retrieval then returns a whole idea instead of a sentence cut off mid-thought.
  2. 2Carry the document title into every chunk as context, so an isolated section still knows which manual or equipment it belongs to.
  3. 3Cap each chunk near 900 characters (about 220 tokens) to keep each passage focused, so a single embedding is not asked to summarise an over-long, multi-topic span.
  4. 4When a section is larger than the cap, pack it into windows along paragraph and sentence boundaries, and overlap about 150 characters between windows so an answer that straddles a boundary is never lost.
  5. 5Merge fragments below the minimum size into their neighbour, so we never store a near-empty embedding that adds noise to the search.
Embeddings

Each chunk is embedded with gemini-embedding-001 over plain HTTP on the free Gemini tier — no native runtime, so it works on any serverless host. Seed chunks ship invectors.json; anything you add is embedded the same way.

Cosine search

At query time only the question is embedded. Cosine similarity ranks every chunk and the top six form the context. The Knowledge page lets you watch this live, with the matching chunks and their scores.

Reliability and fallbacks

A live demo cannot depend on one provider staying up. Every model call has a backup, and the app stays useful even with no AI key at all.

  • Rate limit. If Groq returns 429 or a server error, the call retries on gemini-2.5-flash-lite automatically.

  • Degraded. Without a language model, extraction falls back to keyword rules and answers return the closest chunk. If embeddings are unavailable, a keyword search backs up retrieval so questions still return relevant passages.

  • Clear signal. /api/health reports which capabilities are live, and a banner appears when the app is running in a degraded mode.

Models
  • whisper-large-v3-turboSpeech to textGroq

    Transcribes the clip, biased toward HVAC equipment and fault codes via a domain prompt.

  • openai/gpt-oss-120bExtractionGroq

    Turns the transcript into a work order, validated field by field against a Zod schema.

  • llama-3.3-70b-versatileVoice agentGroq

    Tool calling to create, update, or close orders, capped at five reasoning steps.

  • llama-3.1-8b-instantGrounded answersGroq

    Fast model that answers only from the retrieved manual chunks, so replies stay short.

  • gemini-embedding-001EmbeddingsGoogle

    Free embeddings API over plain HTTP. Produces 768-dimension vectors with retrieval task types.

  • gemini-2.5-flash-liteFailoverGoogle

    Automatic backup for extraction, agent, and answers when Groq is rate-limited.

API routes
  • POST/api/processTranscribe, extract, and save one inspection.
    body: audio or textreturns: { transcript, workOrder, saved }
  • POST/api/askGrounded retrieval answer with the chunks it used.
    body: audio or textreturns: { answer, sources, matches, degraded }
  • POST/api/agentRun a spoken create, update, or close command.
    body: audio or textreturns: { command, reply }
  • POST/api/transcribeRaw audio to text, no extraction.
    body: audio (webm)returns: { text }
  • GET/api/knowledgeThe whole knowledge base with 2D coordinates for the map.
    body: nonereturns: { stats, chunking, points, sources }
  • POST/api/knowledgeChunk, embed, and store new knowledge.
    body: md file, text, or audioreturns: { source, count, added }
  • POST/api/knowledge/searchRanked chunks with scores, no model call.
    body: { query }returns: { results }
  • GET/api/work-ordersList the 50 most recent work orders.
    body: nonereturns: { orders }
  • GET/api/activityRecent voice transcripts for the dashboard.
    body: nonereturns: { activity }
  • GET/api/healthLiveness plus which capabilities are configured.
    body: nonereturns: { ok, llm, stt, supabase, kb }
Data model

One Zod schema is the single contract shared by AI extraction, the agent tools, and the Postgres columns, so a field name never drifts between layers.

Work order, extracted fields
  • equipment_codeAsset tag, for example AHU-12 or CH-02.
    string
  • inspection_resultWhat the technician observed.
    string
  • fault_codeFault reference, for example F-203.
    string
  • locationWhere the asset sits.
    string
  • severitylow, medium, high, or critical.
    enum
  • action_takenRemediation performed or recommended.
    string
  • parts_requiredItems of { name, part_number, quantity }.
    array
kb_chunks, the knowledge store
  • idStable chunk id, source plus index.
    text
  • sourceDocument the chunk came from.
    text
  • chunk_indexPosition of the chunk within its document.
    int
  • headingThe section heading the chunk sits under.
    text
  • textThe chunk content that gets embedded.
    text
  • embedding768 floats from Gemini, used for cosine search.
    jsonb
Offline and realtime
Offline

When the signal drops, the recording is stored in the browser through Dexie (IndexedDB). The moment the connection returns it re-uploads to/api/process automatically, transcribes server-side, and marks itself synced.

Realtime

When a work order changes, the server pushes a broadcast event on the work-orders channel and the dashboard refetches at once. The database stays locked behind row-level security — the browser never reads it directly — and a slow poll backs the channel up, shown by the live or polling pill.

Security and cost
Secrets

The GROQ_API_KEY and the Supabase secret key are read only in API routes. The browser sees just the public publishable key, prefixedNEXT_PUBLIC_, which is safe to ship.

Free tier

Groq, Supabase, and Vercel all run on free tiers, and embeddings run locally, so there is no paid NLP API anywhere in the stack. The whole app deploys to one public URL.

Project map
  • lib/stt.tsGroq Whisper transcription with the HVAC domain prompt.
  • lib/extract.tsSchema-validated extraction with a keyword fallback.
  • lib/agent.tsTool-calling voice agent (create / update / close).
  • lib/rag.tsEmbed, cosine search, and grounded answer.
  • lib/kb/chunk.tsThe chunking strategy and its parameters.
  • lib/kb/store.tsSeed plus kb_chunks store, search, add, and remove.
  • lib/llm.tsGroq to Gemini failover on rate limits.
  • lib/schema.tsThe shared contract for AI, agent, and database.
Tech stack
Frontend
  • Next.js 16 (App Router)
  • React 19 + TypeScript
  • Tailwind v4
  • Dexie (IndexedDB)
AI
  • Groq Whisper, speech to text
  • gpt-oss + Llama, extract and agent
  • Gemini embeddings, free tier
  • Web Speech API, spoken replies
Data
  • Supabase Postgres
  • Supabase Realtime
  • Knowledge store: seed + kb_chunks
  • In-memory cosine retrieval
Hosting
  • Vercel, single deploy
  • 100% free tiers
  • No paid NLP APIs
  • Public URL + GitHub