How it works
Field Voice is a hands-free assistant for technicians. They speak, and it transcribes the audio, extracts a structured work order, answers questions from the equipment manuals, and keeps a live record for supervisors. Every part runs on free APIs, with secrets kept on the server and a fallback for every model so a demo never dead-ends.
The browser only records audio and renders results. All keys and model calls live in server-side API routes, so nothing sensitive ships to the client.
- 01The recorder captures a clip with
MediaRecorderand posts it to/api/process. - 02Whisper transcribes the audio, primed by an HVAC domain prompt so codes like
AHU-12survive. - 03The transcript goes to
openai/gpt-oss-120bthroughgenerateObject, which fills the schema field by field. - 04The validated order is inserted into
work_ordersand streamed to the dashboard. The reply is read aloud.
- 01The question is transcribed, then embedded with
gemini-embedding-001into a 768-dimension vector. - 02Cosine similarity ranks every chunk, seed and added, and keeps the top six.
- 03Those chunks become the only context for
llama-3.1-8b-instant, which is told to answer from context alone. - 04The answer, its sources, and the chunks it used return to the worker and are spoken back.
- 01The spoken command is transcribed and handed to
llama-3.3-70b-versatilewith three tools bound. - 02The agent picks one of
createWorkOrder,updateWorkOrder, orcloseWorkOrder. - 03Update and close first locate the order by equipment or fault code with a case-insensitive match.
- 04After acting it returns one short confirmation, capped at five steps so it cannot loop.
Good answers depend on good chunks. A naive fixed-size split would cut a torque table or a procedure in half, so a single embedding would blur two unrelated ideas. Instead the knowledge base uses a heading-anchored, size-bounded, with sentence overlap strategy, so each vector represents one coherent passage. The same chunker runs at build time on the seed manuals and at runtime on anything you add.
- 1Split on Markdown headings so each chunk covers one coherent topic. Retrieval then returns a whole idea instead of a sentence cut off mid-thought.
- 2Carry the document title into every chunk as context, so an isolated section still knows which manual or equipment it belongs to.
- 3Cap each chunk near 900 characters (about 220 tokens) to keep each passage focused, so a single embedding is not asked to summarise an over-long, multi-topic span.
- 4When a section is larger than the cap, pack it into windows along paragraph and sentence boundaries, and overlap about 150 characters between windows so an answer that straddles a boundary is never lost.
- 5Merge fragments below the minimum size into their neighbour, so we never store a near-empty embedding that adds noise to the search.
Each chunk is embedded with gemini-embedding-001 over plain HTTP on the free Gemini tier — no native runtime, so it works on any serverless host. Seed chunks ship invectors.json; anything you add is embedded the same way.
At query time only the question is embedded. Cosine similarity ranks every chunk and the top six form the context. The Knowledge page lets you watch this live, with the matching chunks and their scores.
A live demo cannot depend on one provider staying up. Every model call has a backup, and the app stays useful even with no AI key at all.
Rate limit. If Groq returns 429 or a server error, the call retries on
gemini-2.5-flash-liteautomatically.Degraded. Without a language model, extraction falls back to keyword rules and answers return the closest chunk. If embeddings are unavailable, a keyword search backs up retrieval so questions still return relevant passages.
Clear signal.
/api/healthreports which capabilities are live, and a banner appears when the app is running in a degraded mode.
- whisper-large-v3-turboSpeech to textGroq
Transcribes the clip, biased toward HVAC equipment and fault codes via a domain prompt.
- openai/gpt-oss-120bExtractionGroq
Turns the transcript into a work order, validated field by field against a Zod schema.
- llama-3.3-70b-versatileVoice agentGroq
Tool calling to create, update, or close orders, capped at five reasoning steps.
- llama-3.1-8b-instantGrounded answersGroq
Fast model that answers only from the retrieved manual chunks, so replies stay short.
- gemini-embedding-001EmbeddingsGoogle
Free embeddings API over plain HTTP. Produces 768-dimension vectors with retrieval task types.
- gemini-2.5-flash-liteFailoverGoogle
Automatic backup for extraction, agent, and answers when Groq is rate-limited.
- POST/api/processTranscribe, extract, and save one inspection.body: audio or textreturns: { transcript, workOrder, saved }
- POST/api/askGrounded retrieval answer with the chunks it used.body: audio or textreturns: { answer, sources, matches, degraded }
- POST/api/agentRun a spoken create, update, or close command.body: audio or textreturns: { command, reply }
- POST/api/transcribeRaw audio to text, no extraction.body: audio (webm)returns: { text }
- GET/api/knowledgeThe whole knowledge base with 2D coordinates for the map.body: nonereturns: { stats, chunking, points, sources }
- POST/api/knowledgeChunk, embed, and store new knowledge.body: md file, text, or audioreturns: { source, count, added }
- POST/api/knowledge/searchRanked chunks with scores, no model call.body: { query }returns: { results }
- GET/api/work-ordersList the 50 most recent work orders.body: nonereturns: { orders }
- GET/api/activityRecent voice transcripts for the dashboard.body: nonereturns: { activity }
- GET/api/healthLiveness plus which capabilities are configured.body: nonereturns: { ok, llm, stt, supabase, kb }
One Zod schema is the single contract shared by AI extraction, the agent tools, and the Postgres columns, so a field name never drifts between layers.
- equipment_codeAsset tag, for example AHU-12 or CH-02.string
- inspection_resultWhat the technician observed.string
- fault_codeFault reference, for example F-203.string
- locationWhere the asset sits.string
- severitylow, medium, high, or critical.enum
- action_takenRemediation performed or recommended.string
- parts_requiredItems of { name, part_number, quantity }.array
- idStable chunk id, source plus index.text
- sourceDocument the chunk came from.text
- chunk_indexPosition of the chunk within its document.int
- headingThe section heading the chunk sits under.text
- textThe chunk content that gets embedded.text
- embedding768 floats from Gemini, used for cosine search.jsonb
When the signal drops, the recording is stored in the browser through Dexie (IndexedDB). The moment the connection returns it re-uploads to/api/process automatically, transcribes server-side, and marks itself synced.
When a work order changes, the server pushes a broadcast event on the work-orders channel and the dashboard refetches at once. The database stays locked behind row-level security — the browser never reads it directly — and a slow poll backs the channel up, shown by the live or polling pill.
The GROQ_API_KEY and the Supabase secret key are read only in API routes. The browser sees just the public publishable key, prefixedNEXT_PUBLIC_, which is safe to ship.
Groq, Supabase, and Vercel all run on free tiers, and embeddings run locally, so there is no paid NLP API anywhere in the stack. The whole app deploys to one public URL.
- lib/stt.tsGroq Whisper transcription with the HVAC domain prompt.
- lib/extract.tsSchema-validated extraction with a keyword fallback.
- lib/agent.tsTool-calling voice agent (create / update / close).
- lib/rag.tsEmbed, cosine search, and grounded answer.
- lib/kb/chunk.tsThe chunking strategy and its parameters.
- lib/kb/store.tsSeed plus kb_chunks store, search, add, and remove.
- lib/llm.tsGroq to Gemini failover on rate limits.
- lib/schema.tsThe shared contract for AI, agent, and database.
- Next.js 16 (App Router)
- React 19 + TypeScript
- Tailwind v4
- Dexie (IndexedDB)
- Groq Whisper, speech to text
- gpt-oss + Llama, extract and agent
- Gemini embeddings, free tier
- Web Speech API, spoken replies
- Supabase Postgres
- Supabase Realtime
- Knowledge store: seed + kb_chunks
- In-memory cosine retrieval
- Vercel, single deploy
- 100% free tiers
- No paid NLP APIs
- Public URL + GitHub