01 / Voice
Local voice pipeline.
Built a file-based Whisper → Gemma 4B → Piper pipeline. Streaming and sentence-chunked TTS reduced time-to-first-audio from 11.4 seconds to 3.45 seconds in testing.
A small local AI lab: speech-to-text, local LLM responses, text-to-speech, a physical status display, and a growing vision layer with object recognition, pose estimation, and depth-aware perception.
This project is my hands-on way to learn edge AI without hiding the rough edges. The constraint is useful: an 8 GB Jetson Orin Nano Super can run real local AI pieces, but only if the stack is small, measurable, and honest about memory, latency, thermals, and device I/O.
The goal is not a polished consumer assistant. It is a working bench where voice, vision, display, and automation can be tested one layer at a time.
Built a file-based Whisper → Gemma 4B → Piper pipeline. Streaming and sentence-chunked TTS reduced time-to-first-audio from 11.4 seconds to 3.45 seconds in testing.
Packaged a Jetson TFT monitor around an ST7735 screen with service controls, system readouts, and helper tooling so the board can show its state without needing a full desktop.
Validated a Sunplus USB stereo camera: one UVC stream on /dev/video0, side-by-side dual 1280×720 eyes, MJPG up to 2560×720@30, parallax confirmed, and a small OpenCV stereo-test helper for preview/capture.
The vision layer now has an object-recognition path for turning camera frames into practical scene labels: person, pet, vehicle, tool, cup, or whatever class matters for the next demo.
Pose estimation gives the project a second kind of signal: not just what is in the frame, but how a person is positioned. That matters for safety, workshop, exercise, and robotics-style interactions.
Depth estimation moves the camera from image capture toward spatial awareness. The useful target is simple: combine what the object is, where the person is, and roughly how far away things are.
The project is moving from "camera works" to "the machine can describe useful parts of the scene." That is the base for a local safety assistant, workshop monitor, pet/wildlife camera, or small robotics brain.
Object labels, pose, and depth can feed voice responses, the TFT status screen, Slack/Hermes notifications, or a local dashboard. The same pipeline can say what changed instead of just recording video.
Recognition, pose, and depth are separate enough to benchmark on their own, then combine. That keeps the project honest on FPS, memory, heat, and reliability instead of hiding everything inside one fragile demo.
Whisper.cpp with CUDA, Piper TTS, and Ollama/Gemma were wired into a local speech loop. The key lesson: the board can do it, but browser/dev-tool memory pressure and keep-alive services matter on 8 GB.
The ST7735 display path became a service-style Jetson monitor with helper commands and a dashboard row, after debugging GPIO mode and optional spidev dependencies.
The USB stereo camera was verified with OpenCV and VNC preview. The next quality step was locking camera settings before depth estimation and model experiments.
The project now has the core perception functions needed for a useful local vision bench: object recognition for scene labels, pose estimation for body position, and depth estimation for approximate distance. The next work is not proving the ideas exist; it is making them stable together under the Jetson's memory, camera, and FPS constraints.
Most useful projects are not a single finished demo; they are a chain of small verified steps. This page starts the project blog so future entries can break out voice, display, camera, model benchmarks, and final applications into their own navigable notes.
Back to projects