Run Nate on Your Machine

Nate is built as a desktop-native voice assistant. Deep-learning speech models (Faster-Whisper STT, Piper TTS ONNX, and Silero VAD) are compute-intensive and require large RAM footprints that exceed standard cloud hosting limits (such as standard 512MB free-tier constraints). Additionally, a voice-to-voice workflow requires direct, low-latency access to your local microphone and speaker hardware. Therefore, the pipeline must run locally.

1. System Prerequisites

Python 3.10+: Core programming language.
Node.js 18+ & npm: Runtime to compile and serve the Next.js chat interface.
Git: To clone the code repositories.
Nvidia CUDA (Optional): If you have an RTX GPU, ensure CUDA Toolkit 11.8+ is installed to accelerate speech recognition.

2. Installation Guide

Step A: Clone the Repository

git clone https://github.com/NoelNinanSheri1307/Nate.git
cd Nate

Step B: Initialize Python Backend

Create a virtual environment and install backend libraries (VAD, speech processing, and WebSocket layers):

python -m venv venv
# Activate the environment:
# Windows (PowerShell):
.\venv\Scripts\Activate.ps1
# macOS/Linux:
source venv/bin/activate

# Install dependencies:
pip install -r requirements.txt
pip install openwakeword

Step C: Setup Environment Configuration

Create a .env file in the root directory to store your Gemini API Key:

GEMINI_API_KEY=your_gemini_api_key_here
GEMINI_MODEL=gemini-3.1-flash-lite
WHISPER_MODEL=small

Step D: Download local Piper TTS files

Download the local voice synthesis binaries and place them under the models/piper/ directory. Your file hierarchy must look like:

models/piper/
├── piper.exe                  # Local C++ Piper runtime
├── en_US-joe-medium.onnx      # Vocal character model
├── en_US-joe-medium.onnx.json # Vocal phoneme configurations
└── espeak-ng-data/            # Phoneme lookup database

Step E: Initialize Frontend Node Packages

cd frontend
npm install

3. Running the Experience

Start both endpoints in separate terminals from the project root directory:

Terminal 1: Python API Backend

# Activate venv first
.\venv\Scripts\Activate.ps1
uvicorn server:app --reload

Terminal 2: React UI Client

cd frontend
npm run dev

Once running, open http://localhost:3000 in your browser. Turn on the "Wake Word" header switch, say "Hey Jarvis" or "Hey Mycroft", and speak.

4. Pipeline Architecture

Nate uses a fully pipelined, asynchronous architecture to overlap networking latency with deep learning synthesis.

Microphone → Silero VAD → Faster-Whisper (STT) → Memory State → Gemini 2.5 Flash → Sentence Splitting → Piper TTS → Speaker

Zero-Gap VAD: Keeps the device stream active. VAD checks speech probabilities in parallel.
Model Warm-Up: ONNX engines are loaded into RAM once during startup, ensuring subsequent speech synthesis completes in under 200ms.
Sentence Splitting: Text chunks emitted by the Gemini Stream are split into sentences on punctuation marks, allowing synthesis to play speech while the rest of the text is still generating.

5. Troubleshooting & FAQ

Q: I get "Port already in use" errors during backend startup?

A: Another uvicorn or local process is bound to port 8000. Run netstat -ano | findstr 8000 to find the PID, and kill the process.

Q: Can I run speech synthesis on AMD / CPU?

A: Yes. Piper TTS is optimized to execute on CPU. Faster-Whisper will automatically select CPU mode (`int8` execution) if CUDA-compatible Nvidia drivers are not detected.

Q: The wake word is not responding?

A: Make sure the package openwakeword is installed inside your active Python environment. Ensure your default input microphone is connected and unmuted.

GitHub Code

Repository & Issues

Docs (README)

Architecture & Details

Latest Release

Binaries & Downloads

Back to App

Showcase Interface