How to Build a Python Voice Assistant with Google GenAI

Building Your Own Voice-Controlled AI Assistant: A Complete Guide to google-genai, SpeechRecognition, pyttsx3, and PyAudio

Have you ever wanted to build your own personal voice assistant—something like a custom, local version of Jarvis or Alexa—powered by cutting-edge generative AI?

By combining modern cloud intelligence with robust audio libraries in Python, you can bridge the gap between human speech and advanced machine reasoning in just a few lines of code. The secret recipe relies on a powerful quartet of Python packages: google-genai, SpeechRecognition, pyttsx3, and pyaudio.

The Core Quartet: What Do They Do?

Before jumping into the installation and code, let’s break down what each package brings to the table:

• google-genai: Google’s official, modern Python SDK for interacting with Gemini models. It handles the “brain” of your application, generating intelligent, context-aware responses to whatever you ask it.
• SpeechRecognition: A versatile library that acts as your assistant’s ears. It easily integrates with various engines and APIs (including Google’s web speech API) to translate spoken audio bytes into clean text strings.
• PyAudio: The underlying engine wrapper that binds Python to your operating system’s audio drivers. Without it, SpeechRecognition wouldn’t be able to hook into your computer’s microphone to capture live sound.
• pyttsx3: Your assistant’s voice box. Unlike cloud-based text-to-speech tools, pyttsx3 is a completely offline text-to-speech conversion library that utilizes your operating system’s native native speech synthesis engine (sapi5 on Windows, nsss on macOS, and espeak on Linux).

Step 1: Installation

Open your terminal or command prompt and run the following pip command to install all four dependencies:

pip install google-genai SpeechRecognition pyttsx3 pyaudio

Troubleshooting Audio Installation:

• Windows: If pyaudio throws an error during installation, you may need to install pre-compiled wheel binaries matching your Python version via pip install pipwin && pipwin install pyaudio.
• Linux (Debian/Ubuntu): You might need to install PortAudio system dependencies first:

  sudo apt-get install portaudio19-dev python3-pyaudio
  sudo apt install espeak-ng libespeak1  # Needed for pyttsx3 on Linux
  

• macOS: You may need to run brew install portaudio via Homebrew before installing pyaudio.

Step 2: Setting Up Your API Key

To use the Google GenAI SDK, you need an API key from Google AI Studio. Set it as an environment variable so your Python script can securely pick it up:

• Mac/Linux: export GEMINI_API_KEY="your_api_key_here"
• Windows (Command Prompt): set GEMINI_API_KEY="your_api_key_here"
• Windows (PowerShell): $env:GEMINI_API_KEY="your_api_key_here"

Step 3: Putting It All Together (The Code)

Here is a complete, self-contained Python script that listens to your voice via microphone, transcribes it, sends it to the Gemini model for processing, and reads the response back to you out loud.

import os
import speech_recognition as sr
import pyttsx3
from google import genai

def initialize_assistant():
    # Initialize Text-to-Speech Engine
    tts_engine = pyttsx3.init()
    tts_engine.setProperty('rate', 160)  # Speed of speech
    
    # Initialize Google GenAI Client
    # It automatically looks for the GEMINI_API_KEY environment variable
    client = genai.Client()
    
    return tts_engine, client

def speak(engine, text):
    """Converts text string into spoken audio response."""
    print(f"n🤖 Gemini: {text}")
    engine.say(text)
    engine.runAndWait()

def listen_to_user(recognizer, microphone):
    """Captures audio from the microphone and converts it to text."""
    with microphone as source:
        print("n🎤 Adjusting for ambient noise... Please wait.")
        recognizer.adjust_for_ambient_noise(source, duration=1)
        print("🎙️ Listening... Speak now!")
        
        try:
            # Listen for the user's input with a timeout limit
            audio = recognizer.listen(source, timeout=5, phrase_time_limit=10)
            print("🔄 Processing speech...")
            
            # Recognize speech using Google's free tier web speech API
            text = recognizer.recognize_google(audio)
            print(f"👤 You said: {text}")
            return text
        except sr.WaitTimeoutError:
            print("⌛ Listening timed out. No speech detected.")
            return None
        except sr.UnknownValueError:
            print("❓ Sorry, I could not understand what you said.")
            return None
        except sr.RequestError as e:
            print(f"⚠️ Could not request results from Google Speech Recognition service; {e}")
            return None

def main():
    tts_engine, client = initialize_assistant()
    recognizer = sr.Recognizer()
    microphone = sr.Microphone()
    
    speak(tts_engine, "Hello! I am your Gemini voice assistant. Ask me anything, or say exit to quit.")
    
    while True:
        user_input = listen_to_user(recognizer, microphone)
        
        if not user_input:
            continue
            
        # Check for exit command
        if user_input.lower() in ["exit", "quit", "goodbye"]:
            speak(tts_engine, "Goodbye! Have a wonderful day.")
            break
            
        try:
            # Query Gemini using the standard flash model for quick responses
            response = client.models.generate_content(
                model='gemini-2.5-flash',
                contents=user_input,
            )
            
            ai_reply = response.text
            speak(tts_engine, ai_reply)
            
        except Exception as e:
            error_msg = "I'm sorry, I encountered an error connecting to the Gemini brain."
            print(f"Error: {e}")
            speak(tts_engine, error_msg)

if __name__ == "__main__":
    main()

What’s Next?

By executing pip install google-genai speechrecognition pyttsx3 pyaudio, you have unlocked an entire ecosystem of local hardware interfacing and cloud-level artificial intelligence.

From here, you can extend your project by:

• Adding system prompts or persona instructions to your client.models.generate_content call to give your assistant a specific attitude or expertise.
• Handling conversational history so your assistant remembers previous turns in the dialogue.
• Deploying your code onto a small edge device like a Raspberry Pi to create a custom smart speaker for your desk!