A comprehensive tutorial for building a Retrieval-Augmented Generation (RAG) service that uses Google Cloud to process documents and enable powerful semantic search with natural language responses.

Retrieval-Augmented Generation (RAG) combines search capabilities with generative AI to produce more accurate, contextualized answers. The system works by:

1. Converting documents into vector embeddings
2. Storing these embeddings in a vector database
3. Retrieving relevant information based on semantic similarity when queried
4. Using an LLM to generate natural language responses based on the retrieved context

• Process various Google Workspace documents:
  • Google Docs
  • Google Sheets
  • Google Slides
  • Microsoft Word (.docx)
  • Microsoft Excel (.xlsx)
  • Microsoft PowerPoint (.pptx)
• Generate embeddings using Hugging Face models
• Store and query vector embeddings with Chroma DB
• Generate contextual responses using OpenAI models

• Python 3.8+
• Google Cloud Service Account with access to Drive, Docs, Sheets, and Slides APIs
• OpenAI API key (for generating responses)

1. Clone this repository:

   git clone https://github.com/TribalScale/google-cloud-rag.git
   cd google-cloud-rag
   

2. Create and activate a virtual environment:

   # Using venv (Python's built-in virtual environment)
   python -m venv venv
   
   # Activate the virtual environment
   # On Windows:
   venv\Scripts\activate
   # On macOS/Linux:
   source venv/bin/activate
   

3. Install dependencies:

   pip install langchain-community langchain-core langchain-openai "langchain[text-splitters]" python-dotenv google-api-python-client google-auth-httplib2 google-auth-oauthlib openpyxl python-docx python-pptx sentence-transformers chromadb
   

4. Create .env file from template:

   cp .env.template .env
   

5. Configure your environment variables in .env:

   • GOOGLE_SERVICE_ACC: Your Google service account credentials JSON
   • OPENAI_API_KEY: Your OpenAI API key
   • CHROMA_PATH: Local path for Chroma DB storage (e.g., "./chroma_db")

from index import upload_google_drive

# Replace with your Google Drive folder ID
upload_google_drive("your-drive-folder-id")

from index import query_rag

# Ask a question based on the processed documents
query_rag("What is the revenue forecast for Q3?")

1. Document Processing (google_cloud.py): Extracts text from various Google Workspace and Microsoft Office documents.

2. Text Splitting (rag_db_service.py): Divides documents into manageable chunks for more efficient retrieval.

3. Embedding Generation (get_embeddings.py): Creates vector embeddings using the Hugging Face model.

4. Vector Storage (rag_db_service.py): Stores document chunks and their embeddings in a Chroma vector database.

5. Retrieval and Response (index.py): Retrieves relevant document chunks based on query similarity and generates natural language responses using OpenAI's models.