Mind map generator

Generate a mind map from file

Comment

Mind Map Generator - Text-to-Knowledge Graph API

Inspiration

The inspiration for this project came from the overwhelming amount of unstructured text data we encounter daily - from research papers and business documents to PDFs and articles. Traditional text processing tools often fall short in helping users understand the relationships between different concepts and entities within large documents. We envisioned a solution that could automatically extract meaningful knowledge graphs from any text, transforming linear information into visual, interconnected mind maps that reveal hidden patterns and relationships. This would empower researchers, students, and professionals to quickly grasp complex information and discover insights that might otherwise remain buried in walls of text.

What it does

Mind Map Generator is a serverless text-to-knowledge graph API that transforms unstructured text into structured, shareable knowledge graphs. The system:

  • Extracts Knowledge Graphs: Uses advanced AI (GPT-4) to identify entities, relationships, and concepts from any text input
  • Processes Multiple Formats: Accepts direct text input or uploaded PDF files for automatic text extraction and analysis
  • Generates Visual Data: Creates structured graph data with nodes (entities) and edges (relationships) that can be visualized as mind maps
  • Enables Sharing: Provides shareable links for knowledge graphs with built-in expiration and view tracking
  • Supports Real-time Processing: Offers asynchronous file processing with polling endpoints to handle large documents
  • Provides RESTful Access: Delivers a complete API with CORS support for easy frontend integration

The generated knowledge graphs reveal entities like people, organizations, concepts, and locations, along with their relationships such as "WORKS_AT," "GRADUATED_FROM," or "SPECIALIZES_IN," making complex information instantly more comprehensible.

How we built it

We built this project using a modern serverless architecture on AWS, leveraging multiple cutting-edge technologies:

Backend Architecture:

  • AWS Lambda: Serverless functions for all API endpoints and processing logic
  • API Gateway: RESTful API management with automatic CORS handling
  • AWS SAM (Serverless Application Model): Infrastructure as code for deployment and resource management
  • DynamoDB: NoSQL database for caching processed graphs and managing share links
  • S3: Object storage for file uploads and processed results

AI and Processing:

  • LangChain: Framework for building AI-powered applications with document processing capabilities
  • OpenAI GPT-4: Large language model for intelligent entity and relationship extraction
  • AWS Secrets Manager: Secure storage for API keys and sensitive configuration

File Processing:

  • PyPDF2: Python library for PDF text extraction
  • Lambda Layers: Optimized dependency packaging for faster cold starts
  • Asynchronous Processing: Event-driven architecture for handling large file uploads

Development Tools:

  • Python 3.11: Primary programming language with async/await support
  • Docker: Containerized layer building for consistent Lambda deployments
  • Git: Version control with comprehensive project history

The architecture follows microservices principles with each Lambda function handling specific responsibilities: health checks, text processing, file uploads, share link generation, and graph retrieval.

Challenges we ran into

CORS Configuration Complexity: One of the biggest challenges was properly configuring CORS for browser-based access. We had to navigate the intricate relationship between API Gateway settings, Lambda response headers, and preflight OPTIONS requests to ensure seamless cross-origin access.

Lambda Layer Dependencies: Managing AI and ML dependencies in Lambda layers proved challenging due to size constraints and platform compatibility. We had to create Docker-based build processes to ensure Linux-compatible packages and optimize layer sizes for faster cold starts.

DynamoDB Schema Design: Designing an efficient DynamoDB schema that supports both direct access by share_id and querying by file_id required careful consideration of Global Secondary Indexes and access patterns.

Asynchronous Processing Flow: Implementing the polling mechanism for file processing required balancing user experience with resource efficiency. We had to design a system that provides real-time feedback while handling variable processing times for different document sizes.

Error Handling and Resilience: Building robust error handling across multiple AWS services, including proper retry logic, timeout management, and user-friendly error messages, required extensive testing and iteration.

AI Model Integration: Integrating LangChain with OpenAI's GPT-4 for consistent and accurate knowledge graph extraction required fine-tuning prompts and handling various text formats and edge cases.

Accomplishments that we're proud of

Seamless AI Integration: Successfully integrated cutting-edge AI technology (GPT-4) with serverless architecture to create intelligent, automated knowledge extraction that rivals human-level understanding of text relationships.

Scalable Architecture: Built a truly serverless solution that can automatically scale from zero to thousands of concurrent users without any infrastructure management, while maintaining cost efficiency through pay-per-use pricing.

Complete API Ecosystem: Delivered a production-ready API with comprehensive documentation, proper error handling, security considerations, and real-world usability features like share links and view tracking.

Cross-Platform Compatibility: Achieved full CORS compliance and created detailed integration guides that enable frontend developers to easily integrate the API into web applications, mobile apps, or any HTTP-capable platform.

Robust File Processing: Implemented sophisticated asynchronous file processing that handles various document types, provides real-time status updates, and gracefully manages processing failures.

Developer Experience: Created extensive documentation, example code, and testing tools that make the API accessible to developers of all skill levels.

What we learned

Serverless Best Practices: Gained deep insights into Lambda function optimization, including cold start mitigation, memory allocation strategies, and effective layer management for complex dependencies.

AI/ML in Production: Learned how to integrate large language models into production systems, including prompt engineering, response parsing, error handling, and cost optimization strategies.

API Design Philosophy: Developed expertise in designing intuitive, RESTful APIs that balance functionality with simplicity, including proper status code usage, consistent response formats, and comprehensive error messaging.

AWS Service Integration: Mastered the art of orchestrating multiple AWS services (Lambda, API Gateway, DynamoDB, S3, Secrets Manager) to create cohesive, reliable applications.

Asynchronous System Design: Learned to design systems that provide excellent user experience for long-running operations through polling, status tracking, and progressive feedback.

Frontend-Backend Communication: Gained valuable experience in CORS configuration, API security, and creating developer-friendly integration experiences.

What's next for Mind Map Generator

Enhanced Visualization: Develop a dedicated frontend application with interactive mind map visualization, allowing users to explore knowledge graphs through drag-and-drop interfaces, zoom capabilities, and customizable layouts.

Advanced AI Features:

  • Implement sentiment analysis to add emotional context to relationships
  • Add support for multiple languages and cross-language knowledge extraction
  • Integrate image and multimedia content analysis for richer knowledge graphs

Enterprise Features:

  • Add user authentication and team collaboration capabilities
  • Implement role-based access control and private graph sharing
  • Create API key management and usage analytics dashboards

Extended File Support:

  • Add support for Word documents, PowerPoint presentations, and Excel files
  • Implement OCR for scanned documents and images
  • Support for web page URL analysis and real-time content extraction

Integration Ecosystem:

  • Build plugins for popular tools like Notion, Confluence, and Google Workspace
  • Create Zapier integrations for workflow automation
  • Develop mobile SDKs for iOS and Android applications

Performance Optimization:

  • Implement caching strategies for frequently accessed graphs
  • Add support for real-time collaborative editing of knowledge graphs
  • Optimize processing times through parallel processing and model fine-tuning

Analytics and Insights:

  • Add graph analytics to identify key entities and relationship patterns
  • Implement trend analysis across multiple documents
  • Create recommendation engines for related content discovery

The ultimate vision is to create a comprehensive knowledge management platform that transforms how individuals and organizations interact with information, making complex data instantly accessible and actionable through intelligent visualization.

Built With

Share this project:

Updates