Merge v4.0 GenAI features into v3.0 with conditional compilation#5339
Merged
Conversation
The proxy_debug macro requires a verbosity level as the second parameter. Fixed the call in Anomaly_Detector::analyze() to include the level.
- nl2sql_unit_base-t.cpp: Initialization, configuration, persistence, error handling - nl2sql_prompt_builder-t.cpp: Prompt construction, schema context, edge cases - nl2sql_model_selection-t.cpp: Model routing logic, latency handling, fallback Tests follow ProxySQL TAP framework patterns and use CommandLine helper for environment-based configuration.
- test_nl2sql_e2e.sh: End-to-end testing with --mock and --live modes - Tests complete workflow from natural language to executed SQL - Includes test schema setup, LLM configuration, and 8 test cases - Supports both mocked LLM responses (fast) and live LLM testing
User Documentation: - README.md: Complete user guide with examples, configuration, troubleshooting Developer Documentation: - ARCHITECTURE.md: System architecture, components, flow diagrams - API.md: Complete API reference for all variables, structures, and methods - TESTING.md: Testing guide with templates and best practices All documentation follows "very very very" thorough standards with comprehensive examples, diagrams, and cross-references.
Add 'using std::vector;' declaration to resolve conflicts with macros defined in included headers.
- nl2sql_integration-t.cpp: Schema-aware conversion, multi-table queries - Tests JOIN queries, aggregations, complex patterns - Tests error recovery and cross-schema queries - 30 tests across 6 categories Tests require running ProxySQL instance with admin interface to create test schema and validate SQL generation.
Remove DATABASE_DISCOVERY_REPORT.md and DATABASE_QUESTION_CAPABILITIES.md from root directory. These were moved to scripts/mcp/DiscoveryAgent/ ClaudeCode_Headless/examples/ in commit 6dd2613 but the root copies remained as stale files.
Phase 5: MCP Tool Implementation for NL2SQL
This commit implements the AI Tool Handler for the MCP (Model Context
Protocol) server, exposing NL2SQL functionality as an MCP tool.
**New Files:**
- include/AI_Tool_Handler.h: Header for AI_Tool_Handler class
- Provides ai_nl2sql_convert tool via MCP protocol
- Wraps NL2SQL_Converter and Anomaly_Detector
- Inherits from MCP_Tool_Handler base class
- lib/AI_Tool_Handler.cpp: Implementation
- Implements ai_nl2sql_convert tool execution
- Accepts parameters: natural_language (required), schema,
context_tables, max_latency_ms, allow_cache
- Returns JSON response with sql_query, confidence, explanation,
cached, cache_id
- scripts/mcp/test_nl2sql_tools.sh: Test script for NL2SQL MCP tool
- Tests ai_nl2sql_convert via JSON-RPC over HTTPS
- 10 test cases covering SELECT, WHERE, JOIN, aggregation, etc.
- Includes error handling test for empty queries
- Supports --verbose, --quiet options
**Modified Files:**
- include/MCP_Thread.h: Add AI_Tool_Handler forward declaration and pointer
- lib/Makefile: Add AI_Tool_Handler.oo to _OBJ_CXX list
- lib/ProxySQL_MCP_Server.cpp: Initialize and register AI tool handler
- Creates AI_Tool_Handler with GloAI components
- Registers /mcp/ai endpoint
- Adds cleanup in destructor
**MCP Tool Details:**
- Endpoint: /mcp/ai
- Tool: ai_nl2sql_convert
- Parameters:
- natural_language (string, required): Natural language query
- schema (string, optional): Database schema name
- context_tables (string, optional): Comma-separated table list
- max_latency_ms (integer, optional): Max acceptable latency
- allow_cache (boolean, optional): Check semantic cache (default: true)
**Testing:**
Run the test script with:
./scripts/mcp/test_nl2sql_tools.sh [--verbose] [--quiet]
See scripts/mcp/test_nl2sql_tools.sh --help for usage.
Related: Phase 1-4 (Documentation, Unit Tests, Integration Tests, E2E Tests)
Related: Phase 6-8 (User Docs, Developer Docs, Test Docs)
Phase 3: Anomaly Detection Implementation
This commit implements a comprehensive multi-stage anomaly detection
system for real-time SQL query security analysis.
**Core Detection Methods:**
1. **SQL Injection Pattern Detection** (lib/Anomaly_Detector.cpp)
- Regex-based detection of 11 SQL injection patterns
- Suspicious keyword detection (11 patterns)
- Covers: tautologies, union-based, comment-based, stacked queries
2. **Query Normalization** (lib/Anomaly_Detector.cpp:normalize_query)
- Converts to lowercase
- Removes SQL comments
- Replaces string/numeric literals with placeholders
- Normalizes whitespace
3. **Rate Limiting** (lib/Anomaly_Detector.cpp:check_rate_limiting)
- Per user/host query rate tracking
- Configurable time windows (3600s default)
- Auto-block on threshold exceeded
- Prevents DoS and brute force attacks
4. **Statistical Anomaly Detection** (lib/Anomaly_Detector.cpp:check_statistical_anomaly)
- Z-score based outlier detection
- Abnormal execution time detection (>5s)
- Large result set detection (>10000 rows)
- Behavioral profiling per user
5. **Embedding-based Similarity** (lib/Anomaly_Detector.cpp:check_embedding_similarity)
- Placeholder for vector similarity search
- Framework for sqlite-vec integration
- Detects novel attack variations
**Query Flow Integration:**
- Added `detect_ai_anomaly()` to MySQL_Session (line 3626)
- Integrated after libinjection SQLi detection (line 5150)
- Blocks queries when risk threshold exceeded (default: 0.70)
- Sends error response with anomaly details
**Status Variables Added:**
- `ai_detected_anomalies`: Total anomalies detected
- `ai_blocked_queries`: Total queries blocked
- Available via: `SELECT * FROM stats_mysql_global`
**Configuration (defaults):**
- `enabled`: true
- `risk_threshold`: 70 (0-100)
- `similarity_threshold`: 85 (0-100)
- `rate_limit`: 100 queries/hour
- `auto_block`: true
- `log_only`: false
**Detection Pipeline:**
```
Query → SQLi Check → AI Anomaly Check → [Block if needed] → Execute
(libinjection) (Multi-stage)
```
**Files Modified:**
- include/MySQL_Session.h: Added detect_ai_anomaly() declaration
- include/MySQL_Thread.h: Added AI status variables
- lib/Anomaly_Detector.cpp: Full implementation (700+ lines)
- lib/MySQL_Session.cpp: Integration and query flow
- lib/MySQL_Thread.cpp: Status variable definitions
**Next Steps:**
- Add unit tests for each detection method
- Add integration tests with sample attacks
- Add user and developer documentation
Related: Phase 1-2 (NL2SQL foundation and testing)
Related: Phase 4 (Vector storage for embeddings)
Added 95 tests (50 unit + 45 integration) and 4 documentation files: Test Files: - test/tap/tests/anomaly_detection-t.cpp (50 unit tests) * Initialization and configuration tests * SQL injection pattern detection * Query normalization * Rate limiting * Statistical anomaly detection * Integration scenarios * Configuration management * False positive handling - test/tap/tests/anomaly_detection_integration-t.cpp (45 integration tests) * Real SQL injection pattern detection with actual queries * Legitimate query passthrough verification * Multi-user rate limiting scenarios * Statistical anomaly detection * Log-only mode configuration Documentation (doc/ANOMALY_DETECTION/): - README.md: User guide with quick start, configuration, examples - API.md: Complete API reference for Anomaly_Detector class - ARCHITECTURE.md: System architecture and design documentation - TESTING.md: Testing guide with test categories and examples All tests compile successfully and follow the TAP framework pattern used throughout ProxySQL.
Implemented semantic caching for NL2SQL using sqlite-vec and GenAI module: Changes to lib/AI_Features_Manager.cpp: - Create virtual vec0 tables for similarity search: * nl2sql_cache_vec for NL2SQL cache * anomaly_patterns_vec for threat patterns * query_history_vec for query history Changes to include/NL2SQL_Converter.h: - Add get_query_embedding() method declaration Changes to lib/NL2SQL_Converter.cpp: - Add GenAI_Thread.h include and GloGATH extern - Implement get_query_embedding() - calls GloGATH->embed_documents() - Implement check_vector_cache() - sqlite-vec KNN search with cosine distance - Implement store_in_vector_cache() - stores embedding and updates vec table - Implement clear_cache() - deletes from both main and vec tables - Implement get_cache_stats() - returns cache entry/hit counts - Add vector_to_json() helper for sqlite-vec MATCH queries Features: - Uses GenAI module (llama-server) for embedding generation - Cosine similarity search via sqlite-vec vec_distance_cosine() - Configurable similarity threshold (ai_nl2sql_cache_similarity_threshold) - Automatic hit counting and timestamp tracking
Implemented embedding-based threat pattern detection using GenAI and sqlite-vec: Changes to lib/Anomaly_Detector.cpp: - Add GenAI_Thread.h include and GloGATH extern - Implement get_query_embedding(): * Calls GloGATH->embed_documents() via llama-server * Normalizes query before embedding for better quality * Returns std::vector<float> with embedding - Implement check_embedding_similarity(): * Generates embedding for query if not provided * Performs sqlite-vec KNN search against anomaly_patterns table * Uses cosine distance (vec_distance_cosine) for similarity * Calculates risk score based on severity and distance * Returns AnomalyResult with pattern details and blocking decision - Implement add_threat_pattern(): * Generates embedding for threat pattern example * Stores pattern with embedding in anomaly_patterns table * Updates anomaly_patterns_vec virtual table for KNN search * Returns pattern ID on success Features: - Semantic similarity detection against known threat patterns - Configurable similarity threshold (ai_anomaly_similarity_threshold) - Risk scoring based on pattern severity (1-10) and similarity - Automatic threat pattern management with vector indexing
Add unit test for vector features including: - Virtual vec0 table creation verification - NL2SQL vector cache configuration tests - Anomaly embedding configuration tests - Vector database file verification - Status variables validation - Cache statistics interface tests - GenAI module availability checks 20 tests covering configuration and infrastructure validation. Tests can be extended with actual embedding generation once llama-server is running in the test environment.
Add helper script showing sample threat patterns that can be added to the Anomaly Detection system for testing embedding similarity. Includes 10 sample patterns: 1. OR 1=1 tautology (severity 9) 2. UNION SELECT data extraction (severity 8) 3. Comment injection (severity 7) 4. Sleep-based DoS (severity 6) 5. Benchmark-based DoS (severity 6) 6. INTO OUTFILE exfiltration (severity 9) 7. DROP TABLE destruction (severity 10) 8. Schema probing (severity 3) 9. CONCAT injection (severity 8) 10. Hex encoding bypass (severity 7)
Improve Anomaly_Detector with full threat pattern CRUD operations: Changes to lib/Anomaly_Detector.cpp: - Implement list_threat_patterns(): * Returns JSON array of all threat patterns * Shows pattern_name, pattern_type, query_example, severity, created_at * Ordered by severity DESC (highest risk first) - Implement remove_threat_pattern(): * Deletes from both anomaly_patterns and anomaly_patterns_vec tables * Proper error handling with error messages * Returns true on success, false on failure - Improve get_statistics(): * Add threat_patterns_count to statistics * Add threat_patterns_by_type breakdown * Shows patterns grouped by type (sql_injection, dos, etc.) - Add count_by_pattern_type query for categorization Features: - Full CRUD operations for threat patterns - JSON-formatted output for API integration - Statistics include both counts and categorization - Proper cleanup of both main and virtual tables
…rn management NL2SQL_Converter improvements: - Implement get_query_embedding() using GenAI module - Implement check_vector_cache() with KNN search via sqlite-vec - Implement store_in_vector_cache() with embedding storage - All stub methods now fully functional Anomaly_Detector improvements: - Implement add_threat_pattern() with embedding generation - Stores patterns in both main table and virtual vec table - Returns pattern ID on success, -1 on error Documentation: - Add comprehensive VECTOR_FEATURES documentation - README.md (471 lines): User guide and quick start - API.md (736 lines): Complete API reference - ARCHITECTURE.md (358 lines): System architecture - TESTING.md (767 lines): Testing guide and procedures This completes the vector features implementation, enabling: - Semantic similarity caching for NL2SQL queries - Embedding-based threat pattern detection - Full CRUD operations for threat patterns
- Change from runtime_mysql_servers with variable_name column - To mysql_servers with ai_* prefix columns - This matches the actual schema where AI variables are stored
Simple script to verify that all vector features are properly implemented: - NL2SQL vector cache methods - Anomaly threat pattern management - sqlite-vec integration - GenAI module integration - Documentation completeness
- test_external_live.sh: Interactive script for testing with custom LLM - EXTERNAL_LLM_SETUP.md: Complete guide for external model configuration Covers: - Custom LLM endpoint configuration for NL2SQL - llama-server configuration for embeddings - Architecture overview - Configuration variables - Testing procedures - Troubleshooting tips
Remove Ollama-specific provider code and use only generic OpenAI-compatible and Anthropic-compatible providers. Ollama is now used via its OpenAI-compatible endpoint at /v1/chat/completions. Changes: - Remove LOCAL_OLLAMA from ModelProvider enum - Remove ai_nl2sql_ollama_model and ai_nl2sql_ollama_url variables - Remove call_ollama() function from LLM_Clients.cpp - Update default configuration to use OpenAI provider with Ollama URL - Update all documentation to reflect generic-only approach Configuration: - ai_nl2sql_provider: 'openai' or 'anthropic' (default: 'openai') - ai_nl2sql_provider_url: endpoint URL (default: Ollama OpenAI-compatible) - ai_nl2sql_provider_model: model name - ai_nl2sql_provider_key: API key (optional for local endpoints) This simplifies the codebase by removing a separate code path for Ollama and aligns with the goal of avoiding provider-specific variables.
Add comprehensive SQL validation with confidence scoring based on: - SQL keyword detection (17 keywords covering DDL/DML/transactions) - Structural validation (balanced parentheses and quotes) - SQL injection pattern detection - Length and quality checks Confidence scoring: - Base 0.4 for valid SQL keyword - +0.15 for balanced parentheses - +0.15 for balanced quotes - +0.1 for minimum length - +0.1 for FROM clause in SELECT statements - +0.1 for no injection patterns - -0.3 penalty for injection patterns detected Low confidence (< 0.5) results are logged with detailed info. Cache storage threshold updated to 0.5 confidence (from implicit valid_sql). This improves detection of malformed or potentially malicious SQL while providing granular confidence scores for downstream use.
Add comprehensive validation for AI features configuration variables to prevent invalid states and improve error messages. Changes: - Add validate_url_format(): Checks for http:// or https:// prefix and host part - Add validate_api_key_format(): Validates API key format, checks for whitespace, minimum length, and incomplete key patterns (sk- with <20 chars, sk-ant- with <25 chars) - Add validate_numeric_range(): Validates numeric values are within min/max range - Add validate_provider_name(): Ensures provider is 'openai' or 'anthropic' - Update set_variable() to call validation functions before setting values Validated variables: - ai_nl2sql_provider: Must be 'openai' or 'anthropic' - ai_nl2sql_provider_url: Must have http:// or https:// prefix - ai_nl2sql_provider_key: No whitespace, minimum 10 chars - ai_nl2sql_cache_similarity_threshold: Range [0, 100] - ai_nl2sql_timeout_ms: Range [1000, 300000] (1 second to 5 minutes) - ai_nl2sql_max_cloud_requests_per_hour: Range [1, 10000] - ai_anomaly_similarity_threshold: Range [0, 100] - ai_anomaly_risk_threshold: Range [0, 100] - ai_anomaly_rate_limit: Range [1, 10000] - ai_vector_dimension: Range [128, 4096] This prevents misconfigurations and provides clear error messages to users when invalid values are provided. Fixes compilation issue by moving validation helper functions before set_variable() to resolve forward declaration errors.
Add comprehensive error details to help users debug NL2SQL conversion issues. Changes: - Add error_code, error_details, http_status_code, provider_used fields to NL2SQLResult - Add NL2SQLErrorCode enum with structured error codes: * SUCCESS, ERR_API_KEY_MISSING, ERR_API_KEY_INVALID, ERR_TIMEOUT * ERR_CONNECTION_FAILED, ERR_RATE_LIMITED, ERR_SERVER_ERROR * ERR_EMPTY_RESPONSE, ERR_INVALID_RESPONSE, ERR_SQL_INJECTION_DETECTED * ERR_VALIDATION_FAILED, ERR_UNKNOWN_PROVIDER, ERR_REQUEST_TOO_LARGE - Add nl2sql_error_code_to_string() function for error code conversion - Add format_error_context() helper to create detailed error messages including: * Query (truncated if too long) * Schema name * Provider attempted * Endpoint URL * Specific error message - Add set_error_details() helper to populate error fields - Update error handling in convert() to use new error details - Track provider_used in successful conversions This provides much better debugging information when NL2SQL conversions fail, making it easier to identify misconfigurations and connectivity issues. Fixes #1 - Improve Error Messages
Add comprehensive structured logging for NL2SQL LLM API calls with request correlation, timing metrics, and detailed error context. Changes: - Add request_id field to NL2SQLRequest with UUID-like auto-generation - Add structured logging macros: * LOG_LLM_REQUEST: Logs URL, model, prompt length with request ID * LOG_LLM_RESPONSE: Logs HTTP status, duration_ms, response preview * LOG_LLM_ERROR: Logs error phase, message, and status code - Update call_generic_openai() signature to accept req_id parameter - Update call_generic_anthropic() signature to accept req_id parameter - Add timing metrics to both LLM call functions using clock_gettime() - Replace existing debug logging with structured logging macros - Update convert() to pass request_id to LLM calls Request IDs are generated as UUID-like strings (e.g., "12345678-9abc-def0-1234-567890abcdef") and are included in all log messages for correlation. This allows tracking a single NL2SQL request through all log lines from request to response. Timing is measured using CLOCK_MONOTONIC for accurate duration tracking of LLM API calls, reported in milliseconds. This provides much better debugging capability when troubleshooting NL2SQL issues, as administrators can now: - Correlate all log lines for a single request - See exact timing of LLM API calls - Identify which phase of processing failed - Track request/response metrics Fixes #2 - Add Structured Logging
This commit adds configurable retry logic with exponential backoff for NL2SQL LLM API calls. Changes: - Add retry configuration to NL2SQLRequest (max_retries, retry_backoff_ms, retry_multiplier, retry_max_backoff_ms) - Add is_retryable_error() to identify retryable HTTP/CURL errors - Add sleep_with_jitter() for exponential backoff with 10% jitter - Add call_generic_openai_with_retry() wrapper - Add call_generic_anthropic_with_retry() wrapper - Update NL2SQL_Converter::convert() to use retry wrappers Default retry behavior: - 3 retries with 1000ms initial backoff - 2.0x multiplier, 30000ms max backoff - Retries on empty responses (transient failures) Part of: Phase 3 of NL2SQL improvement plan
This commit adds comprehensive unit tests for the AI configuration validation functions used in AI_Features_Manager. Changes: - Add test/tap/tests/ai_validation-t.cpp with 61 unit tests - Test URL format validation (validate_url_format) - Test API key format validation (validate_api_key_format) - Test numeric range validation (validate_numeric_range) - Test provider name validation (validate_provider_name) - Test edge cases and boundary conditions The test file is self-contained with its own copies of the validation functions to avoid complex linking dependencies on libproxysql. Test Categories: - URL validation: 15 tests (http://, https:// protocols) - API key validation: 14 tests (OpenAI, Anthropic formats) - Numeric range: 13 tests (min/max boundaries) - Provider name: 8 tests (openai, anthropic) - Edge cases: 11 tests (NULL handling, long values) All 61 tests pass successfully. Part of: Phase 4 of NL2SQL improvement plan
This commit updates the NL2SQL documentation to reflect the new features added in v0.2.0: README.md changes: - Added Request Configuration section with retry parameters - Added Error Handling section with error code table - Added Request Correlation section with log format examples - Updated Results section with error columns - Updated Troubleshooting with retry behavior documentation - Added v0.2.0 to Version History API.md changes: - Updated NL2SQLRequest struct with request_id and retry config fields - Updated NL2SQLResult struct with error details fields - Added NL2SQLErrorCode enum documentation - Updated Result Format with new columns - Expanded Error Codes section with structured error codes TESTING.md changes: - Added Validation Tests to test suite overview - Documented ai_validation-t.cpp test categories - Added instructions for running validation tests - Documented all 61 test cases across 5 categories
Add comprehensive TAP unit tests for NL2SQL internal functions: - Error code conversion (5 tests): Validate nl2sql_error_code_to_string() covers all 13 defined error codes plus UNKNOWN_ERROR - SQL validation patterns (17 tests): Test validate_and_score_sql() * Valid SELECT queries (4 tests) * Non-SELECT queries (4 tests) * Injection pattern detection (4 tests) * Edge cases (4 tests): empty, lone keyword, semicolons, complex queries - Request ID generation (12 tests): Test UUID-like ID generation * Format validation (20 assertions for 10 IDs) * Uniqueness (100 IDs checked for duplicates) * Hexadecimal character validation - Prompt building (8 tests): Test build_prompt() * Basic prompt structure (3 tests) * Schema context inclusion (3 tests) * Section ordering (1 test) * Special character handling (2 tests) Note: Tests are self-contained with standalone implementations matching the logic in NL2SQL_Converter.cpp.
…e the
MCP server was already running. The issue was caused by improper cleanup of
handler objects during reinitialization.
Root cause:
- ProxySQL_MCP_Server destructor deletes mysql_tool_handler
- The old code tried to delete handlers again after deleting the server,
causing double-free corruption
The fix properly handles handler lifecycle during reinitialization:
1. Delete Query_Tool_Handler first (server destructor doesn't clean this)
2. Delete the server (which also deletes MySQL_Tool_Handler via destructor)
3. Delete other handlers (config/admin/cache/observe) created by old server
4. Create new MySQL_Tool_Handler with updated configuration
5. Create new Query_Tool_Handler
6. Create new server (recreates all handlers with new endpoints)
This ensures proper cleanup and prevents double-free issues while allowing
runtime reconfiguration of MySQL connection parameters.
- `model` is a required parameter for embedding request in
OpenAI API specification.
- https://platform.openai.com/docs/api-reference/embeddings
Signed-off-by: Wazir Ahmed <wazir@proxysql.com>
- Implement Logger class with 5 levels (error/warn/info/debug/trace) - Add --log-level option to control verbosity - Add comprehensive logging throughout ingestion pipeline - Change transaction handling: commit per-source instead of all-in-one - Update INGEST_USAGE_GUIDE.md with logging documentation
Read only queries are no longer flagged as non-readonly when starting with double dash comments.
- `sqlite-vec` requires that `knn` queries (using the `MATCH` operator for vector similarity search) must have a `LIMIT` clause at the same query level as the `MATCH` clause. - Execute `knn` queries as a subquery and then do `JOIN`s with `rag_chunks` and `rag_documents`. Signed-off-by: Wazir Ahmed <wazir@proxysql.com>
… std::tolower UB - Remove std::cerr calls that bypassed --log-level (no more duplicate output) - Use localtime_r for thread-safe timestamp formatting - Fix std::tolower undefined behavior with unsigned char cast - Minor: add markdown language tag, standardize CLI option docs
feat: Improve logging for MCP and RAG tools
RAG: Fix query embedding and vector search
Add detailed logging and per-source commits to rag_ingest
This commit merges the experimental v4.0 GenAI/MCP features into the stable v3.0 branch using conditional compilation. All v4.0 features are disabled by default and only enabled when PROXYSQLGENAI=1 is set at compile time. Changes: Build System: - Modified main Makefile to pass PROXYSQLGENAI flag to sub-makefiles - Modified deps/Makefile to conditionally build sqlite-vec and sqlite-rembed - Modified lib/Makefile to add PSQLGA flag and include GenAI object files - Modified src/Makefile to add PSQLGA flag and conditional linking Headers (wrapped with #ifdef PROXYSQLGENAI): - All 20 new GenAI header files in include/ - Modified cpp.h, proxysql_glovars.hpp, proxysql_admin.h - Modified ProxySQL_Admin_Tables_Definitions.h for GenAI/MCP tables Source Files: - All 22 new GenAI source files in lib/ wrapped with #ifdef PROXYSQLGENAI - Modified src/main.cpp for conditional global variables and init/shutdown - Modified Admin_Handler.cpp for conditional command handlers - Modified Admin_Bootstrap.cpp for conditional table registration - Modified Admin_FlushVariables.cpp for conditional variable flushing - Modified ProxySQL_Admin.cpp for conditional admin methods - Modified ProxySQL_Admin_Stats.cpp for conditional MCP stats functions - Modified proxy_sqlite3_symbols.cpp to always compile (needed by core) - Modified MySQL_Session.cpp for conditional GenAI function calls Test Files: - Renamed test_mcp_query_rules-t to test_mcp_query_rules-t.sh - Renamed test_mcp_rag_metrics-t to test_mcp_rag_metrics-t.sh - Modified anomaly_detection-t.cpp for conditional test execution Usage: # Build without GenAI (v3.0 mode - default) make clean && make build_deps -j$(nproc) && make build_lib -j$(nproc) && make build_src -j$(nproc) # Build with GenAI (v4.0 mode) make clean && PROXYSQLGENAI=1 make build_deps -j$(nproc) && PROXYSQLGENAI=1 make build_lib -j$(nproc) && PROXYSQLGENAI=1 make build_src -j$(nproc)
This commit fixes compilation errors when building with PROXYSQLGENAI=1: Header file fixes: - Add missing #endif /* PROXYSQLGENAI */ before header guards in multiple headers - Remove #include cpp.h from GenAI headers to avoid circular dependencies Source file fixes: - Add #include proxysql.h to GenAI .cpp files that were missing it - Add #include Static_Harvester.h to Query_Tool_Handler.cpp for forward decl Build system fixes: - Remove vec.o from libproxysql.a (it's linked separately in src/Makefile) - Prevents duplicate symbol errors during linking Test files: - Rename MCP test files with .sh suffix to prevent make clean deletion Both build modes now work: - make build_lib_debug (without GenAI) - PROXYSQLGENAI=1 make build_lib_debug (with GenAI)
Add sqlite-vec and sqlite-rembed libraries to TAP test linking when PROXYSQLGENAI=1 is set. The proxy_sqlite3_symbols.cpp in libproxysql.a references sqlite3_vec_init which requires these libraries. Changes to test/tap/tests/Makefile: - Add libsqlite_rembed.a and vec.o to STATIC_LIBS when PROXYSQLGENAI=1 - Add GenAI libraries to sqlite3-t target - Add GenAI libraries to setparser_test, setparser_test2, setparser_test3 targets - Fix typo: -EXCLUDE_TRACKING_VARIABLES -> -DEXCLUDE_TRACKING_VARIABLES
|
Important Review skippedToo many files! This PR contains 248 files, which is 98 over the limit of 150. You can disable this status message by setting the
✨ Finishing touches🧪 Generate unit tests (beta)
Important Action Needed: IP Allowlist UpdateIf your organization protects your Git platform with IP whitelisting, please add the new CodeRabbit IP address to your allowlist:
Reviews will stop working after February 8, 2026 if the new IP is not added to your allowlist. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Summary of ChangesHello @renecannao, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly expands ProxySQL's capabilities by integrating experimental v4.0 Generative AI (GenAI) and Model Context Protocol (MCP) features into the stable v3.0 branch. The core strategy revolves around conditional compilation, allowing these advanced functionalities to be selectively enabled without impacting existing v3.0 builds. This includes the introduction of a comprehensive Retrieval-Augmented Generation (RAG) subsystem, enhanced build processes for new SQLite extensions like Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Changelog
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request is a significant undertaking, merging the v4.0 GenAI features into the v3.0 branch using conditional compilation. The approach of using a PROXYSQLGENAI flag in the Makefiles to control the build is sound. The PR also introduces a substantial amount of new functionality, including a proof-of-concept RAG ingestion tool and extensive documentation.
My review focuses on the core integration aspects and the newly introduced code. While the overall direction is good, I have identified a few key areas for improvement:
- A critical security vulnerability in the new
rag_ingest.cpptool due to manual SQL query construction. - Inconsistencies in versioning within the main
Makefile. - Some cleanup opportunities in the
deps/Makefileand the newly added documentation files to improve maintainability.
Please see the detailed comments for specific suggestions.
Comment on lines
+1011
to
+1038
| static void insert_doc(MySQLDB& db, | ||
| int source_id, | ||
| const std::string& source_name, | ||
| const std::string& doc_id, | ||
| const std::string& pk_json, | ||
| const std::string& title, | ||
| const std::string& body, | ||
| const std::string& meta_json) { | ||
| g_logger.debug(std::string("Inserting document: ") + doc_id + | ||
| ", title_length=" + std::to_string(title.size()) + | ||
| ", body_length=" + std::to_string(body.size())); | ||
|
|
||
| std::string e_doc_id = sql_escape_single_quotes(doc_id); | ||
| std::string e_source_name = sql_escape_single_quotes(source_name); | ||
| std::string e_pk_json = sql_escape_single_quotes(pk_json); | ||
| std::string e_title = sql_escape_single_quotes(title); | ||
| std::string e_body = sql_escape_single_quotes(body); | ||
| std::string e_meta = sql_escape_single_quotes(meta_json); | ||
|
|
||
| // Use std::ostringstream to avoid fixed buffer size issues | ||
| std::ostringstream sql; | ||
| sql << "INSERT INTO rag_documents(doc_id, source_id, source_name, pk_json, title, body, metadata_json) " | ||
| << "VALUES('" << e_doc_id << "', " << source_id << ", '" << e_source_name << "', '" | ||
| << e_pk_json << "', '" << e_title << "', '" << e_body << "', '" << e_meta << "')"; | ||
|
|
||
| db.execute(sql.str().c_str()); | ||
| g_logger.trace("Document inserted successfully"); | ||
| } |
There was a problem hiding this comment.
This function, along with others in this file (insert_chunk, insert_fts, doc_exists, etc.), constructs SQL queries by manually formatting strings. The custom sql_escape_single_quotes function is used for escaping, but this approach is highly vulnerable to SQL injection attacks. A purpose-built escaping function from the database driver is always safer, but using prepared statements is the best practice.
An attacker could potentially craft input (e.g., in table names, column names, or data) that bypasses the custom escaping and injects malicious SQL.
Recommendation:
Refactor this and all other database query functions in this file to use prepared statements. The MariaDB C client library provides a robust API for this (mysql_stmt_init, mysql_stmt_prepare, mysql_stmt_bind_param, mysql_stmt_execute). This will eliminate the risk of SQL injection and improve performance and reliability.
static void insert_doc(MySQLDB& db,
int source_id,
const std::string& source_name,
const std::string& doc_id,
const std::string& pk_json,
const std::string& title,
const std::string& body,
const std::string& meta_json) {
g_logger.debug(std::string("Inserting document: ") + doc_id +
", title_length=" + std::to_string(title.size()) +
", body_length=" + std::to_string(body.size()));
MYSQL_STMT* stmt = mysql_stmt_init(db.conn);
if (!stmt) {
g_logger.error("mysql_stmt_init(), out of memory");
fatal("mysql_stmt_init() failed");
}
const char* sql = "INSERT INTO rag_documents(doc_id, source_id, source_name, pk_json, title, body, metadata_json) VALUES(?, ?, ?, ?, ?, ?, ?)";
if (mysql_stmt_prepare(stmt, sql, strlen(sql))) {
g_logger.error(std::string("mysql_stmt_prepare() failed: ") + mysql_stmt_error(stmt));
mysql_stmt_close(stmt);
fatal("mysql_stmt_prepare() failed");
}
MYSQL_BIND bind[7];
memset(bind, 0, sizeof(bind));
bind[0].buffer_type = MYSQL_TYPE_STRING;
bind[0].buffer = (char*)doc_id.c_str();
bind[0].buffer_length = doc_id.length();
bind[1].buffer_type = MYSQL_TYPE_LONG;
bind[1].buffer = (char*)&source_id;
bind[2].buffer_type = MYSQL_TYPE_STRING;
bind[2].buffer = (char*)source_name.c_str();
bind[2].buffer_length = source_name.length();
bind[3].buffer_type = MYSQL_TYPE_STRING;
bind[3].buffer = (char*)pk_json.c_str();
bind[3].buffer_length = pk_json.length();
bind[4].buffer_type = MYSQL_TYPE_STRING;
bind[4].buffer = (char*)title.c_str();
bind[4].buffer_length = title.length();
bind[5].buffer_type = MYSQL_TYPE_STRING;
bind[5].buffer = (char*)body.c_str();
bind[5].buffer_length = body.length();
bind[6].buffer_type = MYSQL_TYPE_STRING;
bind[6].buffer = (char*)meta_json.c_str();
bind[6].buffer_length = meta_json.length();
if (mysql_stmt_bind_param(stmt, bind)) {
g_logger.error(std::string("mysql_stmt_bind_param() failed: ") + mysql_stmt_error(stmt));
mysql_stmt_close(stmt);
fatal("mysql_stmt_bind_param() failed");
}
if (mysql_stmt_execute(stmt)) {
g_logger.error(std::string("mysql_stmt_execute() failed: ") + mysql_stmt_error(stmt));
mysql_stmt_close(stmt);
fatal("mysql_stmt_execute() failed");
}
mysql_stmt_close(stmt);
g_logger.trace("Document inserted successfully");
}| NO_DEBUG := $(O2) -ggdb | ||
| DEBUG := $(ALL_DEBUG) | ||
| CURVER ?= 3.0.6 | ||
| CURVER ?= 4.0.0 |
There was a problem hiding this comment.
The version is hardcoded to 4.0.0. However, the pull request description states that when the PROXYSQLGENAI flag is not set, the build should produce a binary 'identical to v3.0'. A different version number makes the binaries not identical.
To align with the goal of this PR, consider making the version conditional. For example, you could default to a v3.x version and only use v4.0.0 when PROXYSQLGENAI=1 is set.
ifeq ($(PROXYSQLGENAI),1)
CURVER ?= 4.0.0
else
CURVER ?= 3.0.6
endifThis would make the versioning scheme clearer and consistent with the conditional compilation approach.
CURVER ?= 3.0.6
Comment on lines
+267
to
+278
| sqlite3/sqlite3/vec.o: sqlite3/sqlite3/sqlite3.o | ||
| cd sqlite3/sqlite3 && cp ../sqlite-vec-source/sqlite-vec.c . && cp ../sqlite-vec-source/sqlite-vec.h . | ||
| cd sqlite3/sqlite3 && ${CC} ${MYCFLAGS} -fPIC -c -o vec.o sqlite-vec.c -DSQLITE_CORE -DSQLITE_VEC_STATIC -DSQLITE_ENABLE_MEMORY_MANAGEMENT -DSQLITE_ENABLE_JSON1 -DSQLITE_ENABLE_FTS5 -DSQLITE_DLL=1 | ||
|
|
||
| sqlite3/libsqlite_rembed.a: sqlite3/sqlite-rembed-0.0.1-alpha.9.tar.gz | ||
| cd sqlite3 && rm -rf sqlite-rembed-*/ sqlite-rembed-source/ || true | ||
| cd sqlite3 && tar -zxf sqlite-rembed-0.0.1-alpha.9.tar.gz | ||
| mv sqlite3/sqlite-rembed-0.0.1-alpha.9 sqlite3/sqlite-rembed-source | ||
| cd sqlite3/sqlite-rembed-source && SQLITE3_INCLUDE_DIR=$(SQLITE3_INCLUDE_DIR) SQLITE3_LIB_DIR=$(SQLITE3_LIB_DIR) SQLITE3_STATIC=1 $(CARGO) build --release --features=sqlite-loadable/static --lib | ||
| cp sqlite3/sqlite-rembed-source/target/release/libsqlite_rembed.a sqlite3/libsqlite_rembed.a | ||
|
|
||
| sqlite3: sqlite3/sqlite3/sqlite3.o sqlite3/sqlite3/vec.o sqlite3/libsqlite_rembed.a |
There was a problem hiding this comment.
The build rules for sqlite3/sqlite3/vec.o and sqlite3/libsqlite_rembed.a are duplicated here and later in this file (lines 365-379). This redundancy can lead to confusion and maintenance issues.
To improve clarity and maintainability, I recommend removing this block. The definitions on lines 365-379 are correctly associated with the sqlite-vec and sqlite-rembed targets and should be the single source of truth.
Additionally, the sqlite3 target on line 278 should likely be reverted to its original state to avoid building GenAI dependencies unconditionally:
sqlite3: sqlite3/sqlite3/sqlite3.o
Comment on lines
+1
to
+130
| # ProxySQL RAG Subsystem Implementation - Complete | ||
|
|
||
| ## Implementation Status: COMPLETE | ||
|
|
||
| I have successfully implemented the ProxySQL RAG (Retrieval-Augmented Generation) subsystem according to the requirements specified in the blueprint documents. Here's what has been accomplished: | ||
|
|
||
| ## Core Components Implemented | ||
|
|
||
| ### 1. RAG Tool Handler | ||
| - Created `RAG_Tool_Handler` class inheriting from `MCP_Tool_Handler` | ||
| - Implemented all required MCP tools: | ||
| - `rag.search_fts` - Keyword search using FTS5 | ||
| - `rag.search_vector` - Semantic search using vector embeddings | ||
| - `rag.search_hybrid` - Hybrid search with two modes (fuse and fts_then_vec) | ||
| - `rag.get_chunks` - Fetch chunk content | ||
| - `rag.get_docs` - Fetch document content | ||
| - `rag.fetch_from_source` - Refetch authoritative data | ||
| - `rag.admin.stats` - Operational statistics | ||
|
|
||
| ### 2. Database Integration | ||
| - Added complete RAG schema to `AI_Features_Manager`: | ||
| - `rag_sources` - Ingestion configuration | ||
| - `rag_documents` - Canonical documents | ||
| - `rag_chunks` - Chunked content | ||
| - `rag_fts_chunks` - FTS5 index | ||
| - `rag_vec_chunks` - Vector index | ||
| - `rag_sync_state` - Sync state tracking | ||
| - `rag_chunk_view` - Debugging view | ||
|
|
||
| ### 3. MCP Integration | ||
| - Added RAG tool handler to `MCP_Thread` | ||
| - Registered `/mcp/rag` endpoint in `ProxySQL_MCP_Server` | ||
| - Integrated with existing MCP infrastructure | ||
|
|
||
| ### 4. Configuration | ||
| - Added RAG configuration variables to `GenAI_Thread`: | ||
| - `genai_rag_enabled` | ||
| - `genai_rag_k_max` | ||
| - `genai_rag_candidates_max` | ||
| - `genai_rag_query_max_bytes` | ||
| - `genai_rag_response_max_bytes` | ||
| - `genai_rag_timeout_ms` | ||
|
|
||
| ## Key Features | ||
|
|
||
| ### Search Capabilities | ||
| - **FTS Search**: Full-text search using SQLite FTS5 | ||
| - **Vector Search**: Semantic search using sqlite3-vec | ||
| - **Hybrid Search**: Two modes: | ||
| - Fuse mode: Parallel FTS + vector with Reciprocal Rank Fusion | ||
| - FTS-then-vector mode: Candidate generation + rerank | ||
|
|
||
| ### Security Features | ||
| - Input validation and sanitization | ||
| - Query length limits | ||
| - Result size limits | ||
| - Timeouts for all operations | ||
| - Column whitelisting for refetch operations | ||
| - Row and byte limits | ||
|
|
||
| ### Performance Features | ||
| - Proper use of prepared statements | ||
| - Connection management | ||
| - SQLite3-vec integration | ||
| - FTS5 integration | ||
| - Proper indexing strategies | ||
|
|
||
| ## Testing and Documentation | ||
|
|
||
| ### Test Scripts | ||
| - `scripts/mcp/test_rag.sh` - Tests RAG functionality via MCP endpoint | ||
| - `test/test_rag_schema.cpp` - Tests RAG database schema creation | ||
| - `test/build_rag_test.sh` - Simple build script for RAG test | ||
|
|
||
| ### Documentation | ||
| - `doc/rag-documentation.md` - Comprehensive RAG documentation | ||
| - `doc/rag-examples.md` - Examples of using RAG tools | ||
| - Updated `scripts/mcp/README.md` to include RAG in architecture | ||
|
|
||
| ## Files Created/Modified | ||
|
|
||
| ### New Files (10) | ||
| 1. `include/RAG_Tool_Handler.h` - Header file | ||
| 2. `lib/RAG_Tool_Handler.cpp` - Implementation file | ||
| 3. `doc/rag-documentation.md` - Documentation | ||
| 4. `doc/rag-examples.md` - Usage examples | ||
| 5. `scripts/mcp/test_rag.sh` - Test script | ||
| 6. `test/test_rag_schema.cpp` - Schema test | ||
| 7. `test/build_rag_test.sh` - Build script | ||
| 8. `RAG_IMPLEMENTATION_SUMMARY.md` - Implementation summary | ||
| 9. `RAG_FILE_SUMMARY.md` - File summary | ||
| 10. Updated `test/Makefile` - Added RAG test target | ||
|
|
||
| ### Modified Files (7) | ||
| 1. `include/MCP_Thread.h` - Added RAG tool handler member | ||
| 2. `lib/MCP_Thread.cpp` - Added initialization/cleanup | ||
| 3. `lib/ProxySQL_MCP_Server.cpp` - Registered RAG endpoint | ||
| 4. `lib/AI_Features_Manager.cpp` - Added RAG schema | ||
| 5. `include/GenAI_Thread.h` - Added RAG config variables | ||
| 6. `lib/GenAI_Thread.cpp` - Added RAG config initialization | ||
| 7. `scripts/mcp/README.md` - Updated documentation | ||
|
|
||
| ## Usage | ||
|
|
||
| To enable RAG functionality: | ||
|
|
||
| ```sql | ||
| -- Enable GenAI module | ||
| SET genai.enabled = true; | ||
|
|
||
| -- Enable RAG features | ||
| SET genai.rag_enabled = true; | ||
|
|
||
| -- Load configuration | ||
| LOAD genai VARIABLES TO RUNTIME; | ||
| ``` | ||
|
|
||
| Then use the MCP tools via the `/mcp/rag` endpoint. | ||
|
|
||
| ## Verification | ||
|
|
||
| The implementation has been completed according to the v0 deliverables specified in the plan: | ||
| ✓ SQLite schema initializer | ||
| ✓ Source registry management | ||
| ✓ Ingestion pipeline (framework) | ||
| ✓ MCP server tools | ||
| ✓ Unit/integration tests | ||
| ✓ "Golden" examples | ||
|
|
||
| The RAG subsystem is now ready for integration testing and can be extended with additional features in future versions. No newline at end of file |
There was a problem hiding this comment.
This pull request adds several markdown files that appear to summarize the RAG implementation (RAG_COMPLETION_SUMMARY.md, RAG_FILE_SUMMARY.md, RAG_IMPLEMENTATION_COMPLETE.md, RAG_IMPLEMENTATION_SUMMARY.md). The content seems largely duplicated across these files.
To improve repository clarity and reduce maintenance overhead, consider consolidating these into a single, comprehensive summary document. This will make it easier for future developers to understand the work that was done.
Comment on lines
+1
to
+245
| # sqlite-rembed Integration Test Suite | ||
|
|
||
| ## Overview | ||
|
|
||
| This test suite comprehensively validates the integration of `sqlite-rembed` (Rust SQLite extension for text embedding generation) into ProxySQL. The tests verify the complete AI pipeline from client registration to embedding generation and vector similarity search. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| ### System Requirements | ||
| - **ProxySQL** compiled with `sqlite-rembed` and `sqlite-vec` extensions | ||
| - **MySQL client** (`mysql` command line tool) | ||
| - **Bash** shell environment | ||
| - **Network access** to embedding API endpoint (or local Ollama/OpenAI API) | ||
|
|
||
| ### ProxySQL Configuration | ||
| Ensure ProxySQL is running with SQLite3 server enabled: | ||
| ```bash | ||
| cd /home/rene/proxysql-vec/src | ||
| ./proxysql --sqlite3-server | ||
| ``` | ||
|
|
||
| ### Test Configuration | ||
| The test script uses default connection parameters: | ||
| - Host: `127.0.0.1` | ||
| - Port: `6030` (default SQLite3 server port) | ||
| - User: `root` | ||
| - Password: `root` | ||
|
|
||
| Modify these in the script if your configuration differs. | ||
|
|
||
| ## Test Suite Structure | ||
|
|
||
| The test suite is organized into 9 phases, each testing specific components: | ||
|
|
||
| ### Phase 1: Basic Connectivity and Function Verification | ||
| - ✅ ProxySQL connection | ||
| - ✅ Database listing | ||
| - ✅ `sqlite-vec` function availability | ||
| - ✅ `sqlite-rembed` function registration | ||
| - ✅ `temp.rembed_clients` virtual table existence | ||
|
|
||
| ### Phase 2: Client Configuration | ||
| - ✅ Create embedding API client with `rembed_client_options()` | ||
| - ✅ Verify client registration in `temp.rembed_clients` | ||
| - ✅ Test `rembed_client_options` function | ||
|
|
||
| ### Phase 3: Embedding Generation Tests | ||
| - ✅ Generate embeddings for short and long text | ||
| - ✅ Verify embedding data type (BLOB) and size (768 dimensions × 4 bytes) | ||
| - ✅ Error handling for non-existent clients | ||
|
|
||
| ### Phase 4: Table Creation and Data Storage | ||
| - ✅ Create regular table for document storage | ||
| - ✅ Create virtual vector table using `vec0` | ||
| - ✅ Insert test documents with diverse content | ||
|
|
||
| ### Phase 5: Embedding Generation and Storage | ||
| - ✅ Generate embeddings for all documents | ||
| - ✅ Store embeddings in vector table | ||
| - ✅ Verify embedding count matches document count | ||
| - ✅ Check embedding storage format | ||
|
|
||
| ### Phase 6: Similarity Search Tests | ||
| - ✅ Exact self-match (document with itself, distance = 0.0) | ||
| - ✅ Similarity search with query text | ||
| - ✅ Verify result ordering by ascending distance | ||
|
|
||
| ### Phase 7: Edge Cases and Error Handling | ||
| - ✅ Empty text input | ||
| - ✅ Very long text input | ||
| - ✅ SQL injection attempt safety | ||
|
|
||
| ### Phase 8: Performance and Concurrency | ||
| - ✅ Sequential embedding generation timing | ||
| - ✅ Basic performance validation (< 10 seconds for 3 embeddings) | ||
|
|
||
| ### Phase 9: Cleanup and Final Verification | ||
| - ✅ Clean up test tables | ||
| - ✅ Verify no test artifacts remain | ||
|
|
||
| ## Usage | ||
|
|
||
| ### Running the Full Test Suite | ||
| ```bash | ||
| cd /home/rene/proxysql-vec/doc | ||
| ./sqlite-rembed-test.sh | ||
| ``` | ||
|
|
||
| ### Expected Output | ||
| The script provides color-coded output: | ||
| - 🟢 **Green**: Test passed | ||
| - 🔴 **Red**: Test failed | ||
| - 🔵 **Blue**: Information and headers | ||
| - 🟡 **Yellow**: Test being executed | ||
|
|
||
| ### Exit Codes | ||
| - `0`: All tests passed | ||
| - `1`: One or more tests failed | ||
| - `2`: Connection issues or missing dependencies | ||
|
|
||
| ## Configuration | ||
|
|
||
| ### Modifying Connection Parameters | ||
| Edit the following variables in `sqlite-rembed-test.sh`: | ||
| ```bash | ||
| PROXYSQL_HOST="127.0.0.1" | ||
| PROXYSQL_PORT="6030" | ||
| MYSQL_USER="root" | ||
| MYSQL_PASS="root" | ||
| ``` | ||
|
|
||
| ### API Configuration | ||
| The test uses a synthetic OpenAI endpoint by default. Set `API_KEY` environment variable or modify the variable below to use your own API: | ||
| ```bash | ||
| API_CLIENT_NAME="test-client-$(date +%s)" | ||
| API_FORMAT="openai" | ||
| API_URL="https://api.synthetic.new/openai/v1/embeddings" | ||
| API_KEY="${API_KEY:-YOUR_API_KEY}" # Uses environment variable or placeholder | ||
| API_MODEL="hf:nomic-ai/nomic-embed-text-v1.5" | ||
| VECTOR_DIMENSIONS=768 | ||
| ``` | ||
|
|
||
| For other providers (Ollama, Cohere, Nomic), adjust the format and URL accordingly. | ||
|
|
||
| ## Test Data | ||
|
|
||
| ### Sample Documents | ||
| The test creates 4 sample documents: | ||
| 1. **Machine Learning** - "Machine learning algorithms improve with more training data..." | ||
| 2. **Database Systems** - "Database management systems efficiently store, retrieve..." | ||
| 3. **Artificial Intelligence** - "AI enables computers to perform tasks typically..." | ||
| 4. **Vector Databases** - "Vector databases enable similarity search for embeddings..." | ||
|
|
||
| ### Query Texts | ||
| Test searches use: | ||
| - Self-match: Document 1 with itself | ||
| - Query: "data science and algorithms" | ||
|
|
||
| ## Troubleshooting | ||
|
|
||
| ### Common Issues | ||
|
|
||
| #### 1. Connection Failed | ||
| ``` | ||
| Error: Cannot connect to ProxySQL at 127.0.0.1:6030 | ||
| ``` | ||
| **Solution**: Ensure ProxySQL is running with `--sqlite3-server` flag. | ||
|
|
||
| #### 2. Missing Functions | ||
| ``` | ||
| ERROR 1045 (28000): no such function: rembed | ||
| ``` | ||
| **Solution**: Verify `sqlite-rembed` was compiled and linked into ProxySQL binary. | ||
|
|
||
| #### 3. API Errors | ||
| ``` | ||
| Error from embedding API | ||
| ``` | ||
| **Solution**: Check network connectivity and API credentials. | ||
|
|
||
| #### 4. Vector Table Errors | ||
| ``` | ||
| ERROR 1045 (28000): A LIMIT or 'k = ?' constraint is required on vec0 knn queries. | ||
| ``` | ||
| **Solution**: All `sqlite-vec` similarity queries require `LIMIT` clause. | ||
|
|
||
| ### Debug Mode | ||
| For detailed debugging, run with trace: | ||
| ```bash | ||
| bash -x ./sqlite-rembed-test.sh | ||
| ``` | ||
|
|
||
| ## Integration with CI/CD | ||
|
|
||
| The test script can be integrated into CI/CD pipelines: | ||
|
|
||
| ```yaml | ||
| # Example GitHub Actions workflow | ||
| name: sqlite-rembed Tests | ||
| on: [push, pull_request] | ||
| jobs: | ||
| test: | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - uses: actions/checkout@v3 | ||
| - name: Build ProxySQL with sqlite-rembed | ||
| run: | | ||
| cd deps && make cleanpart && make sqlite3 | ||
| cd ../lib && make | ||
| cd ../src && make | ||
| - name: Start ProxySQL | ||
| run: | | ||
| cd src && ./proxysql --sqlite3-server & | ||
| sleep 5 | ||
| - name: Run Integration Tests | ||
| run: | | ||
| cd doc && ./sqlite-rembed-test.sh | ||
| ``` | ||
|
|
||
| ## Extending the Test Suite | ||
|
|
||
| ### Adding New Tests | ||
| 1. Add new test function following existing pattern | ||
| 2. Update phase header and test count | ||
| 3. Add to appropriate phase section | ||
|
|
||
| ### Testing Different Providers | ||
| Modify the API configuration block to test: | ||
| - **Ollama**: Use `format='ollama'` and local URL | ||
| - **Cohere**: Use `format='cohere'` and appropriate model | ||
| - **Nomic**: Use `format='nomic'` and Nomic API endpoint | ||
|
|
||
| ### Performance Testing | ||
| Extend Phase 8 for: | ||
| - Concurrent embedding generation | ||
| - Batch processing tests | ||
| - Memory usage monitoring | ||
|
|
||
| ## Results Interpretation | ||
|
|
||
| ### Success Criteria | ||
| - All connectivity tests pass | ||
| - Embeddings generated with correct dimensions | ||
| - Vector search returns ordered results | ||
| - No test artifacts remain after cleanup | ||
|
|
||
| ### Performance Benchmarks | ||
| - Embedding generation: < 3 seconds per request (network-dependent) | ||
| - Similarity search: < 100ms for small datasets | ||
| - Memory: Stable during sequential operations | ||
|
|
||
| ## References | ||
|
|
||
| - [sqlite-rembed GitHub](https://github.com/asg017/sqlite-rembed) | ||
| - [sqlite-vec Documentation](./SQLite3-Server.md) | ||
| - [ProxySQL SQLite3 Server](./SQLite3-Server.md) | ||
| - [Integration Documentation](./sqlite-rembed-integration.md) | ||
|
|
||
| ## License | ||
|
|
||
| This test suite is part of the ProxySQL project and follows the same licensing terms. | ||
|
|
||
| --- | ||
| *Last Updated: $(date)* | ||
| *Test Suite Version: 1.0* No newline at end of file |
There was a problem hiding this comment.
This README file appears to be for a test script that is not included in the pull request (sqlite-rembed-test.sh). It also contains a hardcoded local path (/home/rene/proxysql-vec/doc).
If this file is relevant, please consider adding the corresponding test script and removing the hardcoded path. If it's not relevant to this PR, it might be better to remove it to avoid confusion.
Signed-off-by: René Cannaò <rene@proxysql.com>
This commit addresses critical issues identified in the PR review: 1. Build system redundancy (src/Makefile): - Remove redundant PROXYSQLGENAI conditionals in linking commands - The flag doesn't affect linking, so simplify to PROXYSQLCLICKHOUSE only 2. Missing PROXYSQLGENAI guard (lib/Admin_Handler.cpp): - Add #ifdef PROXYSQLGENAI around MCP VARIABLES DISK commands - Ensures MCP commands are only available when GenAI is enabled 3. Broken retry logic (lib/LLM_Clients.cpp): - Remove misleading is_retryable_error() checks that used stale values - last_curl_code and last_http_code were never updated - Simplify to retry on empty responses only (documented limitation) - Fix thread-safety: use thread_local std::mt19937 instead of rand() 4. Resource leak (lib/MySQL_Tool_Handler.cpp): - Clean up previously created connections on init failure - Both mysql_init and mysql_real_connect error paths now clean up 5. Race condition (lib/Query_Tool_Handler.cpp): - Add missing pool_lock in find_connection() - Prevents race condition when accessing connection_pool Fixes identified by automated PR review agents.
3 tasks
When PROXYSQLGENAI=1 is set, the major version number in GIT_VERSION is incremented by 1. For example: - 3.0.6-386-g79756e7 becomes 4.0.6-386-g79756e7 - 2.7.1-123-abc1234 becomes 3.7.1-123-abc1234 This allows distinguishing GenAI-enabled builds from standard builds through the version string.
- Add PROXYSQLGENAI to docker-compose.yml environment variables - Modify deb-compliant, rhel-compliant, and suse-compliant entrypoints to pass PROXYSQLGENAI=1 to make commands when set This allows building packages with GenAI support using: PROXYSQLGENAI=1 make ubuntu24 PROXYSQLGENAI=1 make debian12-dbg etc.
The version was being incremented twice when building with PROXYSQLGENAI=1: 1. First on the host (e.g., 3.0.6 -> 4.0.6) 2. Again inside Docker container (4.0.6 -> 5.0.6) The fix uses `$(origin GIT_VERSION)` to detect if GIT_VERSION was passed from the environment (Docker container) or defined in the Makefile (host). The increment only happens when: - PROXYSQLGENAI=1 is set, AND - GIT_VERSION was defined in the Makefile (origin != environment) Changes: - Makefile: Check $(origin GIT_VERSION) before incrementing - docker-compose.yml: Pass GIT_VERSION to containers so they recognize it came from environment This ensures `PROXYSQLGENAI=1 make ubuntu24` produces the correct package name (e.g., proxysql_4.0.6-ubuntu24_amd64.deb).
When running `PROXYSQLGENAI=1 make packages`, the flag needs to be passed to all sub-make processes spawned for each distribution. Adding `export PROXYSQLGENAI` ensures that sub-makes inherit the flag, so all packages are built with GenAI support and the correct version number (e.g., 4.0.6 instead of 3.0.6). This allows building all packages with GenAI using: PROXYSQLGENAI=1 make packages Individual package builds still work: PROXYSQLGENAI=1 make ubuntu24 PROXYSQLGENAI=1 make debian12-dbg etc.
Fix critical issues in GenAI code
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR merges the experimental v4.0 GenAI/MCP features into the stable v3.0 branch using conditional compilation.
Overview
PROXYSQLGENAI=1is set at compile timeKey Changes
PROXYSQLGENAIflag to build system (Makefile, deps/Makefile, lib/Makefile, src/Makefile)#ifdef PROXYSQLGENAI.shsuffix to preventmake cleandeletionBuild Instructions
Without GenAI (v3.0 mode - default)
With GenAI (v4.0 mode)
Components Included
Testing