rag_ingest: convert from SQLite3 API to MySQL protocol (SQLite Server)#5330
Conversation
…SQLite3 API Convert rag_ingest from direct SQLite database access to MySQL protocol communication with SQLite Server. Major changes: - Replace SQLite3 direct API with MySQL/mariadb client library - Update connection flow: rag_ingest -> MySQL protocol -> SQLite Server -> RAG DB - Remove sqlite3 amalgamation dependency from rag_ingest target - Keep all ingestion functionality intact (chunking, FTS5, vec0 embeddings) Schema initialization: - Add init command to create RAG schema via MySQL protocol - Add ingest command to run data ingestion pipeline - Add query command for vector similarity search with embeddings
Add logs to track embedding generation progress: - Show batch size when collecting chunks for embedding - Display OpenAI API call details (endpoint, model, chunk count) - Confirm successful storage after each batch
Add test_rag_ingest_sqlite_server.sh with 5 test phases: - Phase 1: Basic ingestion and watermark sync - Phase 2: WHERE filter support - Phase 3: Chunking support - Phase 4: Stub embeddings - Phase 5: OpenAI-compatible embeddings with vector search Tests rag_ingest tool connecting to ProxySQL SQLite3 Server (port 6030) using MySQL protocol, with backend MySQL as data source. Added documentation
Add verify_sqlite_server() method to MySQLDB class that queries sqlite_master table to confirm the connected server is ProxySQL SQLite3 Server. If verification fails, exit with a helpful error message explaining that rag_ingest writes RAG index data to SQLite Server (default port 6030), not a regular MySQL server. This prevents accidental connection to wrong backend which could cause confusing errors later during RAG index operations.
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Summary of ChangesHello @rahim-kanji, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! The Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
The pull request successfully converts rag_ingest to use the MySQL protocol for connecting to the ProxySQL SQLite3 Server, replacing the direct SQLite3 API. The documentation (CHUNKING_AND_EMBEDDING_GUIDE.md, INGEST_USAGE_GUIDE.md, EMBEDDING_TEST_PLAN.md) has been thoroughly updated to reflect these changes, providing clear instructions and examples for the new connection method and configuration options. The Makefile correctly removes the direct SQLite3 dependency. The new test script (test_rag_ingest_sqlite_server.sh) provides comprehensive testing for the new architecture. However, there are potential SQL/JSON injection vulnerabilities in the test script's SQL/JSON string construction that should be addressed to prevent bad practices from propagating.
| 'posts', 'Id', '${where_sql}', | ||
| '{\"doc_id\":{\"format\":\"posts:{Id}\"},\"title\":{\"concat\":[{\"col\":\"Title\"}]},\"body\":{\"concat\":[{\"col\":\"Body\"}]},\"metadata\":{\"pick\":[\"Tags\",\"Score\"],\"rename\":{\"Tags\":\"tags\",\"Score\":\"score\"}}}', | ||
| '${chunking_json_value}', | ||
| '${embedding_json_value}' | ||
| ); | ||
| " | ||
| else | ||
| # Update existing source | ||
| run_sqlite_server " | ||
| UPDATE rag_sources | ||
| SET chunking_json='${chunking_json_value}', | ||
| embedding_json='${embedding_json_value}', | ||
| where_sql='${where_sql}', | ||
| backend_host='${MYSQL_HOST}', | ||
| backend_port=${MYSQL_PORT}, | ||
| backend_user='${MYSQL_USER}', | ||
| backend_pass='${MYSQL_PASS}' | ||
| WHERE source_id=1; |
There was a problem hiding this comment.
The apply_schema_and_source function constructs SQL queries by directly embedding shell variables like where_sql, chunking_json_override, and embedding_json_override. This practice is vulnerable to SQL injection if these variables contain unescaped malicious input. While this is a test script, it sets a poor example and could lead to security vulnerabilities if copied into production code without proper escaping. It's recommended to properly escape all variables inserted into SQL strings.
| " | ||
|
|
||
| # Insert source with OpenAI embedding config | ||
| openai_embedding_json="{\"enabled\":true,\"provider\":\"openai\",\"api_base\":\"${OPENAI_API_BASE}\",\"api_key\":\"${OPENAI_API_KEY}\",\"model\":\"${OPENAI_MODEL}\",\"dim\":${OPENAI_EMBEDDING_DIM},\"input\":{\"concat\":[{\"col\":\"Title\"},{\"lit\":\"\\n\"},{\"chunk_body\":true}]}}" |
There was a problem hiding this comment.
The openai_embedding_json string is constructed by directly embedding environment variables. If OPENAI_API_BASE, OPENAI_API_KEY, OPENAI_MODEL, or OPENAI_EMBEDDING_DIM contain characters that break out of the JSON string, it could lead to malformed JSON or unexpected behavior. Ensure these variables are properly escaped before embedding them into a JSON string.
2 participants
Convert
rag_ingestfrom direct SQLite3 API to MySQL protocol connection to ProxySQL SQLite3 Server.Key changes: