🧠 How RAG (Retrieval Augmented Generation) Works
📚 What is RAG?
RAG is an AI technique that combines the power of large language models (like GPT-4o) with your own knowledge base. Instead of relying solely on pre-trained knowledge, RAG allows the AI to access and reference your specific documents and content to provide accurate, contextual answers.
🔄 Step-by-Step RAG Process with API Examples
📥 Step 1: Content Ingestion & Processing
What Happens: Your files are received, validated, and processed into clean text.
📋 Sample API Call:
POST /ai-playground/rag/api/endpoints.php
Content-Type: multipart/form-data
action=process_input
openai_api_key=sk-...
input_type=text_file
text_file[]=@document1.txt
text_file[]=@document2.txt
📤 Sample Response:
{
"success": true,
"message": "Input processed successfully",
"input_type": "text_file",
"input_type_friendly": "Text File(s)",
"text_length": 15420,
"chunks_count": 387,
"storage_file": "rag_chunks_2025-08-11_15-30-45.json"
}
✂️ Step 2: Intelligent Chunking
What Happens: Text is broken into overlapping chunks optimized for AI processing.
📋 Sample Chunk Structure:
{
"id": 0,
"text": "This is the first chunk of text that will be processed...",
"start_pos": 0,
"end_pos": 8000,
"length": 8000
}
🔍 Step 3: Embedding Generation
What Happens: Each text chunk is converted to a numerical vector using OpenAI's embedding model.
📋 Sample OpenAI Embedding API Call:
POST https://api.openai.com/v1/embeddings
Authorization: Bearer sk-...
Content-Type: application/json
{
"model": "text-embedding-3-small",
"input": "This is the first chunk of text that will be processed...",
"encoding_format": "float"
}
💾 Step 4: Data Storage
What Happens: All processed data is stored in a structured JSON format for retrieval.
📋 Sample Storage File Structure:
{
"timestamp": "2025-08-11 15:30:45",
"input_type": "text_file",
"chunks_count": 387,
"chunks": [
{
"id": 0,
"text": "This is the first chunk of text...",
"start_pos": 0,
"end_pos": 8000,
"length": 8000,
"embedding": [0.0123, -0.0456, 0.0789, ...]
}
]
}
🔍 Step 5: Query Processing & Retrieval
What Happens: Your question is processed and the most relevant content chunks are retrieved.
📋 Sample Query API Call:
POST /ai-playground/rag/api/endpoints.php
Content-Type: application/x-www-form-urlencoded
action=query
openai_api_key=sk-...
query=What are the main findings in the research?
storage_file=rag_chunks_2025-08-11_15-30-45.json
🤖 Step 6: AI Response Generation
What Happens: GPT-4o generates a comprehensive answer based on your question and the relevant content.
📋 Sample OpenAI Chat API Call:
POST https://api.openai.com/v1/chat/completions
Authorization: Bearer sk-...
Content-Type: application/json
{
"model": "gpt-4o",
"messages": [
{
"role": "system",
"content": "You are a helpful assistant that answers questions based on provided context..."
},
{
"role": "user",
"content": "Context from processed content:\n\n[Relevant chunks]...\n\nQuestion: What are the main findings?"
}
],
"temperature": 0.7,
"max_tokens": 1500
}
🔌 Complete API Reference
📤 Process Input API
POST /ai-playground/rag/api/endpoints.php
Content-Type: multipart/form-data
Required Parameters:
- action=process_input
- openai_api_key=your_api_key
- input_type=url|pdf_file|text_file|direct_input
Optional Parameters (based on input_type):
- url=webpage_url (for input_type=url)
- pdf_file[]=@file1.pdf,@file2.pdf (for input_type=pdf_file)
- text_file[]=@file1.txt,@file2.txt (for input_type=text_file)
- direct_text=your_text_content (for input_type=direct_input)
🔍 Query API
POST /ai-playground/rag/api/endpoints.php
Content-Type: application/x-www-form-urlencoded
Required Parameters:
- action=query
- openai_api_key=your_api_key
- query=your_question_here
Optional Parameters:
- storage_file=filename.json (auto-filled after processing)
📊 Progress API
POST /ai-playground/rag/api/endpoints.php
Content-Type: application/x-www-form-urlencoded
Required Parameters:
- action=get_progress
Response includes:
- percentage: 0-100
- status: current status message
- substatus: detailed progress information
- timestamp: when progress was last updated
💡 Integration Examples
🔄 JavaScript/Fetch Example
// Process a text file
const formData = new FormData();
formData.append('action', 'process_input');
formData.append('openai_api_key', 'sk-...');
formData.append('input_type', 'text_file');
formData.append('text_file[]', fileInput.files[0]);
const response = await fetch('/ai-playground/rag/api/endpoints.php', {
method: 'POST',
body: formData
});
const result = await response.json();
console.log('Processed', result.chunks_count, 'chunks');
🐍 Python Example
import requests
# Process input
files = {'text_file[]': open('document.txt', 'rb')}
data = {
'action': 'process_input',
'openai_api_key': 'sk-...',
'input_type': 'text_file'
}
response = requests.post('/ai-playground/rag/api/endpoints.php',
data=data, files=files)
result = response.json()
# Query the processed content
query_data = {
'action': 'query',
'openai_api_key': 'sk-...',
'query': 'What is this document about?',
'storage_file': result['storage_file']
}
query_response = requests.post('/ai-playground/rag/api/endpoints.php',
data=query_data)
answer = query_response.json()
print(answer['response'])