You’ve used AI. You’ve probably built on top of it. But do you actually know what’s happening under the hood?
Every developer I know has a moment where they go from “this is magic” to “wait, I need to understand what’s actually happening here.”
Modern AI systems are distributed, layered, and engineered like full-scale products, not just models.
In this article, I’ll break down the actual AI stack used in production systems, the same concepts behind ChatGPT, Copilot, Perplexity, and enterprise AI tools at a system design level.
Let’s go layer by layer.
🧠 Layer 1 Embeddings: How AI Understands Meaning
Before AI can answer anything… it needs to “understand” your input.
But here’s the truth:
👉 AI doesn’t understand words.
👉 It understands numbers in high-dimensional space.
What are Embeddings?
An embedding is a vector, a list of floating-point numbers, that represents the meaning of a piece of text (or images, code, etc.)
For example:
| Text | Embedding (simplified) |
| ------- | ------------------------ |
| "dog" | [0.21, -0.45, 0.88, ...] |
| "puppy" | [0.19, -0.47, 0.85, ...] |
| "car" | [0.90, 0.12, -0.33, ...] |👉 Notice something?
“dog” and “puppy” are closer in vector space
“dog” and “car” are far apart
This isn’t keyword matching. It’s the AI having an actual semantic understanding of language.
How it works technically:
Text goes into a neural network (a transformer-based encoder model like text-embedding-ada-002, sentence-transformers, or nomic-embed). The model outputs a fixed-size vector — typically 768 to 3072 dimensions.
from openai import OpenAI
client = OpenAI()
def get_embedding(text: str) -> list[float]:
response = client.embeddings.create(
input=text,
model="text-embedding-3-small"
)
return response.data[0].embedding # A list of 1536 floatsThat list of 1536 floats? That’s the meaning of your text. Compressed into math.
Why This Matters
This is how AI understands:
Similarity
Context
Intent
🔥 AI doesn’t read meaning, it measures distance.
Real-World Use Cases
Semantic search
Recommendation systems
Chat memory retrieval
Code similarity detection
🗄️ Layer 2 Vector Databases : How AI Remembers
Now that we can convert meaning into vectors… where do we store them?
We can’t stuff an entire knowledge base into an AI’s context window. It’s expensive, slow, and hits hard limits fast.
So how do you give an AI a “memory” of millions of documents?
Enter: Vector Databases
A vector DB stores embeddings and allows fast similarity search. Its purpose is to build one killer query: “Given this embedding, find me the N most similar embeddings stored in this database.” This is called Approximate Nearest Neighbour (ANN) search, and it’s how AI systems retrieve relevant memories in milliseconds, even across billions of records.
Instead of querying like:
SELECT * FROM docs WHERE title = "AI"You query like:
👉 “Find things similar to this meaning”
How It Works
Convert data → embeddings
Store embeddings in vector DB
Query with a new embedding
Retrieve closest matches
import chromadb
client = chromadb.Client()
collection = client.create_collection("company-docs")
# Indexing
collection.add(
documents=["Q3 revenue was $4.2M", "The CEO joined in 2019"],
ids=["doc1", "doc2"]
)
# Querying
results = collection.query(
query_texts=["What was our revenue last quarter?"],
n_results=2
)
print(results["documents"]) # Returns the most relevant docsPopular vector databases to know:
| DB | Best For |
|:--------:|:----------------------------:|
| Pinecone | Managed, production-scale |
| Weaviate | Open-source + graph features |
| Chroma | Local dev, Python-first |
| pgvector | You're already on PostgreSQL |
| Qdrant | High performance, Rust-based |Why It’s Critical
Traditional databases answer:
“What matches exactly?”
Vector DBs answer:
🔥 “What feels similar?”
That’s the foundation of modern AI systems.
🤖 Layer 3 Agents: How AI Decides and Acts
Up until now, the AI has been answering questions. It’s reactive.
Agents flip that.
An agent is an AI system that can:
Plan
Decide
Take actions
Use tools
Iterate
Think of it like this:
A normal LLM is:
💬 “Ask → Answer”
An agent is:
🧠 “Think → Plan → Act → Observe → Repeat”
The agent loop looks like this:
User Goal [Book me the cheapest flight to Bangalore tomorrow]
↓
[Think -> Understand intent]
↓
[Choose Action -> Search flight APIs]
↓
[Execute Tool -> Compare prices]
↓
[Observe Result -> Choose best option]
↓
Final Answer👉 That’s not just generation , that’s decision-making
This is often called ReAct (Reasoning + Acting) —> the model reasons about what to do, acts by calling a tool, observes the result, and repeats.
A simple agent in Python:
from anthropic import Anthropic
client = Anthropic()
tools = [
{
"name": "search_web",
"description": "Search the web for current information",
"input_schema": {
"type": "object",
"properties": {
"query": {"type": "string", "description": "Search query"}
},
"required": ["query"]
}
}
]
def run_agent(user_message: str):
messages = [{"role": "user", "content": user_message}]
while True:
response = client.messages.create(
model="claude-opus-4-5",
max_tokens=1024,
tools=tools,
messages=messages
)
if response.stop_reason == "end_turn":
return response.content[0].text # Done
if response.stop_reason == "tool_use":
tool_call = response.content[1]
result = execute_tool(tool_call.name, tool_call.input)
messages.append({"role": "assistant", "content": response.content})
messages.append({"role": "user", "content": [{
"type": "tool_result",
"tool_use_id": tool_call.id,
"content": result
}]})Core Components of Agents
LLM (brain)
Tools (APIs, DBs)
Memory
Planning logic
🔥 Agents turn AI from passive responder → active problem solver
Types of agent architectures:
Single agent: One model, one loop, multiple tools
Multi-agent: Multiple specialized models collaborating (planner + executor + critic)
Hierarchical agents: A manager agent spawning subagents for parallel tasks
💡 Enjoying this article?
Every week day, I publish practical, production-ready deep dives covering Web development, System Design, Open source projects, Tech industry trends and AI Engineering and tools.
📚 Layer 4 RAG: How AI Stays Accurate and Up-to-Date
LLMs are powerful… but they hallucinate. Because they rely on training data, not live knowledge.
And if you ask them something outside their training data, they’ll do the one thing you don’t want: confidently make something up.
Enter: Retrieval-Augmented Generation (RAG)
RAG = Search + Generate
RAG is the architectural pattern that combines:
A retrieval system (vector DB) to find relevant, current information
A generation model (LLM) to synthesize that information into a response
How It Works
User Question
↓
Embed the question
↓
Query vector DB → Top-K relevant chunks
↓
Inject chunks into LLM prompt as context
↓
LLM generates answer grounded in retrieved contextIn code:
def rag_answer(user_question: str) -> str:
# Step 1: Retrieve
query_embedding = get_embedding(user_question)
relevant_docs = vector_db.search(query_embedding, top_k=5)
# Step 2: Build context
context = "\n\n".join([doc.text for doc in relevant_docs])
# Step 3: Generate with grounding
prompt = f"""Answer the question based ONLY on the following context.
Context:
{context}
Question: {user_question}
If the answer isn't in the context, say you don't know."""
return llm.generate(prompt)Why RAG beats fine-tuning for most use cases:
| | Fine-tuning | RAG |
|:------------------:|:----------------------:|:--------------------:|
| Update knowledge | Retrain the model | Update the vector DB |
| Cost | High | Low |
| Hallucination risk | Still present | Lower (grounded) |
| Source citations | Hard | Easy |
| Best for | Behavior/style changes | Knowledge injection |🔥 RAG turns AI from “guessing machine” into “knowledge system”
🔌 Layer 5 MCP: How AI Connects to Real-World Tools
AI is powerful… but useless if it can’t interact with the real world.
You now have an AI that understands language, remembers information, takes actions, and stays accurate. But there’s still a gap.
Every agent needs tools. And historically, every team built their own custom integrations. Slack integration here, GitHub integration there. It was a mess of one-off connectors.
MCP the Model Context Protocol changes this.
MCP is an open protocol (developed by Anthropic) that standardizes how AI models connect to external data sources and tools. Think of it like USB-C but for AI integrations. One standard connector, any device.
Instead of building custom integrations for everything, MCP provides:
Standard interfaces
Tool discovery
Secure communication
What MCP Enables
File system access
Database queries
API integrations
Dev tools (VS Code, Git, etc.)
The MCP architecture has three players:
┌─────────────────┐ MCP Protocol ┌─────────────────┐
│ AI Model │ ◄─────────────────────► │ MCP Server │
│ (MCP Client) │ │ (Your Tool) │
└─────────────────┘ └─────────────────┘
│
┌────────┴────────┐
│ Real Resources │
│ Files, DBs, │
│ APIs, Git... │
└─────────────────┘MCP Host: The AI application (Claude, VS Code, your custom agent)
MCP Client: Handles the protocol communication
MCP Server: Exposes tools, resources, and prompts over the standard protocol
What an MCP server exposes:
Tools: Functions the AI can call (e.g.,
read_file,create_issue,query_database)Resources: Data sources the AI can read (e.g., local files, DB records)
Prompts: Reusable prompt templates for common workflows
Building a simple MCP server:
from mcp.server import Server
from mcp.types import Tool, TextContent
server = Server("my-tool-server")
@server.list_tools()
async def list_tools():
return [
Tool(
name="get_weather",
description="Get current weather for a city",
inputSchema={
"type": "object",
"properties": {
"city": {"type": "string"}
}
}
)
]
@server.call_tool()
async def call_tool(name: str, arguments: dict):
if name == "get_weather":
city = arguments["city"]
weather_data = fetch_weather_api(city)
return [TextContent(type="text", text=str(weather_data))]🧩Putting It All Together: The Full AI Stack
Here’s the complete system, end-to-end:
User Input
│
▼
[Embeddings] ← Convert query to semantic vector
│
▼
[Vector DB] ← Find relevant documents/memories
│
▼
[RAG Pipeline] ← Inject retrieved context into prompt
│
▼
[LLM + Agent] ← Reason, decide, plan next action
│
▼
[MCP Tools] ← Execute real-world actions
│
▼
Response to UserEach layer solves a specific problem:
Embeddings → The AI can understand what you mean
Vector DBs → The AI can remember at scale
RAG → The AI answers from real, current data
Agents → The AI can act, not just respond
MCP → The AI can connect to anything
The magic isn’t in any single piece. It’s in how they compose.
What This Means for You as a Developer
You don’t need to implement all five layers for every project. But you need to know when to reach for each one.
Building a Q&A bot over company docs? Start with RAG. Just embeddings + vector DB + LLM.
Building a coding assistant that reads files and runs tests? That’s an agent + MCP tools for filesystem and shell access.
Building a customer support system with product knowledge and CRM access? That’s the full stack.
The mental model to carry: Modern AI is a composable system. Each layer adds a new capability. Your job as an engineer is to know which layers your product needs and to wire them together well.
The AI isn’t magic. It’s just a very well-designed system. And now you know how it works.
Thank You for Reading!
I hope you found it helpful and informative. If you have any questions or feedback, feel free to leave a comment below. Your support and engagement mean a lot to me.