Context Engineering for LLMs with Redis

Learn to build production-ready AI agents with optimized context management, memory systems, and intelligent retrieval using Redis, LangGraph, and LangChain.

---

What You'll Learn

This hands-on course teaches practical context engineering patterns through building a Course Advisor Agent:

• RAG Fundamentals — Build retrieval-augmented generation systems with Redis vector search
• Context Engineering — Optimize token efficiency using progressive disclosure and hierarchical context
• LangGraph Agents — Create observable, stateful workflows with tool calling
• Hybrid Search — Combine semantic search with exact-match retrieval using NER
• Memory Systems — Implement working memory and long-term memory for personalization
• ReAct Pattern — Build agents with explicit reasoning (Thought → Action → Observation)

---

Prerequisites

Requirement	Details
Python	3.11 or higher
Redis Stack	Local Docker or Redis Cloud
OpenAI API Key	For GPT-4o access
Agent Memory Server	For memory stages (5+)
Docker	For running Redis and Agent Memory Server

---

Installation

1. Clone the Repository

git clone https://github.com/redis-developer/context-eng-matters.git
cd context-eng-matters

2. Create Virtual Environment

# Using UV (recommended)
curl -LsSf https://astral.sh/uv/install.sh | sh
uv sync

# Or using pip
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate
pip install -e .

3. Set Environment Variables

# Create .env file from example
cp .env.example .env

# Edit .env and add your OpenAI API key

Environment Variables:

Variable	Required	Default	Description
OPENAI_API_KEY	✅ Yes	-	Your OpenAI API key for embeddings and LLM
REDIS_URL	No	redis://localhost:6379	Redis connection URL
AGENT_MEMORY_SERVER_URL	No	http://localhost:8088	Agent Memory Server URL
REDIS_INDEX_NAME	No	course_catalog	Redis index name for course data
OPENAI_MODEL	No	gpt-4o	OpenAI model to use

4. Start Services

# Start Redis and Agent Memory Server
docker-compose up -d

# Verify services are running
docker ps

5. Load Course Data into Redis

The notebooks and progressive agents require course data in Redis. Load the hierarchical course data:

# Load hierarchical courses into Redis (recommended)
uv run python -m redis_context_course.scripts.load_hierarchical_courses \
  -i src/redis_context_course/data/hierarchical/hierarchical_courses.json \
  --force

Options:

• --force / -f: Clear existing data before loading (use when reloading after data changes)
• --summary-index / -s: Custom index name (default: course_summaries)
• --details-prefix / -d: Custom details prefix (default: course_details)

Alternative: For backward compatibility with flat course format:

uv run python -m redis_context_course.scripts.ingest_courses \
  --catalog src/redis_context_course/data/courses.json \
  --index-name hierarchical_courses \
  --clear

Note: If you regenerate the course catalog, always use --force to reload Redis data.

6. Verify Installation

# Run tests
uv run pytest tests/ -v

# Or with pip
pytest tests/ -v

---

Quick Start

Try the ReAct agent with visible reasoning in under 2 minutes:

cd progressive_agents/stage4_hybrid_search

# Ask about a course with visible reasoning
python cli.py --show-reasoning "What are the prerequisites for CS002?"

Example output:

🧠 Reasoning Trace:
================================================================================
💭 Thought: The user is asking about prerequisites. I'll use exact match.

🔧 Action: search_courses
   Input: {"query": "CS002", "intent": "PREREQUISITES", "search_strategy": "exact_match"}
👁️  Observation: Found CS002 - Data Structures and Algorithms...

💭 Thought: I found the course. Prerequisites: CS001 (Introduction to Programming).

✅ FINISH
================================================================================

📝 Answer:
CS002 (Data Structures and Algorithms) requires CS001 (Introduction to Programming) as a prerequisite.

---

Project Structure

context-eng-matters/
├── src/redis_context_course/     # Core library
│   ├── course_manager.py         # CourseManager - basic Redis vector search
│   ├── hierarchical_course_manager.py  # HierarchicalCourseManager - two-tier retrieval
│   ├── hierarchical_context.py   # HierarchicalContextAssembler - progressive disclosure
│   ├── models.py                 # Pydantic data models
│   └── scripts/                  # Data generation utilities
│
├── workshop/                     # Comprehensive workshop (6 notebooks, ~6,000 lines)
│   ├── 01_introduction_to_context_engineering.ipynb  # Context types, token budgeting
│   ├── 02_rag_essentials.ipynb                       # Vector embeddings, semantic search (~1,000 lines)
│   ├── 03_data_engineering.ipynb                     # Data pipelines, chunking strategies (~1,700 lines)
│   ├── 04_memory_systems.ipynb                       # Working + long-term memory (~2,000 lines)
│   ├── 05_building_agents.ipynb                      # LangGraph, tool calling
│   └── 06_capstone_comparison.ipynb                  # Stage 4 vs 6 comparison
│
├── progressive_agents/           # 6 agent implementations (learning path)
│   ├── stage1_baseline_rag/
│   ├── stage2_context_engineered/
│   ├── stage3_full_agent_without_memory/
│   ├── stage4_hybrid_search/            # Hybrid search + ReAct
│   ├── stage5_working_memory/           # Session-based memory + ReAct
│   └── stage6_full_memory/              # Working + Long-term memory + ReAct
│
├── notebooks/                    # Full course (11 Jupyter notebooks)
│   ├── section-1-context-engineering-foundations/
│   ├── section-2-retrieved-context-engineering/
│   ├── section-3-memory-systems/
│   └── section-4-tools-and-agents/
│
├── tests/                        # Test suite
├── examples/                     # Usage examples
└── docker-compose.yml            # Redis + Agent Memory Server

---

Progressive Agents

The progressive_agents/ directory contains a learning path from basic RAG to production-ready agents:

graph LR
    S1[Stage 1<br/>Baseline RAG] --> S2[Stage 2<br/>Context Engineering]
    S2 --> S3[Stage 3<br/>Full Agent]
    S3 --> S4[Stage 4<br/>Hybrid Search]
    S4 --> S5[Stage 5<br/>Working Memory]
    S5 --> S6[Stage 6<br/>Full Memory]

Loading

Stage	Key Feature	What's New
1	Baseline RAG	Raw JSON context, ~5000 tokens
2	Context Engineering	Progressive disclosure, ~1000 tokens
3	Full Agent	LangGraph, intent classification, quality eval
4	Hybrid Search	NER + FilterQuery, visible reasoning (ReAct)
5	Working Memory	Session-based conversation history (1 tool)
6	Full Memory	Working + long-term memory (3 tools)

👉 See full documentation →

---

Notebooks

Section	Topics	Duration
1. Context Engineering Foundations	What is context engineering, assembly strategies	2-3 hrs
2. Retrieved Context Engineering	RAG fundamentals, crafting & optimizing context	2.5-3 hrs
3. Memory Systems	Working/long-term memory, compression strategies	4-5 hrs
4. Tools and Agents	LangGraph, tool calling, semantic tool selection	3.5-4.5 hrs

Start learning:

uv run jupyter notebook notebooks/

Open: section-1-context-engineering-foundations/01_what_is_context_engineering.ipynb

---

Key Technologies

Technology	Purpose
Redis Stack	Vector storage, semantic search, caching
RedisVL	Vector search library with FilterQuery
LangGraph	Stateful agent workflows
LangChain	LLM application framework
Agent Memory Server	Working and long-term memory management
OpenAI GPT-4o	Language model for reasoning

---

Documentation

Document	Description
SETUP.md	Detailed setup and troubleshooting
workshop/README.md	Intensive workshop guide
progressive_agents/README.md	Agent stages documentation
notebooks/README.md	Full course notebook guides

---

Contributing

Contributions are welcome! Please read CONTRIBUTING.md for guidelines.

---

License

MIT License — See LICENSE for details.