Use this file to discover all available pages before exploring further.
Penetration testing requires deep domain knowledge: which payloads bypass specific WAFs, how to extract data from blind SQL injection, what an LFI vulnerability requires for RCE. Rather than encoding all of this in a fixed system prompt, LuaN1aoAgent uses Retrieval-Augmented Generation (RAG) — an on-demand knowledge retrieval system that fetches the most relevant attack techniques at the moment they are needed.
The primary knowledge source is PayloadsAllTheThings, an extensive open-source collection of attack payloads, bypass techniques, and vulnerability exploitation methods.
# Clone the knowledge base into the expected directorymkdir -p knowledge_basegit clone https://github.com/swisskyrepo/PayloadsAllTheThings \ knowledge_base/PayloadsAllTheThings
The knowledge service scans the entire knowledge_base/ directory, so custom documents can be added alongside PayloadsAllTheThings.
# From rag/rag_kdprepare.pydef list_kb_files(root: str) -> List[Tuple[str, str]]: """Walks knowledge_base/ and returns all .md and .txt files.""" docs: List[Tuple[str, str]] = [] base_dir = kb_root_dir(root) for dirpath, dirnames, filenames in os.walk(base_dir): for filename in filenames: if filename.endswith(".md") or filename.endswith(".txt"): full = os.path.join(dirpath, filename) doc_id = os.path.relpath(full, root) docs.append((doc_id, full)) return docs
The knowledge base is encoded as a FAISSIndexIDMap2 wrapping an IndexFlatIP (inner product) index for cosine similarity search.
# From rag/rag_kdprepare.pyif os.path.isfile(index_fp): index = faiss.read_index(index_fp) print(f"[INFO] Loaded existing index, current vectors: {index.ntotal}")else: base_index = faiss.IndexFlatIP(384) # Inner product = cosine similarity on normalized vectors index = faiss.IndexIDMap2(base_index) print("[INFO] Created new empty index")
All vectors are L2-normalized before storage, so inner-product search is equivalent to cosine similarity:
Uses all-MiniLM-L6-v2 from the local model directory rag/models/all-MiniLM-L6-v2.
Produces 384-dimensional dense vectors with strong semantic similarity for security domain text.
# From rag/rag_kdprepare.pylocal_model_dir = os.path.join(root, "rag", "models", "all-MiniLM-L6-v2")embedder = create_embedder(local_model_dir)
A deterministic hash-based embedder used when the Sentence-Transformers model is unavailable. Maps each word token to a position in a 384-dim vector via SHA-256 and normalizes.
class OfflineHasherEmbedder: """ Offline hash embedding: maps token sha256 hashes to a fixed dimension and normalizes. Fallback when Sentence-Transformers model cannot be downloaded. """ def __init__(self, dim: int = 384): self.dim = dim def _hash_embed(self, text: str) -> List[float]: vec = [0.0] * self.dim for token in re.findall(r"\b\w+\b", text.lower()): d = hashlib.sha256(token.encode("utf-8")).digest() idx = int.from_bytes(d[0:4], "little") % self.dim sign = 1.0 if (d[4] & 1) else -1.0 vec[idx] += sign # normalize ...
Hash embedding is a coarse approximation. Semantic similarity quality is significantly lower than Sentence-Transformers. Use only as a last resort.
The knowledge service runs as a standalone FastAPI process, exposing a simple HTTP API on port 8081 (configurable via KNOWLEDGE_SERVICE_PORT).
# From rag/knowledge_service.pyapp = FastAPI( title="LuaN1ao Knowledge Service", version="3.0", lifespan=lifespan)
On startup, the service initializes the RAG client and builds the index:
async def _initialize_knowledge_base(): """Initialize the unified RAG client (async-safe).""" async with _rag_client_lock: if _rag_client is None: loop = asyncio.get_running_loop() # Run sync initialization in executor to avoid blocking the event loop await loop.run_in_executor(None, _initialize_knowledge_base_sync)
During execution, the Executor can call the retrieve_knowledge MCP tool to inject relevant domain knowledge into its reasoning context before crafting an attack payload:
The returned chunks are injected into the Executor’s message history as a user observation, giving the agent specific, document-grounded payload examples to work from.Similarly, the distill_knowledge tool allows the agent to write new insights back into a custom knowledge document — enabling cross-task knowledge accumulation:
The preparation script supports incremental updates — only new or modified documents are re-vectorized:
# Build or update the FAISS indexcd ragpython -m rag_kdprepare
The manifest file (rag/faiss_db/faiss_manifest.json) tracks the SHA-256 hash and mtime of every indexed document. On subsequent runs:
Documents with unchanged hashes are skipped.
Modified documents have their old chunks removed from the index and replaced with new chunks.
Deleted documents have all their chunks removed.
# From rag/rag_kdprepare.py — incremental logicif not doc_manifest or doc_manifest.get("hash") != digest: # New or updated — re-vectorize to_upsert_docs.append((doc_id, content, meta))# else: hash matches, skip
Force-rebuild options are available for development:
# Force rebuild all documentspython -m rag_kdprepare --force-all# Force rebuild documents matching a patternpython -m rag_kdprepare --force-doc=SQLInjection# Via environment variableRAG_FORCE_ALL=true python -m rag_kdprepare
Chunk size significantly affects retrieval quality. Smaller chunks (100–300 chars) produce more precise results for specific payload lookups. Larger chunks (500–1000 chars) preserve more surrounding context and work better for technique explanations. Tune with RAG_MIN_CHUNK_SIZE and RAG_MAX_CHUNK_SIZE in your .env.