Back to home
🗣️ Socratic Dialogue Bot
A terminal-based philosophical sparring partner that questions your reasoning instead of answering your questions
The Problem
LLMs are trained to be helpful — they want to answer your questions. But what if you need the opposite? While studying philosophy independently, I wanted a tool that would push back on my thinking the way a seminar partner would: surfacing hidden assumptions, catching contradictions, and forcing me to defend my positions. Not a tutor that lectures. A genuine interlocutor.
The Architecture
Each user message flows through a multi-stage pipeline where different GPT-4 calls are tuned for different jobs:
- Claim Extraction (
temp=0) — Deterministic parsing of positions and implicit assumptions from natural language into structured JSON - Contradiction Detection (
temp=0) — New positions compared against all prior positions with calibrated sensitivity (distinguishing genuine contradictions from refinements) - Periodic Summarization (
temp=0.3) — Every 6 turns, a dialectical progress summary of positions taken, tensions unresolved, and assumptions unexamined - Socratic Response (
temp=0.8) — Creative, targeted questioning informed by the full accumulated dialogue state
The core architectural insight is treating the system prompt as a live state container, not a static personality card. The prompt is rebuilt every turn with the full DialogueState — positions, assumptions, contradictions, turn count — serialized as JSON and injected into a placeholder. This is what makes the difference between a bot that asks generic philosophy questions and one that asks you about your specific reasoning.
Why the Temperature Gradient Matters
The pipeline uses three distinct temperatures because each stage has a fundamentally different job. Extraction and contradiction detection need determinism — the same input should always produce the same structured JSON. Summarization needs controlled variation — faithful to the state data but not robotic. The Socratic response needs creative range — varied questioning strategies, novel thought experiments, surprising angles. This is the same pattern used in production LLM systems: separate classification from generation and tune each independently.
Constraining Helpfulness as a Micro-Alignment Problem
Getting GPT-4 to consistently refuse to answer questions required layered behavioral constraints: an explicit rule, a self-correction cue ("If you catch yourself about to say 'I think...', stop"), a dedicated redirect rule for direct pressure, and role framing as "interlocutor" rather than "assistant." Each layer addresses a different failure mode. This is a practical instance of the alignment challenge — shaping model behavior against its trained default.
Token Management & Session Persistence
Conversations can run indefinitely thanks to automatic history compression — when the token count approaches GPT-4's 8K context window, older messages are replaced with a generated summary while the 4 most recent turns are preserved verbatim. Sessions save and restore as JSON, and export to formatted Markdown with a dialectical appendix. The DialogueState dataclass was designed for JSON serialization from day one, which made persistence essentially free when the time came to implement it.
🛠️ Tech Stack
🚀 Features
Multi-stage LLM pipeline — 4 sequential GPT-4 calls per turn, each with purpose-specific temperature and prompt design
Dynamic state injection — System prompt rebuilt every turn with full dialogue state (positions, contradictions, assumptions) as live JSON
Claim extraction & deduplication — Structured JSON parsing of philosophical positions with semantic + exact-match dedup
Contradiction detection with calibrated sensitivity — 5 calibration rules and positive/negative few-shot examples to distinguish genuine contradictions from refinements
Socratic constraint enforcement — Layered behavioral rules that prevent the model from abandoning the questioning method, even under direct pressure
Automatic history compression — Token-aware context management using tiktoken, replacing older messages with generated summaries while preserving recent turns
Session persistence & Markdown export — Save/load full dialogue state as JSON, export conversations with dialectical appendix
Graceful error handling — Three-tier strategy: exponential backoff retry, per-stage graceful degradation with safe defaults, user-facing recovery prompts
🧗♂️ Challenges
Enforcing non-helpfulness — Instruction-tuned LLMs are deeply inclined to answer questions directly. Getting GPT-4 to consistently refuse to give its own position required explicit rules, self-correction cues, and redirect handlers — each addressing a different failure mode
Calibrating contradiction sensitivity — A naive prompt flags every conceptual relationship as a tension. Tuning required 5 explicit calibration rules and carefully chosen few-shot examples on both sides of the decision boundary to separate genuine contradictions from refinements and emphasis shifts
Reliable structured output without JSON mode — Built pre-JSON-mode (fall 2023), extraction required defensive parsing: stripping markdown fences,
raw_decode()for trailing text, shape validation, and graceful fallback to empty defaults on any failurePipeline state ordering — The contradiction checker needs to compare new positions against prior positions, but extraction appends to the same list. Required snapshotting state before mutation — a subtle sequencing bug that surfaces only when extraction and contradiction detection share mutable state
🧠 Learnings
System prompts are state machines, not personality cards. Injecting evolving dialogue state into the prompt on every turn was the single biggest quality lever — it's what makes questions targeted rather than generic
Different temperatures for different pipeline stages is a small but critical insight. Deterministic parsing and creative generation are fundamentally different jobs that need different model configurations
Layered, numbered behavioral rules are far more effective than prose paragraphs for constraining LLM behavior. Each rule addresses a specific failure mode and can be tested independently
Designing data models for serialization from day one made session persistence, state injection, and history compression all come together with minimal friction — the
DialogueStatedataclass paid dividends across every phaseGraceful degradation beats strict correctness for personal tools — if extraction fails, proceed with no new claims rather than crashing the conversation. The three-tier error strategy (retry, degrade, recover) mirrors production system design