Knowledge Extraction Engine

The Problem

A senior professional with 10+ years of experience has hundreds of rules in their head — when to push, when to wait, what red flags to watch for. That knowledge is the most valuable asset in the business, and it only exists in one person's brain. If they leave, it walks out the door.

The Approach

An AI interviewer uses Critical Decision Method to walk the expert through real scenarios and pull out the underlying rules. I chose CDM because it gets specific, concrete answers instead of abstract descriptions. Each rule gets structured: what triggers it, what action to take, why, exceptions, and any hard numbers. Coding agents built the app. I wrote the specs and reviewed every output.

Key Design Decisions

1. 1.Deduplication: Jaccard similarity matching catches when an expert says the same thing two different ways across sessions. Without this, you'd have dozens of near-duplicate rules.
2. 2.Exception detection: auto-flags rules where exception data ended up in the wrong field. Caught 28 out of 279. I anticipated this failure mode because exceptions are the hardest part of knowledge extraction.
3. 3.Progressive disclosure: rules organized into 3 levels of detail. Downstream agents load only what they need for a given task. Models degrade past 50K tokens. I know this from running agents every day.
4. 4.Publishing: review, confirm, publish with version tracking. Outputs both markdown and JSON because different downstream systems need different formats.

Results

Stack