How production AI systems get built

Challenge

A seed-to-Series A analytics SaaS team needed a customer-facing AI assistant inside the product. Users asked the same workflow questions in support tickets, and product leadership wanted intelligent search over docs, schemas, and past reports, without standing up a dedicated AI engineering team.

Architecture

Retrieval-augmented generation over product docs, schema metadata, and approved help content; tool-calling into read-only product APIs for context; a thin orchestration layer with prompt templates, citation requirements, and a human-handoff path into support. Model choice kept vendor-neutral behind an adapter so the team could swap providers later.

Implementation

Scoped a fixed-scope proof of value: one in-app copilot surface, retrieval index pipeline, evaluation harness for grounded answers, and integration with the existing auth and tenancy model. Shipped citation UI, rate limits, and admin controls for which corpora the assistant can see.

Evaluation

Offline eval set of representative user questions with expected citations; online shadow metrics for answer usefulness and escalation rate; regression checks before each retrieval or prompt change. No launch without a defined fail-closed path when retrieval confidence is low.

Deployment

Rolled out behind a feature flag to a cohort of design partners, then expanded after support and product reviewed transcripts. Monitoring covers latency, cost per session, hallucination/escalation rates, and retrieval freshness.

Controls & governance

• Answers require citations from approved corpora or escalate to support
• Tenant isolation enforced at retrieval and tool-call layers
• Admin kill switch and corpus allowlists for each environment
• Full conversation logs retained for product and security review

Measurable outcomes

• Users get grounded answers inside the product instead of opening tickets for routine questions
• Product team can iterate on prompts and corpora without rewriting the app
• Evaluation harness makes quality regressions visible before release
• Cost and latency tracked per session so the feature stays operable at scale

Challenge

A growth-stage company running HubSpot and Zendesk had leads and support threads stalling between tools. Sales and support lacked applied AI experience, and leadership wanted agents that could qualify, summarize, and route work, not another disconnected chatbot demo.

Architecture

Event-driven agent that listens to CRM and support webhooks, maintains short-lived task state, and calls approved tools (create/update ticket, enrich lead, draft reply, escalate). Guardrails define which actions are automatic vs. require human approval. Secrets and tokens stay in the customer’s cloud; Khanex never becomes a long-term data warehouse.

Implementation

Built the agent orchestration, tool adapters for CRM and support, playbooks for lead qualification and tier-1 reply drafts, and a review queue for messages that need human approval. Integrated with existing SSO and audit logging rather than inventing a parallel admin surface.

Evaluation

Scored draft quality against approved examples; measured false-auto-send risk in a staging dry-run; tracked tool-call success rates and human override frequency. Go/no-go tied to accuracy thresholds agreed in the discovery sprint.

Deployment

Started in draft-only mode for support, then enabled auto-actions for low-risk lead enrichment. Production rollout included runbooks, on-call ownership, and weekly quality reviews with ops leads.

Controls & governance

• High-impact actions (send email, close deal) require human approval until thresholds are met
• Agents only call allowlisted tools and fields
• Every tool call is logged with actor, payload summary, and outcome
• Customer retains ownership of CRM and support data; retention policies respected

Measurable outcomes

• Qualified leads get a first touch while the opportunity is still warm
• Support receives structured summaries instead of raw ticket piles
• Teams keep working in HubSpot and Zendesk, with no new system of record
• Override and accuracy metrics show whether the agent is earning more autonomy

Challenge

An established technology-enabled company had policies, SOPs, and customer-facing runbooks spread across Google Drive, Notion, and Slack. New hires and ops staff spent hours searching; answers were inconsistent; leadership wanted internal AI that improved productivity without leaking sensitive content.

Architecture

Private knowledge assistant with permission-aware retrieval over synced corpora; ACL mirroring from source systems; separate indexes for general ops vs. restricted finance/legal content; answer format that prefers links and excerpts over free invention. Deployed in the customer’s VPC or equivalent private environment.

Implementation

Ingested and chunked approved document sets, wired identity-aware access, built Slack and web entry points, and added feedback buttons for wrong or stale answers. Included a lightweight admin workflow to promote corrected answers back into source docs.

Evaluation

Golden-question set covering onboarding, policy, and escalation paths; permission tests ensuring restricted docs never appear for unauthorized users; freshness checks when source files change. Measured time-to-answer in a pilot cohort versus baseline search.

Deployment

Pilot with one ops pod, then company-wide rollout after security review. Ongoing optimization covers corpus hygiene, prompt tuning, and expanding connectors only when access control is verified.

Controls & governance

• Retrieval respects source ACLs: no “see everything” service account for end users
• Restricted corpora isolated; default deny when permissions are ambiguous
• Citations required so employees can verify against the source document
• PII and credential patterns redacted from logs where feasible

Measurable outcomes

• Employees get consistent answers with links back to the governing SOP
• Onboarding depends less on chasing the one person who “knows how it works”
• Ops leadership can see which questions fail and which docs need updating
• Security posture matches how the company already manages document access