Knowledge Tree

💡 LLM Panel

2 min read

🛠️ LLM Debug Log Panel Spec

🧩 Overview

  • Scope: Internal observability panel only.
  • Objective: Transparent & debuggable AI behavior.
  • Deps: api.js (wrapper: callAPI), ui.js (panel patterns).

💡 Observability Pillars

  • 💬 Prompts & user feedback.
  • 🔗 Multi-step workflow tracing.
  • ⏱️ Latency & token usage.
  • 🚨 Eval & error rates.

⚡ Actionable Implementation Phases

Phase 1: Instrumentation

  • Wrap callAPI to compute start/end timestamps.
  • Capture context: request ID, node ID, model, settings.
  • Parse token usage & record error codes/retries.

Phase 2: UI Implementation

  • Add “Debug” button near “Export” (top-right toolbar).
  • Open a right-side drawer/modal.
  • Build tabs: Logs, Trace, Metrics, Eval.

Phase 3: Hygiene & Control

  • Implement in-memory log cap & “Clear” action.
  • Add export to JSON.
  • Add PII redaction toggle.

📋 Panel Specifications

Log Entry Fields

FieldDescription / Source
id / request_id / node_idIdentifiers
timestamp / latency_msTiming
model / settings_snapshotConfiguration
prompt_preview_redactedRedacted input
response_preview_parsed / response_preview_rawOutputs
tokens_in / tokens_outToken count
status / error_code / retriesExecution status
  • ⚡ Actions: Copy redacted prompt, Copy response, Export logs, Clear logs.
  • 🔍 Filtering: Filter logs by node_id, status, latency, model.

🔗 Trace View

  • Spans: build_prompt ➔ api_request ➔ parse_response ➔ render_tree.
  • Visual: Vertical timeline.

📊 Metrics View

  • Counters: Average latency (avg_latency_ms), total tokens in/out (total_tokens_in/total_tokens_out), error rate.
  • Usage: Break down usage per-model.

♿ UI & Accessibility Guidelines

  • Focus Lifecycle: Trap focus inside the debug drawer when open. Loop focus via Tab/Shift+Tab and restore focus to the trigger button on close. Escape key closes the drawer immediately.
  • Perf-UX: Minimize full-panel rerender scope for debug tab updates. Optimize redraw strategy under frequent updates to avoid UI lag.
  • Mobile & Motion: Prevent transform shifts on mobile debug drawer (body.debug-drawer-open .container { transform: none; }) to support reduced motion fallbacks.

🔒 Privacy & Retention

  • Defaults: Redaction enabled 🛡️, raw prompt storage disabled.
  • Retention: In-memory only (opt-in for persistence).

🔗 Sources

Graph View

Backlinks