Agent Platform
Designing the testing and debugging experience for multi-agent AI apps
Overview
Kore.ai's Agent Platform is an enterprise platform for building, orchestrating, and deploying multi-agent AI systems. Organizations use it to build specialized AI agents, each with their own tools, knowledge sources, and decision-making logic, into production-ready agentic applications.
The Playground is where the users can interact with their agentic app in real time, validate end-to-end workflows, and debug the entire execution chain, from orchestrator routing to individual tool calls. I designed this experience from the ground up as a core module of the platform.
Background
Kore.ai already had the XO Platform, an automation and conversational AI platform serving hundreds of global organizations. Agent Platform was built from the ground up as a new offering for the agentic era: AI apps that reason, delegate, and act autonomously instead of following traditional programmatic workflows.
I was one of the designers building the platform 0 → 1 and owned several of its core modules. This case study covers the Playground, the platform's testing and debugging experience.
From the earliest stages of design, I was designing for two audiences at once:
- The end users who would build or manage agentic apps on the platform and needed a surface to test them before deploying
- Our own internal teams building the product, who needed to test and debug each new capability as it landed on the platform
Problem
Agentic apps are layered systems. A single user message can set off a chain of actions: the orchestrator agent interprets intent and delegates to a specialized agent, which reasons through multiple steps: invoking tools, querying knowledge bases, and drawing on memory, with guardrails checking what comes in and what goes out. All of this happens behind the scenes in seconds, across multiple layers of logic.
To trust an app like this, you need to see the chain at work:
- Is the request being routed to the right agent?
- Is context surviving each handoff?
- Are tool calls firing with the right parameters?
- When something breaks, where exactly did it break, and did the system recover?
None of this is visible in a chat window. The response is just the last link of a chain that is otherwise a black box.
Design
The initial build was needed yesterday. We needed something solid first that made testing possible for the internal teams. From there, the design was moulded release by release, as new capabilities landed on the platform, as feedback came in from internal use, and as we observed how people actually used it.
Phase I: Making the chain visible
We went into the initial discussions with one conviction already formed from the learnings from the XO Platform: this needed to be a split screen. Conversing with the app and inspecting what it sets in motion are simultaneous activities. So the layout became two halves: the chat on the left, behaving exactly as the deployed app would, and a debug panel on the right, on demand and otherwise out of the way.
The second early call was a visualization of the complete system. Agentic systems were new, to our target users and to us as well. Based on the customer applications from XO Platform, I was able to estimate the scale and complexity of the systems organizations could build. An agentic system that complex is hard to keep track of, and a log of events alone doesn't teach you its shape. So alongside the timeline, we designed a flow view: a node graph of the app's architecture showing how agents connect, which tools they own, and how a request flows between them.
The timeline carried the hardest information problem in the module. Every trace holds payloads, token counts, model info, and timings, and surfacing all of it at once would read like server logs. I designed it around a single principle: scannable first, deep on demand.
Over the next few weeks, through many rounds of ideation and reviews with PMs and the engineering team, the design started getting some shape.
The timeline opens fully collapsed, every trace a collection of nodes, each with a preview of what happened. Clicking a node reveals its children, and each level opens the same way, down to the raw request and response data. Nothing is hidden, and nothing is forced on you.
With the core problem solved, we shipped the first version. The chain was finally visible: a timeline of what happened, and a map of what exists.
Phase II: Learning from early trials
Once the first version was shipped, we ran trials with internal teams and with customers from our pilot programs, observing how they tested and what got in their way. What we learned reshaped the Playground in a few distinct ways.
Making sessions durable
While watching a product manager test a demo they had built, I noticed they kept a document with a scripted set of messages on the side. Whenever they stepped out of the Playground to tweak a configuration, the session was gone on their return, and the only way back to where they were was replaying the script, message by message. We realized we had no way of continuing a session.
That observation sparked the conversation that led to session history: a panel that organizes past sessions and lets you continue any session exactly where it left off.
Custom sessions for scenarios you can't reproduce
Some scenarios couldn't be fully emulated in the Playground: behavior that only shows up for a specific user profile or a specific context. That cut into the core purpose of the module, because a scenario you can't reproduce is a scenario you can't test, and it led to the custom session feature. Users can now configure a session before starting it, setting metadata as session context and adjusting session-level controls like artifacts, streaming, thoughts, and document upload, to recreate the exact state where an issue shows up.
Retaining focus while the session runs
Watching people test revealed another quiet problem. Traces grow long, and while a request was processing, users would lose track of the active node, scrolling through a fully expanded timeline to find the right thing to look at. To reduce this friction, I proposed changing the timeline's default behavior: nodes stay collapsed, the currently running node opens automatically, and when it finishes processing, the expanded state moves to the next node as it begins. The user's focus follows the execution without them explicitly managing it.
Phase III: Keeping pace with an evolving platform
Alongside these refinements, the platform itself kept gaining capabilities, and every capability it gained was an unwritten design brief for this module. Part of my job became watching what was landing on the product and working out its testing implications before it shipped. My scope was a moving target: whatever the platform could do, you should be able to test. Here's a series of feature releases that further shaped the Playground's design.
Rethinking the entry point
The platform added configurable welcome messages, where the app greets the user the moment a session starts. Soon after, discrepancies started showing up in analytics. Opening the Playground immediately started a new session, so users who clicked into it by accident were starting sessions, triggering welcome messages, and logging sessions that never should have existed.
We decided to solve this with a landing page, and it gave us the chance to fix a quieter assumption at the same time. With session history and custom sessions in place, the Playground now had three possible ways to begin, yet we were assuming every visit wanted a fresh one. The landing page starts nothing on its own: it presents the three ways in, start a new session, start a custom session, or continue a previous one, and waits for the user to choose.
Document processing
When document processing landed on the platform, it meant more than adding an attach button to the chat. Testing with documents meant knowing which documents were in context at any point, and what each trace had access to. I designed an upload flow along with a documents-in-context bar above the input, showing what the app is currently working from and letting users manage it mid-conversation. Users could verify not just that the app answered, but that it answered from the right material.
Error handling and model fallbacks
The platform introduced error handling: if a model failed mid-execution, from rate limits or provider outage, a fallback model could take its place and the flow would proceed to a response. Functionally this was a success, but it opened a gap in observability. The trace completed, so nothing looked wrong, yet a model had failed inside it, and anyone debugging performance or cost needed to know.
I updated the timeline to show errors inside successful traces. The failed run and its fallback appear as connected sequential steps, each with its own status, model, and metrics, so you can see at a glance which model failed, which one was used as a fallback, and what the detour cost in latency and tokens. Failure became part of a trace's story rather than a verdict on it.
Voice dictation
Voice dictation was different from the other releases: it had no implications for the chain. It was simply another way of capturing the user's query, so the change stayed scoped to the chat side of the experience. The message composer gained a speak-to-dictate control with a live recording state, and voice-driven interactions could be tested the same way typed ones were.
A new orchestration pattern
The platform had started with the Supervisor orchestration pattern, where requests route from the supervisor out to agents and tools, and results return to the supervisor to compose the final response. The timeline's nesting mirrored that shape exactly.
Then Delegation was introduced as a second orchestration pattern to build with: agents hand requests directly to each other without a supervisor in the middle. For apps built on the Delegation pattern, the nested timeline design made for Supervisor pattern didn't tell the right story.
I went back to basics and asked what a trace should look like for this new pattern. The answer was handoff nodes: explicit markers of control passing from one agent to another, with the delegated work nesting under the agent that received it. The flow view was tailored the same way, drawing delegation paths between agents instead of routing every line through the center. The same timeline now adapts to whichever pattern an app is built on, and tells the right story for each.
Outcome
The Playground shipped as the primary testing and debugging surface in Agent Platform. It sits in the daily loop of everyone building on the platform: build, test in the Playground, adjust, repeat.
It served its second audience too. All through development, the Playground was how our own teams tested Agent Platform itself, with every new capability passing through it before reaching customers. The flow view also found a use beyond debugging: it became a fixture in demos and onboarding, because it visualizes in one glance how a multi-agent app is put together.
Reflection
Design the system, not the screens
Designing against a moving platform forced me to design a system, not screens. The timeline survived new orchestration patterns, fallback chains, and steadily richer traces because its underlying grammar of nested actors, statuses, and progressive depth could absorb new node types without being redesigned.
A module that kept changing under me also taught me to design with room to spare: for node types that didn't exist yet, controls that would arrive later, patterns the roadmap hadn't named. Treating the design as something in progress rather than a finished state is what kept it alive.
Watch people work, don't just ask
The biggest improvements didn't come from feature requests. They came from observation. From noticing the workarounds people built and the friction they had stopped registering. Watching how people actually test surfaced problems that asking them never would have.