CopilotKit Unveils CLiF: Revolutionizing AI Agent Learning with Real-World User Interactions

Empowering AI Agents with Real-World Learning

Developing and deploying AI agents at scale often presents significant hurdles for engineering teams. Despite leveraging advanced models, achieving consistent execution of intricate, end-to-end tasks within specific products remains a common challenge. Traditional methods frequently involve extensive manual evaluation of hundreds or thousands of scenarios to iteratively refine agent behavior. However, the exact sequences of actions that successfully complete these tasks are performed daily by product users. Addressing this gap, CopilotKit, known for its open-source framework and the AG-UI protocol, has introduced CLiF, a platform for continuous learning from comprehensive operational trajectories. CLiF securely logs both agent activity and user interactions within a product, converting this vital data into actionable skills, production-centric evaluations, and datasets for model fine-tuning.

Bridging the Data Gap in Agent Observability

Existing agent observability tools primarily focus on tracking agent-side activities, such as prompt executions, tool calls, and model outputs. While effective at identifying where an agent deviates, they often fail to illuminate the correct operational pathways. Concurrently, product analytics platforms capture user behaviors like clicks and navigation funnels, but this data is typically structured for human analysis rather than for directly instructing or testing an AI agent. A critical disconnect often exists in combining these distinct data streams into a unified timeline for agent improvement. This disparity is significant because agents must infer how various components — prompts, skills, and APIs — integrate to achieve real-world tasks. Product users, conversely, have already mastered these processes, often repeatedly.

Full-Stack Trajectories: CLiF's Core Training Signal

At the heart of CopilotKit Intelligence's CLiF platform lies the concept of the "full-stack trajectory." This foundational data primitive is gathered using AG-UI (Agent-User Interaction Protocol), an open standard developed by CopilotKit to facilitate seamless communication between AI agents and user-facing applications. By integrating AG-UI between the agent and the user interface, the system gains visibility into both critical aspects of a task:

• Agent Trace: A detailed record of every operation, tool invocation, and state change initiated by the agent, along with their respective outcomes.
• Product Trajectory: Information regarding the user's screen display, specific buttons clicked, and triggered server requests. Any direct conversation between the user and the agent is also sequentially logged within this unified timeline.

This combined data forms a comprehensive full-stack trajectory, offering a singular stream for analysis without requiring separate data pipeline merging.

The CLiF Learning Workflow

The learning process within CLiF unfolds across a meticulously designed four-stage loop:

• 1. Capture: Trajectories flow into CopilotKit Intelligence and are categorized into "learning containers." These containers segment learning for distinct workflows, such as customer support or onboarding processes.
• 2. Analyze Patterns: Following a specified threshold of new sessions or a manual trigger, an analytical process commences. This analysis, performed on a static snapshot of the container's sessions, identifies "Insights" – recurring user requests, repeated corrections, and subtle friction points. These friction points are considered strong signals, indicative of areas where agent learning is most needed. Each Insight is fully traceable to its originating events and trajectories.
• 3. Human Review: When sufficient evidence supports a reusable lesson, CLiF proposes it as a skill, formatted as a plain SKILL.md file. An essential human-in-the-loop step requires a person to explicitly approve or reject each proposed skill, either through the Intelligence UI or the CopilotKit CLI. Approved skills are version-controlled and can be deactivated if necessary.
• 4. Deploy and Export: Published skills are delivered to the agent in real-time without requiring a system redeployment. For teams prioritizing stability, agents can be locked to a specific skill revision. Furthermore, skills and raw trajectory data are fully exportable for integration with other agent frameworks or existing observability platforms.

Expanding Beyond Skills: Evals and Fine-Tuning Data

While skills represent a primary output, the comprehensive full-stack trajectories contribute to a total of four distinct, exportable outputs, designed to integrate seamlessly into existing tech stacks:

• Agent Skills: Derived from frequently observed patterns across numerous user sessions, these enhance agent capabilities.
• Production-Based Evaluations: Generated from actual user-performed tasks, capturing realistic inputs and edge cases that are often overlooked in manually crafted evaluation suites. These are exportable to current evaluation platforms.
• Fine-Tuning Datasets: Created from authentic task trajectories, these datasets are ideal for training smaller, more efficient open models on an organization's most critical and frequent workflows.
• Raw Full-Stack Trajectories: Available for export as raw event data, enabling integration with any observability platform or custom analytical use cases.

The platform’s philosophy is to augment existing observability and evaluation tools by providing previously unavailable, high-fidelity data.

A Practical CLiF Scenario

Consider a scenario within a project management platform where agents are tasked with routine operations, such as sprint close-out. An agent, relying on standard documentation, might simply move all unfinished issues to the next sprint. However, CLiF, by securely recording product trajectories, often reveals a more nuanced reality. Observing thousands of sprint close-outs, the platform might uncover that team leads, after using the one-click option, consistently perform additional steps: they reassign certain issues to the backlog and unassign them, or they comment on blocked tasks, tagging relevant individuals. This intricate routine, unwritten and seldom articulated by users, manifests solely through their clicks and requests. Traditional agent traces miss these user-driven actions, and product analytics record clicks without their deeper meaning. CLiF identifies this routine as an "Insight," citing the specific sessions, and then proposes a comprehensive skill for the entire close-out process. Upon review and publication, the internal agent integrates this new skill without requiring a redeployment, and the same capability can be extended to external agents via an MCP server. This continuous learning mechanism also generates relevant evaluation cases and potential fine-tuning data.

Addressing Critical Agent Deployment Challenges

For technical leaders overseeing large-scale agent deployments, CLiF presents strategic solutions to several persistent challenges:

• Evaluation Bottlenecks: Manual evaluation processes are frequently the slowest aspect of agent deployment. Skills derived from actual user workflows fill critical knowledge gaps, while production-based evaluations capture real-world edge cases.
• Managing Contextual Complexity: Agent errors often lead to adding more rules or tools, resulting in bloated context windows and unpredictable behavior. CLiF centralizes learned behaviors in a skill catalog, loading skills only when contextually relevant.
• Cost Efficiency: As agents grow in complexity, their operational costs can rise. Fine-tuning datasets from real trajectories offer a pathway to migrate high-volume tasks to smaller, more cost-effective models.
• Fragmented Ecosystems: Organizations rarely standardize on a single agent framework. CLiF, built on AG-UI, demonstrates compatibility with diverse agent harnesses and underlying models, supporting a wide range of popular frameworks and industry platforms.
• Ensuring Oversight and Governance: As teams independently adopt agent tools, maintaining visibility into agent learning becomes challenging. Every CLiF skill undergoes human review, is traceable to its source data, and remains versioned and reversible. The platform also offers self-hosting for complete data control.

Self-Hosting Capabilities

Designed with enterprise control in mind, CLiF offers full self-hosting capabilities. CopilotKit Intelligence can be deployed within a company’s own Kubernetes cluster using a Helm chart, or on AWS ECS and Fargate, leveraging the organization's existing Postgres, Redis, and identity providers like Okta or Azure AD. This ensures that all trajectories, learned skills, and derived data remain securely within the company's VPC or on-premises environment. For teams prioritizing rapid setup, a managed cloud option is also available.

CLiF represents a significant advancement in the continuous learning paradigm for AI agents, promising to enhance their reliability and efficiency by directly leveraging the intelligence embedded in human-computer interactions.