Product philosophy

RAG Studio product philosophy is underpinned by the following principles.

Measuring quality

  • Quality Through Metrics: Objective metrics are the cornerstone of quality assessment. Metrics provide indicators for evaluating the RAG application’s quality and cost/latency performance and thereby identifying areas for improvement.

  • Comprehensive “always-on” Logging: Metrics work best if they can be computed for any invocation of the RAG app. Therefore, every invocation of the app, both in development and production, must be logged. The log must capture all inputs and outputs, as well as the detailed steps that transform inputs into outputs.

  • Human Feedback as the Benchmark: Collecting human feedback is costly, but its value as a quality measure is unmatched. RAG Studio is designed to make the collection of human feedback as efficient as possible.

  • LLM Judges Scale Feedback: Utilizing RAG LLM judges in tandem with human feedback accelerates the development loop, allowing for quicker development cycles without subsequently scaling the number of human evaluators. However, RAG LLM judges are not a substitute, but rather, an augment to human feedback.

Development lifecycle

  • Rapid Iteration: The cycle of creating and testing new versions of a RAG Application must be quick

  • Effortless Version Management: Tracking and management of versions must be seamless, reducing cognitive load and letting developers concentrate on enhancing the application rather than on administrative tasks.

  • Development and Production Are Unified: The tools, schemas, and processes used in development should be consistent with those in production environments, ensuring a consistent workflow for quality improvement in development to deployment with the same code base.

Opportunities for quality

RAG Studio is built upon the belief that quality opportunities exist across the entire RAG Application - the models, data processing pipelines, and chains. RAG Studio recognizes the interconnected nature of these components: while individual components can and should be optimized in isolation, the impact of these changes must be evaluated within the context of the entire RAG Application.

RAG application architecture all up