Enhancing LLM quality and interpretability with the Vertex AI Gen AI Evaluation Service

Harnessing the power of LLMs presents two challenges: managing their inherent randomness and addressing occasional factual inaccuracies. To address these hurdles, a new workflow has been developed that utilizes the Vertex Gen AI Evaluation Service to automate the selection of the best response from a diverse set of LLM-generated options. This workflow involves generating multiple responses, pairwise evaluating them to identify the best response, and assessing its quality using pointwise evaluation. The financial institution's use case of summarizing customer conversations exemplifies the application of this workflow to real-world tasks. The workflow enhances the accuracy, helpfulness, and conciseness of LLM-generated summaries, fostering trust and transparency in the system's decision-making. The workflow is applicable to any modality or use case, including question answering and summarization. By leveraging the probabilistic nature of LLMs and the Vertex Gen AI Evaluation Service, this workflow enables the full potential of LLMs to be unlocked.