How to Use Structured Generation for LLM-as-a-Judge Evaluations

Structured generation is a machine learning subfield that guides the outputs of generative models to fit specific schemas. It is used to ensure that the generated text follows a predefined structure, such as valid JSON. This technique is essential for building complex, multi-step reasoning agents in LLM evaluations, especially for open source models. The process involves defining a schema and parsing the output to ensure it meets the requirements. For instance, a simple JSON grammar can be defined using the Lark library, which allows for specifying valid and invalid JSON strings. To guide the model's output, a function can be created to recursively sample from the model, using a validation function to check if the generated text is valid or incomplete. This approach can add computational overhead but optimized implementations can minimize latency impact. Structured generation can be applied to LLM-as-a-judge metrics like hallucination detection, where traditional heuristic methods struggle due to the subtlety of the concept. A universally agreed upon definition of "hallucination" is needed, and one such definition comes from a University of Illinois Champagne-Urbana paper, which describes it as a generated output that conflicts with constraints or deviates from desired behavior in actual deployment.