Large-scale text generation with LLMs necessitates automated evaluation due to manual review limitations. The LLM-as-a-Judge approach uses a second LLM to score a primary LLM's output, but raises concerns about the evaluator's trustworthiness. This problem is addressed by building, aligning, and continuously monitoring LLM judges, ensuring they remain anchored in human judgment at every stage. Humans define quality standards, create labeled datasets, monitor judge behavior, and approve updates.A reliable LLM judge is not static; it has a lifecycle involving human ownership in its creation, alignment, operation, and maintenance. This process is illustrated through the task of generating recommendation explanations, which are short messages explaining item recommendations. Quality is crucial as misleading explanations erode user trust.The LLM judge lifecycle comprises four phases: Birth, Training, Deployment, and HITL Alignment & Drift Monitoring. Birth involves human experts crafting benchmark data, including adversarial and boundary examples, which are then labeled by human raters. Training refines the LLM judge to align with human labels and rationales, using self-reflection and rubric updates. Deployment integrates the judge as a gate and critic within the explanation generation system.HITL Alignment & Drift Monitoring ensures continued trustworthiness by periodically benchmarking the judge against human raters and requiring human approval for any system changes. This phase uses a Meta-Judge to compare the LLM judge's rationale with human rationales, catching cases where the judge arrives at the correct answer for the wrong reason. This continuous feedback loop, incorporating new human-labeled data and signals, feeds back into the benchmark and development phases. The system components include a Generator, Judge, Reflector, Meta-Judge, and human experts and raters, all working together across the lifecycle phases to maintain judge integrity.