Etsy's Search Relevance team aims to help buyers find exactly what they seek and sellers reach those buyers. They've developed a Semantic Relevance framework to improve search results beyond engagement signals, using large language models (LLMs). This framework uses human-curated data and LLMs to create relevant, partially relevant, and irrelevant categories. The system utilizes an LLM annotator, a teacher model, and a student model with different trade-offs in accuracy and speed. This three-tiered system improves search by filtering irrelevant listings, enriching features, and adjusting ranking weights. The framework is fully deployed and the percentage of fully relevant listings has increased. Etsy plans to refine relevance understanding, explore fine-grained labels, reduce annotation effort, and streamline their multi-stage model for further improvement.