Pinterest Search is a key surface where users can discover inspiring content that aligns with their information needs, and search relevance measures how well the search results align with the search query. To improve the search relevance model, a 5-level guideline is used to measure the relevance between queries and Pins. A cross-encoder language model is used to predict a Pin's relevance to a query, along with Pin text, and the task is formulated as a multiclass classification problem. The model is fine-tuned using human-annotated data, minimizing cross-entropy loss.To represent each Pin, a varied set of text features is used, including Pin titles and descriptions, synthetic image captions, high-engagement query tokens, user-curated board titles, and link titles and descriptions. However, the cross-encoder LLM-based classifier is hard to scale for Pinterest Search due to real-time latency and cost considerations. Therefore, knowledge distillation is used to distill the LLM-based teacher model into a lightweight student relevance model.The student model uses query-level features, Pin-level features, and query-Pin interaction features to predict 5-scale relevance scores. Knowledge distillation and semi-supervised learning are employed to train the student model, which makes effective use of vast amounts of initially unlabeled data and expands the data to a wide range of languages from around the world.Offline experiments demonstrate the effectiveness of each modeling decision, including the comparison of language models, the importance of enriching text features, and scaling up training labels through distillation. Online results show a +2.18% improvement in search feed relevance, as measured by nDCG@20, and a significant uptick in search fulfillment rates globally.The proposed relevance modeling pipeline effectively generalizes across languages not encountered during training, and the multilingual LLM-based relevance teacher model generalizes across unseen languages. Future work will explore the integration of servable LLMs, vision-and-language multimodal models, and active learning strategies to dynamically scale and improve the quality of the training data.