Google accused of using novices to fact-check Gemini's AI answers

Google has instructed contract workers evaluating its AI system Gemini to not skip any prompts, regardless of their expertise, according to internal guidance viewed by TechCrunch. Previously, contractors could skip prompts outside of their expertise, but now they are told to rate the parts of the prompt they understand and add a note if it's not their area of knowledge. The only exceptions are if a significant amount of information is missing or if the content is harmful and requires specific consent forms. A contractor questioned the change, saying it could decrease accuracy by not giving prompts to those with more expertise. Google responded that the ratings provided by contractors do not directly impact their algorithms, but are a helpful data point to measure system performance. The company also noted that the new language shouldn't affect Gemini's accuracy, as contractors are still rating parts of the prompts they understand. Google emphasized that the ratings are not just about content, but also style, format, and other factors. The company recently released the FACTS Grounding benchmark, which checks LLM responses for factual accuracy and detail. The change in guidance has raised concerns about the reliability of AI evaluations.