On Epistemic Uncertainty of Visual Tokens for Object Hallucinations in LargeVision-Language Models

AI systems can sometimes incorrectly identify objects in images because their internal data representations become uncertain. These representations, called "visual tokens," can be like blurry clues, leading the AI to imagine things that aren't present. Researchers have addressed this by identifying and masking these uncertain tokens, similar to correcting a smudged photo. This method prevents the AI's uncertainty from influencing its interpretation of the image. The masking technique significantly improves the accuracy of the AI's visual descriptions, reducing these "hallucinations." This leads to a more trustworthy understanding of the visual world for these AI models. The approach is relatively simple but has a large positive impact. It works well in combination with other AI enhancements. This ultimately moves us closer to more reliable AI vision systems. This research improves the ability of AI to accurately perceive the world.