Video annotator: building video classifiers using vision-language models and active learning

Video Annotator (VA) is a framework that addresses challenges in training video classifiers. It leverages vision-language models and active learning for efficient annotation, allowing domain experts to guide the process. VA follows a three-step process: search for initial examples using text-to-video search, actively learn and refine annotations using a human-in-the-loop system, and review and refine annotations iteratively. VA enhances sample efficiency, reduces costs, and improves model quality. It enables direct involvement of domain experts in annotation, fostering trust and ownership. Active learning in VA allows users to focus on progressively harder examples, reducing annotation time and improving model performance. VA supports continuous annotation, allowing rapid deployment, monitoring, and correction of edge cases. It empowers users to iterate and improve models without relying on data scientists or third-party annotators. Experiments show that VA leads to higher quality video classifiers compared to baseline methods. VA enables efficient annotation of diverse video understanding tasks. It promotes collaboration between domain experts and machine learning engineers. The authors provide a dataset with 153k labels across 56 tasks annotated using VA, and release code for replication. VA addresses the challenges of conventional classifier training techniques, enhancing efficiency, quality, and user involvement in video annotation. It fosters a sense of ownership and trust in the system, facilitating iterative improvements and rapid deployment of accurate video classifiers.