Information Extraction with Low-Inference-Cost Models: 85% F1 Gain via Foundry Interactive Training
Businesses often need to convert documents into structured data for workflows. This involves extracting correct values and records for downstream systems. Long documents and repetitive tables present challenges to this process. Researchers tested gpt-oss-20b and gpt-oss-120b as cost-effective alternatives to advanced frontier models. Initially, both models showed shortcomings in field coverage. Task-specific fine-tuning was explored to enhance their performance. They used a schema-guided extraction method, providing the model with document text and a JSON schema. Precision, recall, and F1 scores were used to measure extraction accuracy. Fine-tuning significantly improved the F1 scores for both gpt-oss-20b and gpt-oss-120b. While gpt-oss-20b showed a larger relative gain, gpt-oss-120b achieved a higher final F1 score. These models, though not matching frontier models, demonstrated the value of task-specific fine-tuning. The study concludes that evaluating simpler approaches before training and measuring performance are crucial. Custom fine-tuning with tools like Foundry Interactive Training can yield substantial improvements. The final model choice depends on demonstrated quality and overall deployment costs.