Our research introduces guardrailed-AMIE (g-AMIE), an AI system designed for physician oversight in medical diagnostics. G-AMIE can gather patient information through dialogue and generate summaries, differential diagnoses, and management plans. Crucially, it is constrained to prevent providing individualized medical advice. Overseeing physicians review and edit this information via a clinician cockpit interface before patient communication. A randomized virtual OSCE study compared g-AMIE's performance with nurse practitioners and physician assistants/associates operating under similar constraints. Results showed that g-AMIE's diagnostic performance and management plans were preferred by overseers and independent raters. Patient actors also favored g-AMIE's drafted patient messages. The study highlighted g-AMIE's adherence to safety guardrails and its high-quality history taking and SOAP notes. While g-AMIE outperformed controlled groups in several metrics, the workflow was specifically designed for AI and not fully representative of clinician training. Limitations include potential rater disagreement and the cognitive load of oversight. Future work will focus on optimizing verbosity and exploring real-world settings. This framework represents a significant step towards human-AI collaboration in medical diagnostics.