This Article demonstrates that direct preference optimization (DPO) can be used to effectively align an unsupervised structure-conditioned language model with biophysical information. The aligned model, ProteinDPO, achieves stability prediction competitive with that of task-specific models and consistently outperforms unsupervised and fine-tuned versions of the model.