Machine learning models require fine-tuning on domain-specific data, but this can be problematic due to privacy concerns. Differential privacy (DP) allows for training models while respecting privacy, but most work focuses on example-level DP, which has drawbacks. User-level DP is a stronger form of privacy that guarantees an attacker can't learn about their data, and it's used in federated learning. Learning with user-level DP is harder and requires adding more noise, which gets worse with larger models. The paper focuses on fine-tuning large language models with user-level DP in datacenter training. The authors modify stochastic gradient descent (SGD) to add noise and limit the effect each user has on the model. They compare two methods, Example-Level Sampling (ELS) and User-Level Sampling (ULS), which differ in how they sample data. The authors optimize these algorithms for large language models, finding that ULS is generally better, and both methods perform better than no fine-tuning despite the strict privacy requirement. The optimizations enable model trainers to fine-tune their models to sensitive datasets while providing strong user protections.