Entropy quantifies the surprise associated with a random event's outcome. A fair coin, with its unpredictable results, exhibits high entropy, while a rigged coin with mostly predictable outcomes has low entropy. Mathematically, entropy is the expected value of the surprise, calculated as the sum of probabilities of each outcome multiplied by the negative logarithm of its probability. The logarithm of a probability represents surprise, with rarer events causing more surprise due to their smaller probabilities. The negative sign ensures a positive "surprise score," and multiplying by probability and summing yields the average surprise over many trials.Cross-entropy extends this concept to situations where the true probability distribution is unknown, which is common in machine learning. It measures the surprise encountered when using a predicted distribution to calculate probabilities based on the true distribution. The key difference lies in using the logarithm of the predicted probability (q(x)) while still weighting by the true probability (p(x)) in the sum. If the predicted distribution closely matches the true distribution, cross-entropy will be low. Conversely, if the model is confidently incorrect, the predicted probability's logarithm will be a large negative number, resulting in a high cross-entropy value. This behavior makes cross-entropy an ideal loss function for machine learning models.In neural network classification, the model outputs a probability distribution over potential classes. Cross-entropy loss effectively compares this predicted distribution to the true class label, often represented as a one-hot vector. For a one-hot encoded true label, the cross-entropy formula simplifies, focusing solely on the predicted probability of the correct class. During training, the neural network aims to maximize the predicted probability of the correct class, thus minimizing the cross-entropy loss towards zero. This process drives the model to make increasingly accurate and confident predictions.