Demystifying Machine Learning

Summary:

• K-fold cross-validation
  • Divide dataset into $K$ sets. Keep one set as test and use remaining as train. Repeat for each set.
• 10-fold cross-validation: Most common.
• Leave-one-out cross-validation: Keep one instance as test, and use remaining as train. Repeat for each instance.
• N x K-fold cross-validation or repeated cross-validation: Repeat $K$-fold cross-validation with $N$ different splits.
• Two use cases:
  • Compute a more accurate/less biased estimate of your algorithm’s performance
  • For a less biased hyperparameter tuning
• Combine both use cases with nested cross-validation
  • “Outer” cross-validation for algorithm’s performance estimate
  • “Inner” cross-validation for hyperparameter selection