Error estimation

• Using Root Mean Square Error (RMSE), you canc compare the observed value from the training data with the diatom-inferred value
  • RMSE from the training set will underestimate RMSE of prediction (RMSEP), as we learned in statistics (predictions in linear models)
• Ideally, we could take out some data from training our model and use that for testing. But this wastes expensive data, so often we use computer-generated replicates in a process called Cross-Validation (AKA jack-knifing, bootstrapping)
  • Basically it "'scrambles" our modern data hundreds of times to evaluate the robustness of the model
  • This used to take a very long time but it's not so bad now
  • Modern computers make it easy to test model robustness

Sources of Error

RMSEP composed of:

1. Within lake variability, up to 15% of total RMSEP - different cores from different parts of the lake will have different species. Even within one core, there can be lots of random variation (random effects) about
2. Variability in environmental data - the time of year, weather, contamination when you take the sample can greatly affect environmental variables like pH, temperature etc. Often lakes can't be monitored all year
3. Variability due to other environmental gradients - lots of confounding effects of nutrients, pH and temperature are all related, and sometimes you cannot identify which one is affecting your proxy.
4. Variance in the model. If you have a huge model, you get more noise and larger RMSEP - always go back to your original data and see if it makes sense.
   1. Make sure you actually understand what your script is doing!

Model evaluation - examples

• Evaluate the percentage of fossil biota represented in the modern training set
• Check floristic similarity (e.g. Chi-square distance between fossil sample and closest modern analogue)

Other evaluation measures

• Compare reconstructions with instrumental data
• Compare multiple reconstructions for the same proxy with different methods
• Consensus reconstruction - use different methds for the same variable and see how different they are.
  • Tree rings, other values...
  • It's better to have these differences than to have one unreliable estimator - then we can think critically about each estimate and see which isbest
• Compare different reconstructions from different proxies from the same core