Expert thread: common pitfalls and solutions in the development of custom risk metrics

One recurring pitfall is overfitting—designing a metric so closely to past data that it loses relevance for future cases. Teams often mistake historical accuracy for ongoing reliability. Our method is to build in cross-validation: splitting data into multiple sets, testing on one, refining on another. This catches blind spots early.

A second challenge is inadequate peer review. When only one or two voices dominate, the metric’s weaknesses may go unnoticed. We assign rotating peer reviewers—someone new challenges the model each cycle. This fosters a culture of honest critique.

Documentation can lag behind model changes, creating confusion later. We insist on living documentation—updated at every stage, visible to all. When everyone contributes, the record stays fresh and future audits are smoother.

Bias can sneak in through familiar data sources. We encourage teams to diversify not only their data, but also their reviewer backgrounds. A mix of industry and academic voices often surfaces new perspectives.