Safety is not directly affected by your best performance, or even your average performance. What matters, directly, is your worst-ever performance. This is called the minimax principle: make sure your worst-case performance is good enough. […]
If your usual tolerances are tight enough, then even on the super-rare occasions when your performance is ten times worse than usual, you will still have a wide margin of safety.