Start with three signals you will actually read: request errors, queue depth, disk heading toward drama. Store logs as long as your legal story allows and your disk forgives—JSON lines beat fancy if someone ever greps them. When HTTP surfaces misbehave, MDN’s status code guide keeps incident rooms from inventing vocabulary.
Alert hygiene is emotional labor
If your phone buzzes without action, demote it. Sleep is part of availability too—just less tweetable.