Your first VPS: what should be done before putting a site on it?

You’ve got a fresh VPS and can log in. What comes next before you point a real website at it?

Updates, SSH access, a firewall and backups are a starting point. Monitoring and a way back in if an access change goes wrong deserve a place on the list too.

What would you tell someone setting up their first server? Short, practical suggestions would be useful here, particularly the things that are easy to forget.