The simplest way to do this, is to put the server on a private vpn (I use Tailscale, there are others) and expose ports only to the vpn. Then you share access to the vpn with your friends.
With Tailscale, this is as simple as sending them a share link for the host. They will need to have an account at Tailscale, and have the client running, but they will then be able to access the host with a static ip address.
As a general rule of thumb, nothing should be exposed to the public internet unless you want that service to be public access and then you need to keep it up to date. If a vulnerability doesn’t currently exist for the service, one will sooner rather than later. SSH, especially password only ssh, can be broken into fairly easily. If you must expose ssh to the public internet for whatever reason, you need to be using IP white lists, password protected keys, change the default port, and turn off service advertisements and ping responses. I’m probably missing something. When someone scans your server randomly, they should see nothing. And if they fail login they should be ip blocked.