Server software is the program that runs your server: the .jar file you launch. It determines what you can install: plugins or mods. The choice usually comes down to one question:
- Players join with unmodded Minecraft — choose Paper: it's faster than the vanilla server and runs Spigot and Bukkit plugins;
- You want more gameplay settings — Purpur— a Paper fork that supports the same plugins.
- Everyone uses the same modpack — Fabric or NeoForge— match the mod loader used by the pack.
- You want several servers in one network — place a Velocity proxy in front of them.
The vanilla server from minecraft.net works well for playing with friends without extras. Setup is covered in How to create a Minecraft server.