Valheim limits the amount of world data exchanged between the server and clients. When several players and many objects are concentrated in one area, this limit can cause lag even if the host machine has sufficient resources.
A community modification intended to address this behavior is available at https://valheim.thunderstore.io/package/Dalayeth/NetworkFix/.
The modification must be installed in compatible versions on both the server and every connecting client. Review its current documentation and compatibility before installation.