If you took Element/Matrix and removed everything from it that is required for federation, then it would perform like Signal.

This is only true for the server side. Federation doesn’t affect the client much at all (for example, rocket.chat is now a matrix client, even though it wasn’t one for the majority of its life).

Matrix clients are mildly harder to build because of, as you mentioned, end to end encryption, but also because of the many ways you can make mistakes in the code that synchronises clients to the server and cause performance issues or bugs. Even dispute this, though, there are a massive variety of matrix clients: https://matrix.org/ecosystem/clients/