If you are building an application that is specifically meant to be used with one exact hardware module, your best option would probably be to use PKCS directly. You will have less abstraction layers, and less dependency on external libraries.
Furthermore, some modules do not implement PKCS completely, or even correctly. If you are using PKCS calls in your application, you have more flexibility to work around these limitations, assuming that with EVP, you'd be using the PKCS #11 backend. You'll also have the direct ability to add key management functionality to your application, should you desire to do so.