HSMs contain a number of elements that are able to detect attempts to compromise their integrity, both at the software level and at the physical level. Physical security is usually provided by opaque epoxy for key elements of the HSM or by a number of sensors that detect attempts at so-called side channel attacks. Just a simple intrusion under the HSM's cover will ensure irreversible deletion of protected data. In terms of protection against the common unwanted visitor in the datacenter, it is common for most HSMs to have the entire front panel locked and therefore not even basic access to the basic configuration.

The actual delivery of the HSM from the factory is monitored by the manufacturer after delivery to us. It is then up to us to inspect the delivered HSM, which includes checking various elements to ensure that the HSM has not been tampered with en route. Examples include holographic stickers, logs with the declared product number or verification of the checksum data for the delivered software. A separate and very important chapter is the actual setup of the HSM and key handling processes and setting up an appropriate security policy.

Keys in an HSM can usually be protected using a software token with a PIN, a physical USB token or smart cards. In the case of smart cards, the activation of the key, i.e. its use by the application, is dependent on the operator inserting a card or multiple cards into the HSM. In fact, the key itself can be protected using the so-called M of N card quorum, where M cards out of N cards need to be inserted to activate the key.

There is one important feature of HSM that is related to the use of smart cards and that is the possibility of remote access. In the normal case, HSMs are located somewhere in the data centre and most often in other geographical locations. If the HSM does not support remote management, i.e., the ability to insert smart cards into a reader physically located at home, the added security of smart cards can be time consuming.

The very way of working with multiple keys and multiple protective equipment is one of the important elements of safety. The concept of role separation is used by all major HSM vendors and includes secure key separation, where an application cannot access another application's key (e.g., a file service to a CA), user role separation, where an operator cannot access multiple smart cards at the same time, and for example, an HSM administrator should not have access to keys at all. What differs between vendors is the ability to set the extent of the separation, the number of assignable roles, but also, for example, the reverse option of pooling access from multiple servers of one application to a single key.

The architecture of the actual work with keys and their protection means varies greatly from one HSM manufacturer to another, which determines both the way of working with HSM and the licensing models. These may limit performance, the maximum number of keys, the number of connectable HSM clients, but also the possibility of backup or remote access.

Individual vendors vary widely in the way keys are backed up and in the configuration of the HSM itself, from cumbersome approaches to special physical tokens to the ability to back up encrypted objects by normal file system means.

The communication with HSM clients, i.e. with servers that use protected keys, is practically the same for major HSM vendors and supports common standards such as PKCS#11, OpenSSL, JCE or Microsoft CNG.