It is a plug-in card (PCI) or external device (RS232/SCSI/IP/USB/PCMCIA) for a general purpose computer and may even be an embedded system itself.
The job of the HSM is to securely generate and/or store long term secrets for use in cryptography and physically protect the access to and use of those secrets over time. Generally these are private keys used in Public-key cryptography; some HSMs also allow for hardware protection of symmetric keys.
Many HSM systems have a means to securely backup the keys either in a wrapped form via the computer's operating system or externally using a smartcard or some other USB token. The most robust HSM systems are those where secrets are not exported in plaintext form even when migrating between HSMs or performing backup operations.
Read the rest of the article here