RSA is a public key cryptography system used to secure data transmitted over the internet. It is most commonly used in the establishment of an SSL/TLS session – and by the OpenVPN protocol (and sometimes IKEv2) to secure the TLS handshake.
This algorithm is called RSA because of the surnames of the three men who proposed it in 1977 (Ron Rivest, Adi Shamir, and Leonard Adleman). It is an asymmetric encryption system that uses two RSA Keys, known as a key pair.
As with other public-key encryption systems, RSA key exchange involves the sharing of a public key that is derived from the private key at the time of generation. In this type of encryption system, anybody with access to the private key can infer the public key.
Due to the complex mathematical system involved, the opposite (deriving the private key from the public key), is impossible. This is why it is safe to share the public key over the internet to establish a secure connection and begin sharing encrypted data.
RSA key exchange
The important thing to remember is that before you can share encrypted data over the internet, it is first necessary to establish a secure connection between the client and the server.
To do this, a key exchange – called a handshake – must occur so that both parties can agree on the keys that will be used to encrypt the data.
Currently, there are five different algorithms that clients can use to carry out that key exchange, of which RSA is one. We have included all five algorithms below:
- RSA
- Diffie-Hellman
- Elliptic Curve Diffie-Hellman
- Ephemeral Diffie-Hellman
- Ephemeral Elliptic Curve Diffie-Hellman
It is worth noting that some clients (such as the WireGuard protocol) leverage other cryptographic primitives such as Curve25519 to establish the handshake. However, this is still just an elliptic curve designed for use with the elliptic curve Diffie–Hellman key agreement scheme mentioned above.
The RSA Keys
The RSA key-pair is the name for the public and private keys used by the RSA algorithm. The public RSA key is the encryption key, whereas the private key (which must be kept secret to ensure that only the intended recipient can read the data) is the decryption key.
One thing worth noting is that the RSA algorithm is actually pretty slow, primarily because of its asymmetric nature. As a result, protocols often leverage RSA as part of an encryption suite to transmit shared keys for symmetric key cryptography, which are then used for the bulk encryption/decryption.
Thus, RSA is not usually leveraged to encrypt bulk data itself, but rather to establish the means for sharing data encrypted with a faster symmetric encryption algorithm like AES.
RSA Keys and VPNs
If your VPN provider uses the OpenVPN or SSTP protocol to establish a secure tunnel between you and its VPN servers, this means that the VPN client is using RSA keys to secure the TLS Handshake.
The OpenVPN protocol uses RSA on the control channel to pass over the symmetric keys required for the AES encryption used on the data channel. For that handshake to be secure, the RSA key size should be a minimum of 2048 bits.
Many VPN providers nowadays use 4096-bit keys, but most experts do not consider this strictly necessary for security purposes. Thus, an OpenVPN tunnel established with an RSA handshake key size of 2048 bit is not yet considered a cause for concern.
Finally, it is also worth noting that some VPNs also use RSA to secure the TLS handshake in their implementation of IKEv2 (such as ProtonVPN, for example, which implements IKEv2 with an AES-256 symmetric cipher using RSA-4096 to secure the TLS handshake).