Optimal asymmetric encryption padding
Optimal Asymmetric Encryption Padding (OAEP) is a probabilistic padding scheme designed for asymmetric encryption algorithms, such as RSA, that transforms a plaintext message into a padded block suitable for encryption, ensuring semantic security and resistance to chosen-ciphertext attacks under the random oracle model.[1] Introduced by Mihir Bellare and Philip Rogaway in their 1994 paper "Optimal Asymmetric Encryption: How to Encrypt with RSA," OAEP employs a Feistel-like structure with two random oracle hash functions, G and H, to randomize the padding process, allowing efficient encryption of messages nearly as long as the modulus while maintaining strong security guarantees.[1]
The scheme operates by generating a random seed r, which is used to mask the plaintext through XOR operations with the outputs of G and H, producing a padded value that is then encrypted using the public-key primitive (e.g., RSA exponentiation).[1] Decryption reverses this by applying the inverse primitive and unmasking, with built-in checks to detect invalid ciphertexts and prevent information leakage.[2] OAEP's "optimal" designation reflects its balance of efficiency—one trapdoor permutation evaluation for encryption and decryption—and provable security properties, including plaintext awareness, which implies non-malleability and security against adaptive chosen-ciphertext attacks (CCA2).[1]
Standardized in PKCS#1 version 2.0 (RFC 2437) and refined in subsequent versions, including the current PKCS#1 v2.2 (RFC 8017), OAEP is recommended for new RSA-based encryption applications due to vulnerabilities in older padding schemes like PKCS#1 v1.5, such as padding oracle attacks.[2] It incorporates a mask generation function (MGF1) and a digest function (e.g., SHA-256), with support for an optional label to bind the encryption to specific contexts, enhancing its applicability in protocols like TLS and secure messaging.[2] Despite its security proofs relying on idealized assumptions, real-world implementations must address side-channel risks and use appropriate hash functions to maintain robustness.[2]
Background and Motivation
Role of Padding in Asymmetric Encryption
In asymmetric encryption schemes such as RSA, padding refers to the process of augmenting the plaintext message with additional structured data, including random bits, before applying the encryption function, thereby transforming a potentially deterministic operation into a probabilistic one.[1] This addition of randomness and formatting is essential to obscure the underlying message structure and prevent adversaries from exploiting predictable patterns in ciphertexts.[3]
Unpadded asymmetric encryption, exemplified by textbook RSA where the ciphertext is computed as c = m^e \mod n for plaintext m, message exponent e, and modulus n, suffers from inherent determinism: identical plaintexts always produce identical ciphertexts, allowing an attacker to detect repetitions or infer message properties through frequency analysis or other statistical methods.[1] This lack of probabilistic behavior renders such schemes vulnerable to chosen-plaintext attacks (CPA), where an adversary can encrypt chosen messages to compare against observed ciphertexts and deduce information about unknown plaintexts, as well as chosen-ciphertext attacks (CCA), where partial decryption oracles enable manipulation of ciphertexts to reveal plaintext details.[3] Without padding, these vulnerabilities compromise the confidentiality of the encryption, as the trapdoor permutation underlying RSA does not inherently conceal all partial information about the input.[1]
Padding addresses these issues by expanding the plaintext to the full block size (typically just below the modulus length in octets), incorporating random nonces or seeds to ensure each encryption of the same message yields a unique ciphertext, and embedding redundancy or checksums for integrity verification during decryption.[3] For instance, the padded block is formatted to span nearly the entire modulus size, preventing direct embedding of short messages that could leak size information, while the randomness thwarts pattern-based attacks.[1] Upon decryption, the recipient checks the padding structure; invalid padding indicates tampering or errors, enabling rejection of malformed ciphertexts without exposing plaintext, thus mitigating CCA risks.[3]
A core benefit of padding is its role in achieving semantic security, also known as indistinguishability under chosen-plaintext attack (IND-CPA), which requires that no efficient adversary can distinguish the encryption of one plaintext from another of equal length with non-negligible probability.[3] In public-key systems like RSA, semantic security cannot be realized through the raw permutation alone due to its determinism and lack of diffusion, but padding introduces the necessary randomization to approximate the ideal of perfect secrecy, ensuring that ciphertexts reveal no useful information about the plaintext beyond its length.[1] This probabilistic transformation is a prerequisite for secure asymmetric encryption in practice.[3]
Limitations of Earlier Padding Schemes
Earlier padding schemes for RSA encryption, such as PKCS#1 v1.5, introduced structure to handle message formatting but retained vulnerabilities due to their predictable formats and insufficient randomization. In PKCS#1 v1.5, the encryption block consists of a block type byte (02 for encryption), a padding string of random non-zero bytes (PS), a single 00 byte delimiter, and the message data (D), ensuring the total length matches the modulus size.[4] While PS provides some randomness, the rigid format—particularly the fixed block type and delimiter—allows attackers to exploit implementation details, such as error messages revealing whether a ciphertext conforms to the padding rules.[4]
A prominent vulnerability in PKCS#1 v1.5 is Bleichenbacher's adaptive chosen-ciphertext attack, which leverages a "padding oracle" where the decryptor inadvertently leaks information about padding validity through differing error responses.[5] This attack enables an adversary to iteratively refine a target ciphertext by submitting a relatively small number (on the order of hundreds of thousands to a few million) of modified ciphertexts and observing the oracle's responses, allowing practical recovery of the plaintext for 1024-bit moduli.[5] In SSL/TLS implementations using RSA key transport, this oracle arose from servers distinguishing between decryption failures due to invalid padding versus other errors, allowing full message decryption without the private key.[5]
Even simpler schemes exacerbated these issues. Raw RSA encryption, without any padding, directly applies the modular exponentiation to the plaintext, limiting messages to less than the modulus length and rendering the scheme deterministic—identical plaintexts yield identical ciphertexts, enabling traffic analysis and chosen-plaintext attacks.[6] Moreover, raw RSA is malleable: an attacker can multiply a ciphertext by a factor modulo the modulus to produce a related plaintext, facilitating attacks like the Coppersmith method for small exponents.[6] Basic random padding, such as prepending random bytes to the message, fails to ensure consistent message expansion or provide semantic security, as it lacks a verifiable structure and remains vulnerable to chosen-ciphertext manipulations that alter padding without detection.[6]
These flaws had significant real-world consequences, particularly in early HTTPS deployments around 1998, where Bleichenbacher's attack compromised SSL 3.0 implementations, leading to vulnerabilities in major software like Microsoft Internet Explorer and Netscape Navigator that allowed unauthorized decryption of encrypted sessions.[7] Such breaches highlighted the inadequacy of these schemes for providing IND-CCA security, prompting the development of more robust alternatives.[7]
History and Development
Origins and Key Publications
Optimal Asymmetric Encryption Padding (OAEP) was introduced by Mihir Bellare and Phillip Rogaway in their seminal 1994 paper presented at Eurocrypt, titled "Optimal Asymmetric Encryption: How to Encrypt with RSA."[1] This work proposed OAEP as a padding scheme designed to transform any trapdoor one-way permutation, such as RSA, into a secure public-key encryption method, emphasizing efficiency and provable security properties.[1] The scheme addressed the need for asymmetric encryption that could handle messages of variable length while maintaining strong security guarantees against common attacks.[1]
The key contribution of the 1994 paper lies in its proof that OAEP achieves semantic security against chosen-plaintext attacks (IND-CPA) from the one-wayness of a trapdoor permutation, under the random oracle model, along with a claim of plaintext awareness—a property implying security against chosen-ciphertext attacks (CCA)—though the latter proof was later found to have a gap.[1] This analysis established OAEP as a foundational advancement, showing that it provides strong security properties while requiring only a single application of the underlying permutation and using hash functions modeled as ideal random oracles.[1] The analysis highlighted OAEP's superiority over earlier deterministic padding methods by ensuring probabilistic encryption without excessive overhead.[1]
OAEP drew influence from prior cryptographic constructions, particularly Feistel network structures, which had been analyzed for provable security in pseudorandom permutation generation since the late 1980s. The scheme's two-round Feistel-like design, using XOR operations with hash-derived values, built on these foundations to embed randomness and prevent malleability, adapting symmetric cipher techniques to the asymmetric setting.[1] Developments in the 1990s, such as hybrid encryption transforms, further contextualized OAEP's role in achieving strong security from weaker primitives, though OAEP predated specific proposals like the Fujisaki-Okamoto transform.
Subsequent refinements addressed limitations in the original proof, particularly regarding exact security bounds and instantiations of the random oracle. In 2001, Victor Shoup's paper "OAEP Reconsidered," building directly on Bellare and Rogaway's framework, provided a tighter security analysis, confirming OAEP's IND-CCA security under the RSA assumption while clarifying the implications of the random oracle model.[8] This work resolved earlier concerns about proof tightness and influenced further adoptions by establishing more precise reductions to underlying hardness assumptions.[8]
Standardization Efforts
Optimal Asymmetric Encryption Padding (OAEP) was formalized in PKCS #1 version 2.0, published in October 1998 by RSA Laboratories, as the RSAES-OAEP encryption scheme, with parameters specifying SHA-1 as the underlying hash function and the mask generation function MGF1 based on SHA-1.[9] Subsequent updates in PKCS #1 version 2.1 (2002) extended support to stronger hash functions, including members of the SHA-2 family (SHA-256, SHA-384, SHA-512), to address evolving security requirements while maintaining compatibility; version 2.2 (RFC 8017, November 2016) provided editorial updates and reference revisions without adding new hash functions.[10][2]
The scheme gained broader adoption through inclusion in the IEEE Standard for Public-Key Cryptography (IEEE P1363-2000), which specified OAEP for RSA-based encryption in public-key protocols, emphasizing its role in secure key establishment. Similarly, OAEP was integrated into key establishment mechanisms in NIST Special Publication 800-56B (initially published in 2006 and revised in 2009 and 2014), recommending RSA-OAEP as an approved method for pairwise key transport in federal systems.
In modern protocols, OAEP plays a key role in secure messaging via the Cryptographic Message Syntax (CMS), standardized in RFC 5652 (2009) as an update to PKCS #7, where RFC 3560 (2003) explicitly defines the use of RSAES-OAEP for key transport in enveloped data structures to ensure confidentiality.[11][12] Although TLS 1.3 (2018) eliminated static RSA key exchange in favor of ephemeral Diffie-Hellman methods, OAEP remains relevant for legacy RSA encryption in hybrid contexts within extended TLS deployments.[13]
Amid post-quantum cryptography transitions, NIST's 2024 standards (FIPS 203, 204, and 205) focus on quantum-resistant algorithms like ML-KEM for key establishment, but ongoing discussions in NIST IR 8547 (2024 draft) and related literature explore hybrid variants combining classical schemes such as RSA-OAEP with post-quantum primitives to facilitate gradual migration, with no substantive modifications to the core OAEP mechanism as of 2025.[14][15]
Algorithm Description
Encoding Procedure
The Optimal Asymmetric Encryption Padding (OAEP) encoding procedure transforms a short message into a full-length string suitable for encryption under a trapdoor permutation such as RSA, incorporating randomness to enhance security. The process assumes a security parameter k representing the bit length of the RSA modulus N, with the message block length n = k - k_0, where k_0 is the output length of the hash function H (typically 160 or 256 bits in practice). The input message x must be at most n bits long; if shorter, it is padded by appending zero bits to reach exactly n bits. The public key (N, e) is used only in the final RSA encryption step after padding, while a random seed r of exactly k_0 bits is generated for each encoding. Two cryptographic functions are required: an expanding generator G: \{0,1\}^{k_0} \to \{0,1\}^n and a compressing hash H: \{0,1\}^n \to \{0,1\}^{k_0}, modeled as random oracles in the original design.[1]
The encoding proceeds in the following steps to produce a k-bit padded message w:
-
Generate a random seed r \in \{0,1\}^{k_0} uniformly at random.
-
Compute the masked data block s = x \oplus G(r), where \oplus denotes bitwise XOR and x is the (padded) message.
-
Compute the masked seed t = r \oplus H(s).
-
Concatenate to form the padded message w = s \| t, which has length n + k_0 = k bits.
The output w is then encrypted using the RSA public key as y = w^e \mod N, yielding the ciphertext. This procedure ensures the padded message is exactly k bits long, with randomness introduced via the seed r to prevent deterministic attacks on the underlying permutation. In widely adopted implementations, such as PKCS#1, the procedure is adapted to handle variable-length messages up to k - 2hLen - 2 octets (where hLen is the hash output length in octets) by incorporating a label hash, padding string of zeros, and a delimiter byte in the data block before masking, using a mask-generation function (MGF) derived from a hash like SHA-256 for both G and H. In this version, the encoded message EM is formed as \text{EM} = 0^{00} \| \text{maskedSeed} \| \text{maskedDB}, where maskedDB includes the label hash || PS (zeros) || 0x01 || M, maskedSeed is hLen octets, and maskedDB is k/8 - hLen - 1 octets.[1][2]
Decoding Procedure
The decoding procedure for Optimal Asymmetric Encryption Padding (OAEP) inverts the encoding process to recover the original message from the ciphertext using the recipient's private key. It begins with the RSA decryption step followed by the removal of the OAEP padding, incorporating checks to ensure the integrity of the recovered data. This process relies on the same mask generation function, typically instantiated as MGF1 in standardized implementations (replacing the separate G and H from the original design), and assumes a modulus length of k bits where the seed length is l = hLen * 8 bits (hLen being the hash output length in octets).[16]
Given the ciphertext C and the private key (N, d), the first step computes the decrypted value X = C^d \mod N, which yields a k-bit string representing the padded message EM. If the resulting X does not fit within k bits or violates basic length constraints (e.g., k < 16 hLen + 16 bits, equivalent to k/8 < 2 hLen + 2 octets), the procedure outputs a decryption failure to prevent information leakage.[17]
For the standardized procedure, EM is split into three parts: the first 8 bits form Y (which must be 0x00), the next l bits form maskedSeed (r'), and the remaining k - l - 8 bits form maskedDB (db'). If Y ≠ 0x00, output decryption failure. The unmasking begins by computing the seed candidate seed' = r' \oplus \text{MGF}(db', l), where MGF(db', l) generates an l-bit mask from the input db'. Then, the data block candidate is recovered as db'' = db' \oplus \text{MGF}(seed', k - l - 8), applying the same function MGF to expand seed' into a mask of length k - l - 8 bits. These XOR operations leverage the invertibility of the Feistel-like structure in OAEP to separate the intertwined seed and data components.[1][17]
The final step parses db'' to extract the original message M and verify the label hash lHash = Hash(L). The data block db'' is expected to consist of lHash (first hLen octets), followed by a padding string PS of zero octets, a separator octet (0x01), and then the message M. The procedure indexes after the lHash to find the longest PS of zeros followed by 0x01; if lHash does not match the expected value, no valid 0x01 is found, PS contains non-zero bits, or the first octet of db'' after lHash is non-zero in invalid positions, the ciphertext is rejected as invalid, resulting in a decryption failure. Upon successful validation, the message M is extracted from the portion after the 0x01 and PS.[17]
OAEP's all-or-nothing transform property ensures that partial computations on invalid inputs do not reveal any information about the message or padding without completing the full unmasking and parsing steps, enhancing resistance to error oracle attacks during decryption. This property arises from the diffusion provided by the iterative application of the mask generation function across the entire padded block.[1]
Security Properties
Provable Security Model
Optimal Asymmetric Encryption Padding (OAEP) achieves its security guarantees within the random oracle model (ROM), a theoretical framework where hash functions are modeled as ideal, randomly chosen functions that behave like true random oracles. In this model, OAEP transforms a trapdoor permutation that is secure against chosen-plaintext attacks (CPA), such as RSA, into a full-fledged public-key encryption scheme that is secure against adaptive chosen-ciphertext attacks (CCA). This elevation from IND-CPA to IND-CCA security is a key contribution, as it ensures that even if an adversary can query a decryption oracle with chosen ciphertexts (except for the target one), they cannot distinguish encryptions of different messages.[1]
The original security analysis for OAEP was provided by Bellare and Rogaway in 1994, who aimed to demonstrate that if the underlying trapdoor permutation is IND-CPA secure, then the OAEP scheme is IND-CCA secure under the ROM. However, this proof was later found to be flawed. In 2001, Victor Shoup's analysis ("OAEP Reconsidered") identified issues in the reduction and provided a correct proof in the ROM, though it was not tight—resulting in security parameters with polynomial losses relative to the underlying permutation's security. A tight security reduction, establishing IND-CCA security directly under the RSA assumption (without full IND-CPA for RSA), was subsequently given by Fujisaki, Okamoto, Pointcheval, and Stern in 2001, with parameters approximately preserving the original security bounds under the ROM.[1][18][19]
OAEP is considered "optimal" in this context because it attains the strongest notion of CCA security with the minimal computational overhead: specifically, it requires only two evaluations of random oracle hash functions, avoiding the need for more expensive primitives or multiple rounds of interaction. This efficiency makes OAEP particularly suitable for practical asymmetric encryption while preserving provable security in the ROM.[1]
Despite these strong theoretical foundations, the ROM has faced significant critiques, notably from Canetti, Goldreich, and Halevi, who showed in 2000 that there exist schemes secure in the ROM whose security fails when random oracles are replaced by any concrete hash function, even programmable ones. Their work highlights that the ROM provides idealized security proofs but does not guarantee instantiability with real-world hash functions, raising concerns about the model's applicability to non-programmable hashes. Nevertheless, OAEP has withstood extensive cryptanalysis over decades and remains secure in practice when instantiated with standard hash functions like SHA-256, as evidenced by its continued adoption without known breaks.[20]
Resistance to Specific Attacks
OAEP provides strong resistance to chosen-ciphertext attacks (CCA), particularly those exploiting padding oracles, such as the Bleichenbacher attack that targeted the PKCS#1 v1.5 padding scheme by allowing adversaries to iteratively refine ciphertexts based on decryption error messages indicating padding validity. In contrast, OAEP's use of feistel-like masking with hash functions and a mask generation function ensures that all invalid decryptions appear indistinguishable from random failures, preventing adversaries from gaining incremental information about the plaintext without constant-time implementations to avoid timing leaks.[2][1]
The scheme achieves indistinguishability under adaptive chosen-ciphertext attack (IND-CCA2) security, where an adversary with access to a decryption oracle cannot distinguish encryptions of two chosen plaintexts, except for the challenge ciphertext, even after adaptively querying the oracle. This property is proven in the random oracle model, assuming the computational difficulty of inverting the underlying trapdoor permutation, such as RSA, with tight security reductions established for practical parameters.[19][2]
OAEP's design inherently mitigates side-channel risks by avoiding structures that leak partial information during decryption, unlike PKCS#1 v1.5 where the initial block-type byte could be extracted via faults or power analysis; proper implementation with constant-time hashing and masking further ensures no timing or error distinctions reveal padding details.[2][21]
As of 2025, no fatal cryptanalytic weaknesses have been discovered in OAEP when paired with secure hash functions like SHA-256, though implementations using deprecated hashes such as MD5 should be avoided due to known collisions and preimage vulnerabilities in the hash itself that could undermine the masking process.[2] Like other RSA-based encryptions, OAEP remains susceptible to quantum computing threats via Shor's algorithm, prompting recommendations for transition to post-quantum cryptographic schemes.
Implementations and Applications
Pseudocode Examples
The Optimal Asymmetric Encryption Padding (OAEP) scheme, as formalized in the original proposal, employs a Feistel-like structure with random oracles G and H to process the message before applying the RSA trapdoor permutation, ensuring IND-CCA2 security under the random oracle model.[1] In practical implementations, such as those specified in PKCS#1 v2.2, G and H are realized using a hash function (e.g., SHA-256, producing 256 bits of output) combined with a mask generation function (MGF1) for expansion, while the label L is typically an empty string unless specified otherwise.[3] For a 2048-bit RSA modulus (k = 256 octets), the maximum message length is k - 2*hLen - 2 = 190 octets (1520 bits), where hLen = 32 octets for SHA-256, yielding up to 1520 bits for the message after accounting for padding and masks.[3]
The following pseudocode illustrates the OAEP encoding procedure, assuming octet-string operations and standard cryptographic primitives; it integrates the EME-OAEP encoding primitive with RSA encryption.
function OAEP_Encode(message M, public_key pk=(n, e), params):
// params: hash_function Hash (e.g., SHA-256, hLen = Hash.output_length),
// mgf MGF1, label L (default empty string)
k = bit_length(n) / 8 // modulus length in octets
if |M| > k - 2 * hLen - 2:
error "message too long"
if |L| > 2^61 - 1: // hash input limit example for SHA-256
error "label too long"
lHash = Hash(L)
PS = zeros(k - |M| - 2 * hLen - 2) // padding string of zeros
DB = lHash || PS || 0x01 || M // data block
seed = random_bytes(hLen) // random seed
dbMask = MGF(seed, k - hLen - 1) // mask for DB
maskedDB = DB XOR dbMask
seedMask = MGF(maskedDB, hLen) // mask for seed
maskedSeed = seed XOR seedMask
EM = 0x00 || maskedSeed || maskedDB // encoded message
m = OS2IP(EM) // octet string to integer
c = pow(m, e, n) // RSA encryption
C = I2OSP(c, k) // integer to octet string
return C // ciphertext
function OAEP_Encode(message M, public_key pk=(n, e), params):
// params: hash_function Hash (e.g., SHA-256, hLen = Hash.output_length),
// mgf MGF1, label L (default empty string)
k = bit_length(n) / 8 // modulus length in octets
if |M| > k - 2 * hLen - 2:
error "message too long"
if |L| > 2^61 - 1: // hash input limit example for SHA-256
error "label too long"
lHash = Hash(L)
PS = zeros(k - |M| - 2 * hLen - 2) // padding string of zeros
DB = lHash || PS || 0x01 || M // data block
seed = random_bytes(hLen) // random seed
dbMask = MGF(seed, k - hLen - 1) // mask for DB
maskedDB = DB XOR dbMask
seedMask = MGF(maskedDB, hLen) // mask for seed
maskedSeed = seed XOR seedMask
EM = 0x00 || maskedSeed || maskedDB // encoded message
m = OS2IP(EM) // octet string to integer
c = pow(m, e, n) // RSA encryption
C = I2OSP(c, k) // integer to octet string
return C // ciphertext
This encoding generates a random seed, applies masking via MGF to diffuse randomness across the padded message, and prepends a leading zero octet to ensure the result fits the RSA domain.[3]
The decoding procedure reverses the process, verifying the structure and recovering the original message or failing on inconsistencies.
function OAEP_Decode([ciphertext](/page/Ciphertext) C, private_key sk=(n, d), params):
// params: as in encoding
k = |C|
if k < 2 * hLen + 2:
error "decryption error"
if |L| > 2^61 - 1:
error "label too long"
c = OS2IP(C)
m = pow(c, d, n) // [RSA](/page/RSA) decryption (RSADP)
if m >= n:
error "decryption error"
EM = I2OSP(m, k)
if EM[0] != 0x00:
error "decryption error"
Y = EM[0:1] // should be 0x00
maskedSeed = EM[1 : 1 + hLen]
maskedDB = EM[1 + hLen : k]
seedMask = MGF(maskedDB, hLen)
seed = maskedSeed XOR seedMask
dbMask = MGF(seed, k - hLen - 1)
DB = maskedDB XOR dbMask
lHash_prime = DB[0 : hLen]
lHash = [Hash](/page/Hash)(L)
if lHash_prime != lHash:
error "decryption error"
// Parse DB: find 0x01 separator after lHash and zero PS
i = hLen
while i < len(DB) and DB[i] == 0x00:
i += 1
if i == len(DB) or DB[i] != 0x01:
error "decryption error"
PS = DB[hLen : i] // must be all zeros (implicit check via loop)
M = DB[i + 1 : ] // recovered message to end of DB
return M
function OAEP_Decode([ciphertext](/page/Ciphertext) C, private_key sk=(n, d), params):
// params: as in encoding
k = |C|
if k < 2 * hLen + 2:
error "decryption error"
if |L| > 2^61 - 1:
error "label too long"
c = OS2IP(C)
m = pow(c, d, n) // [RSA](/page/RSA) decryption (RSADP)
if m >= n:
error "decryption error"
EM = I2OSP(m, k)
if EM[0] != 0x00:
error "decryption error"
Y = EM[0:1] // should be 0x00
maskedSeed = EM[1 : 1 + hLen]
maskedDB = EM[1 + hLen : k]
seedMask = MGF(maskedDB, hLen)
seed = maskedSeed XOR seedMask
dbMask = MGF(seed, k - hLen - 1)
DB = maskedDB XOR dbMask
lHash_prime = DB[0 : hLen]
lHash = [Hash](/page/Hash)(L)
if lHash_prime != lHash:
error "decryption error"
// Parse DB: find 0x01 separator after lHash and zero PS
i = hLen
while i < len(DB) and DB[i] == 0x00:
i += 1
if i == len(DB) or DB[i] != 0x01:
error "decryption error"
PS = DB[hLen : i] // must be all zeros (implicit check via loop)
M = DB[i + 1 : ] // recovered message to end of DB
return M
Decoding includes integrity checks on the label hash and padding string; failure in any verification (e.g., non-zero PS or mismatched lHash) results in an error to prevent chosen-ciphertext attacks.[3] This structure's use of dual masking provides the security benefits of full-domain enforceability and resistance to adaptive attacks.[1]
Usage in Cryptographic Standards
Optimal Asymmetric Encryption Padding (OAEP) has been integrated into major cryptographic libraries since the early 2000s, providing robust support for RSA-based encryption. In OpenSSL, the functions RSA_padding_add_PKCS1_OAEP_mgf1 and RSA_padding_check_PKCS1_OAEP_mgf1 implement OAEP as specified in PKCS #1 v2.0, with support added in version 0.9.2b released in 2000. The Bouncy Castle library includes OAEPEncoding for RSA encryption, available since version 1.0 in 2000 and compliant with FIPS 140-2 requirements in later releases. Similarly, Crypto++ supports OAEP through classes like RSAES<OAEP>, introduced in version 2.1 around 1998 but fully aligned with PKCS #1 standards by the early 2000s.[22]
OAEP is employed in several key protocols for secure data transport. In the Transport Layer Security (TLS) protocol version 1.2, RSA-OAEP serves as an option for key transport, enhancing security over legacy padding in scenarios requiring RSA encryption, though TLS 1.3 shifts away from RSA key exchange entirely in favor of Diffie-Hellman-based methods.[23] The World Wide Web Consortium (W3C) XML Encryption standard (2002) mandates RSA-OAEP for key transport, using identifiers like http://www.w3.org/2001/04/xmlenc#rsa-oaep-mgf1p with SHA-1 and MGF1, to protect XML data structures. In the JSON Object Signing and Encryption (JOSE) framework, including JSON Web Encryption (JWE) and JSON Web Signature (JWS), RSA-OAEP and RSA-OAEP-256 are registered algorithms for key encryption, with RSA-OAEP-256 recommended for its use of SHA-256 to mitigate SHA-1 weaknesses.[24]
Best practices for OAEP deployment emphasize security enhancements to prevent common vulnerabilities. Cryptographic implementations should employ strong hash functions such as SHA-256 or higher for both the digest and mask generation function (MGF1), as SHA-1 is deprecated due to collision vulnerabilities. Constant-time operations are essential in OAEP processing to resist timing attacks that could leak information about the plaintext or key, with libraries like OpenSSL incorporating such measures in recent versions. For efficiency with large messages, OAEP should be used in hybrid encryption schemes, where RSA-OAEP encrypts a symmetric key (e.g., AES-256), which then secures the bulk data.[25]
Regarding deprecations and migrations, NIST Special Publication 800-56B Revision 2 (2019) recommends OAEP-based schemes like RSA-KEM for pair-wise key establishment using integer factorization cryptography, superseding older paddings like PKCS #1 v1.5 due to proven vulnerabilities such as Bleichenbacher's attack. As of 2025, major ecosystems have advanced toward full migration: for instance, the Go programming language deprecated RSA PKCS #1 v1.5 encryption in favor of OAEP for FIPS compliance, and AWS CloudHSM ended support for PKCS #1 v1.5 padding in FIPS mode RSA operations on January 1, 2024.[26][27] This shift ensures OAEP's provable security model is standard in production systems. As of 2025, OAEP remains recommended in NIST SP 800-56B Rev. 2 for key establishment.[28]