HOME BLOG ARCHIVE TAGS

A nasty libcrypto type cast

January 27, 2018

Have you ever wondered what happens when OpenSSL EC_KEY* is (incorrectly) cast to RSA*? Tracking a key modulus comparison bug, I discovered it worked by coincidence on 32-bit systems running Linux:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
// on x86
struct EC_KEY {               struct RSA {
    int            version;       int               pad;
    // ------------------------------------------------------
    EC_GROUP      *group;         long              version;
    // ------------------------------------------------------
    EC_POINT      *pub_key;       const RSA_METHOD *meth;
    // ------------------------------------------------------
    BIGNUM        *priv_key;      ENGINE           *engine;
    // ------------------------------------------------------
    unsigned int   enc_flag;      BIGNUM           *n; // <==
    // ------------------------------------------------------
    // (...)
};

Zero-initialized enc_flag was seen as (RSA*)->n, and BN_cmp() NULL checks short-circuited. On 64-bit Linux systems, however, helpful SIGSEGVs were raised:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
// on x64
struct EC_KEY {                         struct RSA {
    int                      version;       int               pad;
    // ----------------------------------------------------------------
    EC_GROUP                *group;         long              version;
    // ----------------------------------------------------------------
    EC_POINT                *pub_key;       const RSA_METHOD *meth;
    // ----------------------------------------------------------------
    BIGNUM                  *priv_key;      ENGINE           *engine;
    // ----------------------------------------------------------------
    unsigned int             enc_flag;      BIGNUM           *n; // <==
    point_conversion_form_t  conv_form;
    // ----------------------------------------------------------------
    // (...)
};

Fortunately, conv_form wasn’t also zero-initialized, and resulting (RSA*)->n pointer became invalid.