variable flen in *_rsa_priv_enc

[Jouni Malinen]

On Fri, Feb 01, 2008 at 04:41:12PM +0100, Carolin Latze wrote:

> I am still working on the integration of the TPM into wpa_supplicant. At
> the moment, I am wondering, why flen in the *_rsa_priv_enc is always 36
> (at least in my scenario). From what I see in cryptoapi_rsa_priv_enc in
> tls_openssl.c, wpa_supplicant also expects it to be 36 as this is 16
> (MD5) + 20 (SHA-1), but why?

The length is required to be 36 since only the standard RSA signing for
TLS is supported (see RFC 2246, Chap. 4.7).

> I need an SHA-1 hash to sign using the TPM.

What exactly are you signing? Are you using DSS instead of RSA signing?

-- 
Jouni Malinen                                            PGP id EFC895FA