Disable FIPS mode when RADIUS is being used

[Jouni Malinen]

On Mon, Feb 24, 2025 at 04:45:10PM -0500, Alan DeKok wrote:
>   OpenSSL can operate in FIPS mode, where this is a configuration flag, and not a build requirement.  Applications like RADIUS need to disable FIPS, otherwise the Response Authenticator and Message-Authenticator won't be calculated correctly.

What is the purpose for forcing FIPS mode to be used by default in
systemwide configuration? If that is to prevent use of deprecated crypto
algorithms and parameters, it feels wrong to disable that. If that is
for some other reason, it would be good to understand what that other
reason is. I could understand doing this in some cases where the Linux
kernel command line parameter fips=1 is used to force certain crypto
operation validation to be performed without any intent to actually
certify the system as being compliant with FIPS requirements.

>   OpenSSL will not return an error when asked to do MD4 / MD5 calculations.  It will just silently do the wrong thing.  For an example FreeRADIUS / eapol_test output, see:

I'm not sure what that claim is based on.. OpenSSL (well, at least 3.4)
reports that the algorithm cannot be fetched. There were number of cases
where hostap.git code did not pass that to upper layers, including many
RADIUS cases, and I fixed those, but in any case, it is quite clear that
the MD4/MD5 operation itself is failing. And with the current hostap.git
snapshot, it is now quite obvious in eapol_test output as well (and its
behavior: it won't send the Authentication-Request message anymore if it
fails to generate the authenticator).

>   I've attached two patches.
> 
>   One patch adds a fips_disable() function to src/crypto/crypto*.c.  The OpenSSL one tries to disable FIPS, and returns an error if it cannot.  The other crypto*.c files have an empty function defined, which always succeeds.

I would prefer to hide this all in OpenSSL specific code in
crypto_openssl.c and not make it more complex for other libraries.

>  The second patch makes the RADIUS client and server call the new fips_disable() function.

Or upper layer code.. RADIUS code should not need to know about such
detail in one of the possible crypto libraries.

I pushed out different way of addressing this internally within
crypto_openssl.c. This is done only in builds that do not include
CONFIG_FIPS=y.

It should also be noted that FIPS mode in OpenSSL is not the only issue.
It is also possible to use openssl.cnf to make applications load only
the fips provider by default. That will break lots of functionality
needed for Wi-Fi, i.e., this is not limited to MD4/MD5. This will also
break cases like use of HMAC-SHA256 with a short salt instead of a key
(i.e., cases that show up in many protocols, including TLS which is
considered an exception within OpenSSL itself but not for external
users). That HMAC-SHA256 case is needed for SAE (WPA3-Personal), i.e.,
this type of OpenSSL configuration is completely unusable for almost all
Wi-Fi use cases..

The changes I added for this will handle both disabling of FIPS mode and
explicit loading of the default provider if the fips provider is loaded.
I added this only for OpenSSL 3.x and newer since the older versions are
not really supported (well, at least freely) anymore.

-- 
Jouni Malinen                                            PGP id EFC895FA