Enable BN_mod_sqrt() for upcoming LibreSSL 3.8.2

This API was inherited from OpenSSL, so it has always been present.
Enable it for the upcoming LibreSSL release.

The test as it was written would fail since LibreSSL returns the other
square root. Improve the test to work with all possible implementations
with a more interesting test case. Also check for error in the simplest
situation where no square root exists.