---
name: "libressl-BN_generate_prime"
section: "3"
platform: "linux"
package: "libressl"
version: "4.4.0-1"
release: "Arch Linux 2026-11-01"
platforms: ["linux"]
url: "https://new.carta.tech/libressl-BN_generate_prime"
---

# libressl-BN\_generate\_prime(3)

generate primes and test for primality

## NAME

**BN\_is\_prime\_ex**, **BN\_is\_prime\_fasttest\_ex**, **BN\_generate\_prime\_ex**, **BN\_GENCB\_call**, **BN\_GENCB\_new**, **BN\_GENCB\_free**, **BN\_GENCB\_set**, **BN\_GENCB\_get\_arg**, **BN\_GENCB\_set\_old** – generate primes and test for primality

## SYNOPSIS

Crypto Library (libcrypto, -lcrypto)

**#include \<openssl/bn.h\>**

*int*\

**BN\_is\_prime\_ex**(*const BIGNUM \*a*, *int nchecks*, *BN\_CTX \*ctx*, *BN\_GENCB \*cb*);

*int*\

**BN\_is\_prime\_fasttest\_ex**(*const BIGNUM \*a*, *int nchecks*, *BN\_CTX \*ctx*, *int do\_trial\_division*, *BN\_GENCB \*cb*);

*int*\

**BN\_generate\_prime\_ex**(*BIGNUM \*ret*, *int bits*, *int safe*, *const BIGNUM \*modulus*, *const BIGNUM \*remainder*, *BN\_GENCB \*cb*);

*int*\

**BN\_GENCB\_call**(*BN\_GENCB \*cb*, *int state\_code*, *int serial\_number*);

*BN\_GENCB* *\**\

**BN\_GENCB\_new**(*void*);

*void*\

**BN\_GENCB\_free**(*BN\_GENCB \*cb*);

*void*\

**BN\_GENCB\_set**(*BN\_GENCB \*cb*, *int (\*cb\_fp)(int, int, BN\_GENCB \*)*, *void \*cb\_arg*);

*void* *\**\

**BN\_GENCB\_get\_arg**(*BN\_GENCB \*cb*);

Deprecated:

*void*\

**BN\_GENCB\_set\_old**(*BN\_GENCB \*cb*, *void (\*cb\_fp)(int, int, void \*)*, *void \*cb\_arg*);

## DESCRIPTION

**BN\_is\_prime\_ex**() and **BN\_is\_prime\_fasttest\_ex**() test whether the number *a* is prime. In LibreSSL, both functions behave identically and use the Baillie-Pomerance-Selfridge-Wagstaff algorithm combined with *checks* Miller-Rabin rounds. The *do\_trial\_division* argument is ignored.

It is unknown whether any composite number exists that the Baillie-PSW algorithm misclassifies as a prime. Some suspect that there may be infinitely many such numbers, but not a single one is currently known. It is known that no such number exists below 2^64.

In order to reduce the likelihood of a composite number passing the primality tests **BN\_is\_prime\_fasttest\_ex**() and **BN\_is\_prime\_ex**(), a number of rounds of the probabilistic Miller-Rabin test is performed. If *checks* is positive, it is used as the number of rounds; if it is zero or the special value BN\_prime\_checks, a suitable number of rounds is calculated from the bit length of *a*.

If NULL is passed for the *ctx* argument, these functions allocate a *BN\_CTX* object internally when they need one and free it before returning. Alternatively, to save the overhead of allocating and freeing that object for each call, the caller can pre-allocate a *BN\_CTX* object and pass it in the *ctx* argument.

**BN\_generate\_prime\_ex**() generates a pseudo-random prime number of at least bit length *bits* and places it in *ret*. Primality of *ret* is tested internally using **BN\_is\_prime\_ex**(). Consequently, for *bits* larger than 64, it is theoretically possible that this function might place a composite number into *ret*; the probability of such an event is unknown but very small.

The prime may have to fulfill additional requirements for use in Diffie-Hellman key exchange:

- If *modulus* is not NULL, a prime is generated that fulfills the condition *ret* % *modulus* = *remainder*. If the *remainder* argument is NULL, 1 is used as the desired remainder.

- If the *safe* argument is non-zero, a safe prime is generated, that is, (*ret* - 1)/2 is also prime.

If *cb* is not NULL, it is used as follows:

- **BN\_GENCB\_call**(*cb*, *0*, *serial\_number*) is called after generating a potential prime number.

- The *state\_code* of 1 is reserved for callbacks during primality testing, but LibreSSL performs no such callbacks.

- When *safe* is non-zero and a safe prime has been found, **BN\_GENCB\_call**(*cb*, *2*, *serial\_number*) is called.

- The callers of **BN\_generate\_prime\_ex**() may call **BN\_GENCB\_call**() with other values as described in their respective manual pages; see SEE ALSO.

In all cases, the *serial\_number* is the number of candidates that have already been discarded for not being prime; that is, *serial\_number* is 0 for the first candidate and then incremented whenever a new candidate is generated.

**BN\_GENCB\_call**() calls the callback function held in *cb* and passes the *state\_code* and the *serial\_number* as arguments. If *cb* is NULL or does not contain a callback function, no action occurs.

**BN\_GENCB\_new**() allocates a new *BN\_GENCB* object.

**BN\_GENCB\_free**() frees *cb*. If *cb* is NULL, no action occurs.

**BN\_GENCB\_set**() initialises *cb* to use the callback function pointer *cb\_fp* and the additional callback argument *cb\_arg*.

The deprecated function **BN\_GENCB\_set\_old**() initialises *cb* to use the old-style callback function pointer *cb\_fp* and the additional callback argument *cb\_arg*.

## RETURN VALUES

**BN\_is\_prime\_ex**() and **BN\_is\_prime\_fasttest\_ex**() return 0 if the number is composite, 1 if it is prime with a very small error probability, or -1 on error.

**BN\_generate\_prime\_ex**() returns 1 on success or 0 on error.

**BN\_GENCB\_call**() returns 1 on success, including when *cb* is NULL or does not contain a callback function, or 0 on error.

**BN\_GENCB\_new**() returns a pointer to the newly allocated *BN\_GENCB* object or NULL if memory allocation fails.

The callback functions pointed to by the *cb\_fp* arguments are supposed to return 1 on success or 0 on error.

**BN\_GENCB\_get\_arg**() returns the *cb\_arg* pointer that was previously stored in *cb* using **BN\_GENCB\_set**() or **BN\_GENCB\_set\_old**().

In some cases, error codes can be obtained by ERR\_get\_error(3).

## SEE ALSO

BN\_new(3), DH\_generate\_parameters(3), DSA\_generate\_parameters\_ex(3), RSA\_generate\_key(3)

## HISTORY

**BN\_generate\_prime\_ex**(), **BN\_is\_prime\_ex**(), **BN\_is\_prime\_fasttest\_ex**(), **BN\_GENCB\_call**(), **BN\_GENCB\_set\_old**(), and **BN\_GENCB\_set**() first appeared in OpenSSL 0.9.8 and have been available since OpenBSD 4.5.

**BN\_GENCB\_new**(), **BN\_GENCB\_free**(), and **BN\_GENCB\_get\_arg**() first appeared in OpenSSL 1.1.0 and have been available since OpenBSD 6.3.