WolfSSL fails on MacOS

[scaprile]

5.7.0_1 , about to get a file from ourselves (locally signed CA)

961c1  3 net.c:199:mg_listen            1 3 https://localhost:12347
961c1  3 net.c:174:mg_connect           2 -1 https://localhost:12347
961c1  3 sock.c:391:mg_connect_resolved 2 4 -> 127.0.0.1:12347 pend
961c1  3 tls_openssl.c:137:mg_tls_init  2 Setting TLS
961c2  3 tls_openssl.c:232:mg_tls_init  2 SSL client OK
961c3  3 sock.c:446:accept_conn         3 5 accepted 127.0.0.1:49197 -> 127.0.0.1:12347
961c3  3 tls_openssl.c:137:mg_tls_init  3 Setting TLS
**961c3  1 event.c:31:mg_error            3 5 SSL_new**
961c5  3 net.c:148:mg_close_conn        3 5 closed
961c5  3 sock.c:301:read_conn           2 4 21:0:0 -2 err 0
961c5  3 sock.c:301:read_conn           2 4 21:0:0 -2 err 0
961c5  3 net.c:148:mg_close_conn        2 4 closed
FAILURE unit_test.c:1278: fetch(&mgr, buf, url, "GET /a.txt HTTP/1.0\n\n") == 200

but it is SSL_new() that fails on the server side

mongoose/src/tls_openssl.c

Lines 136 to 145 in a9902b5

	MG_DEBUG(("%lu Setting TLS", c->id));
	tls->ctx = c->is_client ? SSL_CTX_new(SSLv23_client_method())
	: SSL_CTX_new(SSLv23_server_method());
	#ifdef MG_TLS_SSLKEYLOGFILE
	SSL_CTX_set_keylog_callback(tls->ctx, ssl_keylog_cb);
	#endif
	if ((tls->ssl = SSL_new(tls->ctx)) == NULL) {
	mg_error(c, "SSL_new");
	goto fail;
	}

adding

#if MG_TLS == MG_TLS_WOLFSSL && !defined(OPENSSL_COMPATIBLE_DEFAULTS)
  if (opts->ca.len == 0 || mg_strcmp(opts->ca, mg_str("*")) == 0) {
    SSL_CTX_set_verify(tls->ctx, SSL_VERIFY_NONE, NULL);
  }
#endif

before calling SSL_new() does not help.

[gmholland]

I have also seen this error with WolfSSL 5.6.3 but on Linux. I have not tested on macOS. The workaround was to ensure a cert and key were set on the SSL context using SSL_CTX_use_certificate(tls->ctx, cert) and SSL_CTX_use_PrivateKey(tls->ctx, key) before calling SSL_new()