Configuration file for HTTP Unix daemon

[triska]

In this issue, I would like to:

1. collect everything that has so far been said about configuration files in the context of the HTTP Unix daemon.
2. add new information that has since arisen due to new features of library(ssl)
3. find an elegant, flexible and convenient way to run HTTPS servers with SWI-Prolog.

The (perceived) need for a configuration file was first raised and supported in #54, with Wouter and me in favour of the idea. It was dismissed again because for that concrete issue, a more trivial fix was applied. Now, a few months later and with the availability of new features for HTTPS servers, this is coming up again in #76.

First, I would like to give a short summary of what a typical HTTPS server needs as its configuration. It is evident that in the very near future, HTTPS servers will become the "normal" web servers, since plain HTTP servers will be marked as insecure by all major browsers. Therefore, it is wise to consider how to best accommodate their configuration.

Currently, a typical SWI-Prolog HTTPS server is started by supplying the following command line options to the Unix daemon:

• --https (mandatory)
• --keyfile=File (almost all servers will use this, even though it could theoretically be omitted)
• --certfile=File (again, used by almost all servers).

A significant portions of these servers will also use:

• --redirect to redirect HTTP to HTTPS.
• --cipherlist=Atom to set ciphers with (currently) acceptable security.

Now, with recent changes to library(ssl), the following configurations must also be considered:

• SNI, to supply different certificates depending on the host name of the client request
• multiple certificate to support different ciphers, for example dual-stack RSA/ECDSA servers.

SNI is already supported in the Unix daemon by the hook http:sni_options/2. Multiple certificates are not yet possible. However, it is clear that this use case will become increasingly relevant in the future, and so it should also be possible to supply multiple certificates somehow.

Currently, a typical invocation of an HTTPS server could like like this:

swipl rules.pl proloxy.pl \
   --no-fork --user=www --https --certfile=server.crt \
   --keyfile=server.key \
   --pidfile=/var/run/https_proloxy.pid --workers=32 --syslog=https_proloxy \
   --cipherlist=EECDH+AESGCM:EDH+AESGCM:EECDH+AES256:EDH+AES256:CHACHA20 \
   --keep_alive_timeout=2

I have highlighted options that are specific to HTTPS.

It is evident that such an invocation is extremely inconvenient. Moreover, the following problem arises: If you put all this for example in a .service file for systemd, then you need to do sudo systemctl daemon-reload every time you change the configuration parameters. This is also inconvenient, and there is no way around this.

The point has been raised that the Unix daemon is becoming too complex altogether, providing features that go far beyond its initial conception of a "Unix daemon". To be honest, my impression is almost the opposite: In less than 1000 lines of code (including extensive documentation), the daemon manages to expose an impressive amount of very flexible functionality, far more than I initially thought would be possible by so little glue code. If this is now deemed too complex, I wonder what to think about certain other areas of the code that ships with SWI-Prolog. To me, it is a testament to the elegant architecture of the web framework that the Unix daemon takes comparatively little and quite straight-forward code.

We are almost at the point that all of the above is supported. However, multiple certificates remain pending in #76, and this has now provided the impetus to look more closely at the best approach to include this feature too.

In general, we actually need triples of the form:

• certificate
• key
• password.

Using passwords to protect private keys is certainly advisable, but frequently simply not done in practice, also not by Let's Encrypt: In this case and many others, suitable file permissions are used to severely restrict access to private keys and even certificates. Still, we support also encrypted private keys by means of the --pwfile option (which is to be preferred over --password, since command line arguments can be easily inspected by other processes).

So, we could support multiple certificates by using for example multiple occurrences of --certfile etc. However, this gets increasingly complex and unwieldy to use (the implementation remains rather straight-forward, and does not significantly increase the complexity of the Unix daemon).

Now the main point: I still think that a configuration file would be a great idea to solve all this. The main reason for this is the following:

• HTTPS servers will become increasingly common.
• Let's Encrypt provides free certificates that are easy to set up and use.
• SWI-Prolog is increasingly used for web applications.
• That's a great chance to advertise Prolog as a configuration language.

The Prolog community missed the boat when it came to choosing a syntax for RDF and many other web standards, where Prolog syntax could have played a very important role. Why miss the next boat too? Wouldn't it be nice to have Prolog based configuration files as follows, say config.pl:

https.

certfile('server.crt').
keyfile('server.key').

cipherlist('EECDH+AESGCM:EDH+AESGCM:EECDH+AES256:EDH+AES256:CHACHA20').

user(www).

keep_alive_timeout(2).
workers(32).

And then start the daemon with:

swipl rules.pl proloxy.pl --config=config.pl --no-fork  

Further, SNI could be accommodated by providing http:sni_options/2 in the same file.

Moreover, multiple certfile/1 and keyfile/1 options could appear.

Of course, the following syntax would make even more sense:

server([keyfile='server.key',
        certfile='server.crt',
        https=true,
        cipherlist='EECDH+AESGCM:EDH+AESGCM:EECDH+AES256:EDH+AES256:CHACHA20',
        keep_alive_timeout=2,
        workes=32]).

I think adding this feature to the HTTP Unix daemon would allow a very flexible and convenient configuration of (multiple) HTTPS servers and at the same time advertise Prolog as a nice language for configuration files too.

[JanWielemaker]

Thanks for writing this down. Would be nice if one could place margin notes in issues ... So, I'll give a few quoted replies.

The point has been raised that the Unix daemon is becoming too complex altogether, providing features that go far beyond its initial conception of a "Unix daemon". To be honest, my impression is almost the opposite: In less than 1000 lines of code

This remark is in because ideally there should be a Windows counterpart of this file. That would have to deal with all these certificate and server start issues as well, but not with forking, syslog interactions, etc. But, let us not complicate things further than needed and keep focussing on the http_unix_daemon.pl. If we ever write a http_windows_service.pl or something like that we can still refactor the code and move the shared part into a new file.

[JanWielemaker]

swipl rules.pl proloxy.pl \
   --no-fork --user=www --https --certfile=server.crt \ 
   --keyfile=server.key \
   --pidfile=/var/run/https_proloxy.pid --workers=32 --syslog=https_proloxy \
   --cipherlist=EECDH+AESGCM:EDH+AESGCM:EECDH+AES256:EDH+AES256:CHACHA20 \ 
   --keep_alive_timeout=2

Note that most of the options are for classical init daemons. There is no need for --pidfile or --syslog in systemd and other modern service infrastructures. You typically use that if you fork, so you have the pid and you can control the daemon. Also -keep_alive_timeout=2 is default, so you can omit it.

Somehow, I guess we need a sensible default for the --cipherlist. The average sysadmin has little clue.

[JanWielemaker]

Now the main point: I still think that a configuration file would be a great idea to solve all this. The main reason for this is the following:

We have that by means of library(settings). Prolog however has many ways to deal with configuration that all make sense in some context. If we deal purely with name/value pairs, library(settings) works quite nice, but I'd surely consider using JSON or YAML. Prolog starts to shine if configuration becomes more complicated and we need rules.

That is why I'd like to stick with the idea of using hooks in http_unix_daemon.pl. Based on hooks you can process options, use some name/value config files, define rules, etc.

For example: define http:server(-Server, +Options)as a multi predicate in http_unix_daemon.pl. The server startup code calls findall/3 on the above to find the servers to configure. The hook can proces config(ConfigFile) to do its job. The default is for simple cases and sticks with the current, possibly somewhat extended approach. If you want your Prolog config, just add a library(http/http_config_prolog) and
an application does

:- use_module(library(http/http_unix_daemon)).
:- use_module(library(http/http_config_prolog)).

:- initialization http_daemon.

This way we keep things modular and we keep everybody happy. This style is used throughout the HTTP infrastructure and seems to work well.

[triska]

Perfect, I fully support extending this with hooks. Please note that we should have a solution for #71 first, so that the multifile declaration can be properly taken into account.

I see no need for http_config_prolog: The http:server/2 hook you suggest seems to be well worth adding to the HTTP Unix daemon itself and will not complicate it much.

There is one thing though I still would like to discuss. We must distinguish between 3 kinds of options:

1. options for the Unix daemon itself, for example --redirect and --interactive
2. HTTP options such as keep_alive_timeout/1 and workers/1
3. SSL options such as certificate_key_pairs/1 and cipherlist/1.

In the HTTP framework, (3) appear as part of (2), in the form of ssl(Options).

The remaining task is to make the current option processing so general that most practically arising situations are well covered, or at least can be covered with acceptable work.

The HTTP Unix daemon currently (or at least with the proposed patch) uses the following for (3):

    SSL = ssl([ certificate_key_pairs(Pairs),
                cipher_list(CipherList),
                password(Passwd),
                sni_hook(http_unix_daemon:sni)
              ]).

It is completely OK that certificate_key_pairs/1 and cipher_list/1 occur here, since they can always be present (certificate_key_pairs/1 admits an empty list). Regarding cipher_list/1, I would like to address the point you raised: The Unix daemon is currently using the OpenSSL defaults for cipherlist (unless otherwise specified). To improve security, the best bet of all users is to upgrade to the most recent OpenSSL version, where stronger ciphers have become the default. Note that the default options also cater to outdated clients. With the options I recommended, you will exclude clients that do not implement forward secrecy, and in some use cases that may be undesirable. With newer OpenSSL versions, even better defaults may become available in the future, using ciphers that are currently not even devised. Using explicit ciphers by default would exclude those. To sysadmins, I recommend to use the free SSL Labs security assessment for their servers and ciphers.

password/1 is slightly less OK, because it breaks down when multiple certificates are used and encrypted with different keys. Currently, the same password (if any) must be used for all certificates. This is a current limitation of library(ssl), and could be rather easily overcome by providing a passwords/1 option where a list of passwords can be specified. As I said above, private keys are frequently not encrypted at all (but rather protected by suitably restrictive file permissions), and therefore catering for a single password suffices at least for now.

What's definitely not OK if this is to be extended by hooks is the fixed appearance of sni_hook/1 in the call above. This is because users may want to provide a different hook instead of relying on the one that is currently built from the specification of http:sni_options/2.

Here is a list of use cases (which I expect to occur quite frequently in the near future) that are currently not covered by the Unix daemon, and which should become possible with new hooks:

a) HTTPS servers without default certificate (relying fully on SNI)
b) HTTPS servers with multiple certificates
c) HTTPS servers with a custom SNI hook
d) HTTPS servers with all of the above, where we can still use --redirect to redirect from HTTP.

The most important are (a) and (b), which are the subject of #76.

(c) is lower priority, but there is one important point about it: Currently, http:sni_options/2 makes it easy to define servers that handle different domains (via SNI). A sufficiently general new hook could (and, ideally: also should) make this specific hook obsolete, so that we only need the more general hook and can remove http:sni_options/2 again, simplifying the HTTP daemon. The code to set up a custom SNI hook is not very hard:

    findall(HostName-HostOptions, http:sni_options(HostName, HostOptions), SNIs),
    maplist(sni_contexts, SNIs),
    etc.

sni_contexts(Host-Options) :-
    ssl_context(server, SSL, Options),
    assertz(sni(_, Host, SSL)).

I think this would be doable for users who are interested in SNI, and I will document this in LetSWICrypt, so we could remove http:sni_options/2 in favour of the more general hook.

I mention (d) only for completeness: Of course, the current options that affect the daemon itself should all continue to work.

This may sound a bit complex now, so I would like to point out that we are extremely close to this already now: Almost everything works already, and a single new hook may be sufficient to also cover the remaining use cases in an elegant and flexible way.

[triska]

I think I have found a good way to solve these issues: I will add a new predicate called ssl_add_certificate_key/3 that lets you add additional certificates and keys to an SSL context. This new predicate makes the freshly introduced certificate_key_pairs/1 option obsolete, and I have thus filed pull requests to revert this both in SSL and HTTP (no release is affected).

Update: It turns out that certificate_key_pairs/1 is still necessary to allow such options in http:sni_hook/2, i.e., when combining SNI with multiple certificates.

In addition, there will be a new hook hat is invoked by the HTTPS plugin after creating the SSL context of the server. By defining a clause for that hook and using the new ssl_add_certificate_key/3, you will be able to add your default server.crt and server.key to the context. Thus, the only change that will be necessary for the Unix daemon is to not require a default certificate and key.

With some luck, I may be able to use the new predicate to also solve #70, and I will work on this in 7.5. Note that OpenSSL has no API to remove certificates from a context, so this may require much more work, but updating a certificate may turn out to be easy.

I would greatly appreciate a good naming suggestion for the HTTPS hook that is in line with the existing hooks. I note that some of the hooks currently use the http: module qualification, while others use the http_ prefix. Maybe it is time for https:? For example, what about https:post_server_start(+SSL)?

[triska]

(c) above can likewise be solved with a new predicate like ssl_set_sni_hook/2.

I think such predicates to modify an SSL context after its creation are best to obtain the flexibility we need in this case. In many situations, I would like to use the HTTP Unix daemon and make only such small adjustments without having to roll my own server. A single suitable hook to configure an HTTPS server by giving access to the SSL context after its creation is enough to handle all this.

[triska]

I'm closing this because the new hooks http:ssl_server_create_hook/3 and http:ssl_server_open_client_hook/3 are so flexible that, together with the new SSL predicates ssl_add_certificate_key/3 and ssl_set_sni_hook/3 (as proposed in SWI-Prolog/packages-ssl#88), they provide all mentioned features, both with and without the Unix daemon.

The way to extend this in the future is to add further predicates that allow us to adjust an SSL context, in a pure and — therefore — thread-safe way, of course.