Update nginx-logs.yaml to accept host:port combined NCSA patter

[click]

Expand grok-match for NCSA vcombined format

[LaurenceJJones]

Hey 👋🏻

Just to ask because I never seen this before, you are using the server_name directive in nginx to have domain + port

EG: server_name my.domain.tld:9090;

so when you have the log format set to:

    log_format main '$server_name $remote_addr - $remote_user [$time_local] '
                '"$request" $status $body_bytes_sent '
                '"$http_referer" "$http_user_agent"';
    # Logging
    access_log             /var/log/nginx/access.log main;

The $server_name is what you changed here?

[LaurenceJJones]

Okay I just tested it with ip's but would be the same for domains:

192.168.121.181:8080 192.168.121.1 - - [14/Nov/2024:09:02:03 +0000] "GET / HTTP/1.1" 200 11 "-" "curl/8.11.0"
192.168.121.181 192.168.121.1 - - [14/Nov/2024:09:02:05 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/8.11.0"

 cat sites-available/* | grep server_name
        server_name 192.168.121.181;
        server_name 192.168.121.181:8080;

but obviously if the domain is on a different port the server would be only listening for that domain.

however, if you have 2 domains/subdomains EG:

my.domain.tld
my.domain.tld:443
my.other.tld
my.other.tld:443

However, from nginx point of the view the server doesnt need to have the port assigned in the server name since the listen directive has the port and vhost can just be the domain name but there is a downside to having a port here.

[click]

Base debian/ubuntu apache2.conf:

# The following directives define some format nicknames for use with
# a CustomLog directive.
#
# These deviate from the Common Log Format definitions in that they use %O
# (the actual bytes sent including headers) instead of %b (the size of the
# requested file), because the latter makes it impossible to detect partial
# requests.
#
# Note that the use of %{X-Forwarded-For}i instead of %h is not recommended.
# Use mod_remoteip instead.
#
LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %O" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent

The LogFormat .... vhost_combined is the specific part ref logoutput.
The combined logformat (builtin) in nginx is also the same as the baseline combined for apache2 (both are based on the original NCSA log-format)
vhost_combined is this essentially the combined extended format prefix'ed with $host:$port (and is supported with the apache2 log-grok, but not with nginx log-grok).

Basically, the nginx logparser/grok-section will get 443 as the fqdn_host when using the standard grok (not patched).

Also, having a few separate vhosts on separate ports will show the issue as well (I know I prefer to know the -exact- port a request comes in on, especially when rummaging with http=>https redirects etc).

[LaurenceJJones]

We are investigating an issue on our side, you changes are published, however, there currently issue downloading them remotely.

I will update this PR once we have it resolved.

[LaurenceJJones]

Update: this should now be live and can be downloaded via updates

cscli hub update && cscli hub upgrade