feat: use test host name for marker if localhost alias

[fzipi]

what

• use test's host destination if an alias for localhost

why

• allow to test using different logfiles

caveats

• no tests for now, need to understand how to modify name resolution for a test 🤔 . Maybe https://pkg.go.dev/github.com/foxcpp/go-mockdns#section-readme

Fixes #361

[theseion]

What's the use case?

[theseion]

Ok, I understand the use case.

[fzipi]

Thanks for the feedback!

[theseion]

I just tested this PR against CRS and it causes tests to fail. For example, for one test the host header is set to localhost%00.

@airween How were you expecting to handle those kinds of tests with your setup?

[airween]

@airween How were you expecting to handle those kinds of tests with your setup?

Excellent! This is what I exactly needed! Now I can configure different virtual hosts with separated log files, and I'm able to use Host headers in tests.

Thanks guys!

[fzipi]

@theseion Maybe we can add an url parse here, if we prefix the host with http://. E.g.

u, err := url.Parse("http://localhost%00")
	if err != nil {
		log.Fatal(err)
	}

For the test that fails:

2009/11/10 23:00:00 parse "http://localhost%00": invalid URL escape "%00"

And if err != nil we resort to just localhost?

[theseion]

Let me have a chat with @airween. Let's see what he really needs.

[airween]

Let me explain how do (or would) I use this.

I've created a new feature for mod_security and want to test that. I've created a special rule set for that, so I don't use CRS. Therefore I created an own test set against that rule set. I've created the tests based on ftw-tests-schema because I wanted to use ftw.

But I definitely didn't want to create a new test environment, so I've created a new virtual host in my Apache config, something like this:

<VirtualHost *:80>
        ServerAdmin [email protected]

        ServerName xml.mytest.site

Usually I used to configure vhosts that it uses separated logs:

        ErrorLog /var/log/apache2/xml.mytest.site-error.log
        CustomLog /var/log/apache2/xml.mytest.site-access.log combined

and here is how I configured .ftw.yaml:

---
logfile: '/var/log/apache2/xml.mytest.site-error.log'
logmarkerheadername: 'X-XML-Test'
logtype:
  name: 'apache'

(and I put a special rule into my rule set:

SecRule REQUEST_HEADERS:X-XML-Test "@rx ^.*$" \
    "id:999999,\
    phase:1,\
    pass,\
    t:none,\
    log,\
    msg:'%{MATCHED_VAR}'"

)
Actually there is no other modification, eg I haven't changed system's hosts file, because I thought ftw always sends the requests to address which was given in test's dest_addr (see below).

When I test that patch during the developing phase I use curl to send requests:

curl -H "Host: xml.mytest.site" http://localhost/post ...

After I finished the development I've created tests. A test file looks like this:

---
meta:
  author: "airween"
rule_id: 20100
tests:
  - test_id: 1
    stages:
      - description: "New feature test"
        input:
          dest_addr: "127.0.0.1"
          port: 80
          method: "POST"
          headers:
            User-Agent: "OWASP CRS test agent"
            Host: "xml.mytest.site"
            Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
            Content-Type: "application/xml"

So I thought ftw uses both tests[N].stages[M].input.dest_addr to determine the server's address, and sends the request that address (to the port what given in the test!), AND sends the headers which were given in tests[N].stages[M].input.headers.

I didn't think that ftw wants to resolve the address.

If this is a question to be decided, I would suggest that if both values (dest_addr and Host) are present, then ftw does not need to resolve the target, just sends the request to the value of dest_addr with Host header.

[theseion]

So I thought ftw uses both tests[N].stages[M].input.dest_addr to determine the server's address, and sends the request that address (to the port what given in the test!), AND sends the headers which were given in tests[N].stages[M].input.headers.

That is correct. dest_addr is the address of the web server. However, the internal test requests for marking and flushing must always end up in the log, so we cannot use the Host header from the test. Until now, we simply used localhost, since parsing logs (in the current implementation) requires being on the local machine anyway.

Note that with your current setup you cannot write tests for Host header specific things.

I propose that we amend the schema and add a virtual_host field that we can use for the internal requests, thus decoupling test data from request data without the need to specify the host on the command line, which would be annoying if you wanted to run tests against multiple virtual hosts at the same time.

[airween]

However, the internal test requests for marking and flushing must always end up in the log, so we cannot use the Host header from the test.

Perhaps it's not important to me but I don't see the reason why. I thought each test has a marker and a flusher request, so when ftw reads the test, it builds the marker request too (not before) and uses the dest_addr and Host values. (The flusher request too.)

Until now, we simply used localhost, since parsing logs (in the current implementation) requires being on the local machine anyway.

Yes, that was the issue explained in #361.

Note that with your current setup you cannot write tests for Host header specific things.

Yes, I know, but it wasn't problem for me (in this case).

I propose that we amend the schema and add a virtual_host field that we can use for the internal requests, thus decoupling test data from request data without the need to specify the host on the command line, which would be annoying if you wanted to run tests against multiple virtual hosts at the same time.

What would be the difference between virtual_host and Host? I mean if there will be a test like this:

tests:
  - test_id: 1
    stages:
      - description: "New feature test"
        input:
          dest_addr: "127.0.0.1"
          port: 80
          method: "POST"
          virtual_host: "xml.mytest.site"
          headers:
            User-Agent: "OWASP CRS test agent"
            Host: "xml.mytest.site"

(indeed the virtual_host header could be at other place)

In this case, what value would be used for the request Host header?

Or do you mean virtual_host value is for marker and flusher, and Host is for the test itself?

[theseion]

Perhaps it's not important to me but I don't see the reason why. I thought each test has a marker and a flusher request, so when ftw reads the test, it builds the marker request too (not before) and uses the dest_addr and Host values. (The flusher request too.)

The internal request (mark / flush) uses dest_addr (obviously) but not Host. Imagine the following test:

tests:
  - test_id: 1
    stages:
      - description: "New feature test"
        input:
          dest_addr: "127.0.0.1"
          port: 80
          method: "POST"
          headers:
            User-Agent: "OWASP CRS test agent"
            Host: "localhost%00"
        output:
          expect_ids: [123456]

In this case we want to test that a rule that verifies the Host header is triggered. We mustn't use the value of Host for the internal requests, as it's possible that the web server would block the request. So we have to use a valid value for the Host value of the internal request.

Or do you mean virtual_host value is for marker and flusher, and Host is for the test itself?

Yes. virtual_host would become the value of the internal request's Host header, while host becomes the value of the test request's Host header, same as now. This would guarantee:

1. the value of the Host header for internal requests is always valid
2. we don't have to come up with a heuristic to set the value of the Host header for internal requests that might break for some configurations

Actually, maybe we don't need a field virtual_host, since the value of that field will always be the same as input.host. Maybe it would be better to have use_virtual_host_log, which would take a boolean, default false.

[airween]

We mustn't use the value of Host for the internal requests, as it's possible that the web server would block the request. So we have to use a valid value for the Host value of the internal request.

Ah, now I see the point - thanks.

Then I think the most clear solution would be an optional key (eg. marker_host_header) which describes the real meaning of its function, and can be placed at everywhere: globally (.ftw.yaml), only for a test... or a command line argument. If it's not presented, then ftw can use localhost. It's simple and clear.

What do you think?

[theseion]

That was my initial idea with virtual_host, but then I realised that the value would always be identical to the input.host field. The only instances where the values would differ would be cases where virtual_host would have to be localhost because the input.host would fail to match any virtual host and the request would end up in the default virtual host. But since localhost would be the default for virtual_host, a boolean is enough to achieve what we want.

We can call the field something like use_host_for_marker_request if that makes more sense than use_virtual_host_log.

[airween]

But since localhost would be the default for virtual_host, a boolean is enough to achieve what we want.

Except if the test uses Host header - now I see the point.

We can call the field something like use_host_for_marker_request if that makes more sense than use_virtual_host_log.

Would it be a boolean type or a string? Now I think the string variable is the solution.
If yes, then I think this (use_host_for_marker_request) makes more sense, it better expresses the intention.

[theseion]

I was thinking boolean. Not sure what you mean, what would the value of use_host_for_marker_request be it it's a string value?

[airween]

I was thinking boolean. Not sure what you mean, what would the value of use_host_for_marker_request be it it's a string value?

Consider the following test:

tests:
  - test_id: 1
    stages:
      - description: "New feature test"
        use_host_for_marker_request: "xml.mytest.site"
        input:
          dest_addr: "127.0.0.1"
          port: 80
          method: "POST"
          headers:
            User-Agent: "OWASP CRS test agent"
            Host: "xml.mytest.site%00"
        output:
          no_expect_ids: [123456]

As you can see I put use_host_for_marker_request to the stage with a valid value where I want to test, but the Host header contains an invalid string, and the output has changed to no_expect_ids.

With this method, in special cases we can check the Host header. If we want to check Host header as you explained above, then we can do that in case of "localhost" (or any other name if that's the default vhost in webserver's config - not necessarily the "localhost").

[theseion]

That's a good point. However, you could simply rewrite the test with use_host_for_marker_request: "localhost" (and possibly change no_expect_ids to expect_ids) and check the log of the default virtual host. We always know when writing the test in which virtual host the test request will end up in.

I have no hard opinion on this but I would like to keep things as simple as possible and would prefer boolean over virtual host name.

[theseion]

@fzipi do you want to pick this up again with the resolution we found? Or would you like me to continue?

[fzipi]

Pick it up, I can approve 😄

[theseion]

I've opened coreruleset/ftw-tests-schema#29 to amend the schema.

Reworked, changed author.

[theseion]

@fzipi please review

[fzipi]

LGTM