Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add SecRequestBodyJsonDepthLimit directive #1110

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions coraza.conf-recommended
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,9 @@ SecRule REQUEST_HEADERS:Content-Type "^application/json" \
SecRule REQUEST_HEADERS:Content-Type "^application/[a-z0-9.-]+[+]json" \
"id:'200006',phase:1,t:none,t:lowercase,pass,nolog,ctl:requestBodyProcessor=JSON"

# Configures the maximum JSON recursion depth limit Coraza will accept.
SecRequestBodyJsonDepthLimit 1024

# Maximum request body size we will accept for buffering. If you support
# file uploads then the value given on the first line has to be as large
# as the largest file you are willing to accept. The second value refers
Expand Down
2 changes: 2 additions & 0 deletions experimental/plugins/plugintypes/bodyprocessor.go
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ type BodyProcessorOptions struct {
FileMode fs.FileMode
// DirMode is the mode of the directory that will be created
DirMode fs.FileMode
// RequestBodyRecursionLimit is the maximum recursion level accepted in a body processor
RequestBodyRecursionLimit int
}

// BodyProcessor interface is used to create
Expand Down
38 changes: 28 additions & 10 deletions internal/bodyprocessors/json.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
package bodyprocessors

import (
"errors"
"io"
"strconv"
"strings"
Expand All @@ -13,13 +14,15 @@ import (
"github.com/corazawaf/coraza/v3/experimental/plugins/plugintypes"
)

const ResponseBodyRecursionLimit = -1

type jsonBodyProcessor struct{}

var _ plugintypes.BodyProcessor = &jsonBodyProcessor{}

func (js *jsonBodyProcessor) ProcessRequest(reader io.Reader, v plugintypes.TransactionVariables, _ plugintypes.BodyProcessorOptions) error {
func (js *jsonBodyProcessor) ProcessRequest(reader io.Reader, v plugintypes.TransactionVariables, bpo plugintypes.BodyProcessorOptions) error {
col := v.ArgsPost()
data, err := readJSON(reader)
data, err := readJSON(reader, bpo.RequestBodyRecursionLimit)
if err != nil {
return err
}
Expand All @@ -31,7 +34,7 @@ func (js *jsonBodyProcessor) ProcessRequest(reader io.Reader, v plugintypes.Tran

func (js *jsonBodyProcessor) ProcessResponse(reader io.Reader, v plugintypes.TransactionVariables, _ plugintypes.BodyProcessorOptions) error {
col := v.ResponseArgs()
data, err := readJSON(reader)
data, err := readJSON(reader, ResponseBodyRecursionLimit)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it should never be -1 and always force a limit.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we are assuming we don't trust the downstream service neither?

The directive takes into account that the request might be crafted so it creates a DoS of some kind on the body processor (as it happened in ModSecurity). Will the response value be configurable, and different from the request?

if err != nil {
return err
}
Expand All @@ -41,28 +44,38 @@ func (js *jsonBodyProcessor) ProcessResponse(reader io.Reader, v plugintypes.Tra
return nil
}

func readJSON(reader io.Reader) (map[string]string, error) {
func readJSON(reader io.Reader, maxRecursion int) (map[string]string, error) {
s := strings.Builder{}
_, err := io.Copy(&s, reader)
if err != nil {
return nil, err
}

json := gjson.Parse(s.String())
res := make(map[string]string)
key := []byte("json")
readItems(json, key, res)
return res, nil

if !gjson.Valid(s.String()) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we really need to do this? the first impression is this is very expensive, the second one is when we use a low body limit this will always be invalid isn't/

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not very expensive. Of course, it is more expensive than not doing vlaidation (adds roughly 33% more time in my benchmarks).

The upstream library says that If you are consuming JSON from an unpredictable source then you may want to validate prior to using GJSON. I agree, for the case of a WAF.

Or we can forget about validation, and see if someone can bypass us. 🤷 If we do, we should document clearly this decision that we prefer speed over security here.

return res, errors.New("invalid JSON")
}
json := gjson.Parse(s.String())
err = readItems(json, key, maxRecursion, res)
return res, err
}

// Transform JSON to a map[string]string
// This function is recursive and will call itself for nested objects.
// The limit in recursion is defined by maxItems.
// Example input: {"data": {"name": "John", "age": 30}, "items": [1,2,3]}
// Example output: map[string]string{"json.data.name": "John", "json.data.age": "30", "json.items.0": "1", "json.items.1": "2", "json.items.2": "3"}
// Example input: [{"data": {"name": "John", "age": 30}, "items": [1,2,3]}]
// Example output: map[string]string{"json.0.data.name": "John", "json.0.data.age": "30", "json.0.items.0": "1", "json.0.items.1": "2", "json.0.items.2": "3"}
// TODO add some anti DOS protection
func readItems(json gjson.Result, objKey []byte, res map[string]string) {
func readItems(json gjson.Result, objKey []byte, maxRecursion int, res map[string]string) error {
arrayLen := 0
var iterationError error
if maxRecursion == 0 {
// we reached the limit of nesting we want to handle
return errors.New("max recursion reached while reading json object")
}
json.ForEach(func(key, value gjson.Result) bool {
// Avoid string concatenation to maintain a single buffer for key aggregation.
prevParentLength := len(objKey)
Expand All @@ -77,7 +90,11 @@ func readItems(json gjson.Result, objKey []byte, res map[string]string) {
var val string
switch value.Type {
case gjson.JSON:
readItems(value, objKey, res)
// call recursively with one less item to avoid doing infinite recursion
iterationError = readItems(value, objKey, maxRecursion-1, res)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't it be if maxRecursion == -1 { return maxRecursion } else { return maxRecursion -1 }

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The readItems func is called always with the maxRecursion value > 0. Then each iteration will decrement this value. I'm not following your question :/

if iterationError != nil {
return false
}
objKey = objKey[:prevParentLength]
return true
case gjson.String:
Expand All @@ -97,6 +114,7 @@ func readItems(json gjson.Result, objKey []byte, res map[string]string) {
if arrayLen > 0 {
res[string(objKey)] = strconv.Itoa(arrayLen)
}
return iterationError
}

func init() {
Expand Down
49 changes: 44 additions & 5 deletions internal/bodyprocessors/json_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,21 @@
package bodyprocessors

import (
"errors"
"strings"
"testing"
)

const (
deeplyNestedJSONObject = 15000
maxRecursion = 10000
)

var jsonTests = []struct {
name string
json string
want map[string]string
err error
}{
{
name: "map",
Expand Down Expand Up @@ -56,6 +63,7 @@
"json.f.0.0": "1",
"json.f.0.0.0.z": "abc",
},
err: nil,
},
{
name: "array",
Expand Down Expand Up @@ -116,18 +124,49 @@
"json.1.f.0.0": "1",
"json.1.f.0.0.0.z": "abc",
},
err: nil,
},
{
name: "broken1", # this json test has more opening brackets than closing, so it is broken

Check failure on line 130 in internal/bodyprocessors/json_test.go

View workflow job for this annotation

GitHub Actions / test (1.21.x, ubuntu-latest)

illegal character U+0023 '#'

Check failure on line 130 in internal/bodyprocessors/json_test.go

View workflow job for this annotation

GitHub Actions / lint

invalid character U+0023 '#'

Check failure on line 130 in internal/bodyprocessors/json_test.go

View workflow job for this annotation

GitHub Actions / lint

syntax error: unexpected json in composite literal; possibly missing comma or }

Check failure on line 130 in internal/bodyprocessors/json_test.go

View workflow job for this annotation

GitHub Actions / lint

illegal character U+0023 '#' (typecheck)

Check failure on line 130 in internal/bodyprocessors/json_test.go

View workflow job for this annotation

GitHub Actions / test (1.22.x, ubuntu-latest)

illegal character U+0023 '#'
json: `{"a":{"a":{"a":{"a":{"a":{"a":{"a":{"a":{"a":{"a":{"a":{"a":{"a":{"a":{"a":{"a":{"a": 1 }}}}}}}}}}}}}}}}}}}}}}`,
want: map[string]string{},

Check failure on line 132 in internal/bodyprocessors/json_test.go

View workflow job for this annotation

GitHub Actions / lint

syntax error: unexpected ] in composite literal; possibly missing comma or } (typecheck)
err: errors.New("invalid JSON"),
fzipi marked this conversation as resolved.
Show resolved Hide resolved
},
{
name: "broken2",
json: `{"test": 123, "test2": 456, "test3": [22, 44, 55], "test4": 3}`,
want: map[string]string{
"json.test3.0": "22",
"json.test3.1": "44",
"json.test3.2": "55",
"json.test4": "3",
"json.test": "123",
"json.test2": "456",
"json.test3": "3",
},
err: nil,
},
{
name: "bomb",
json: strings.Repeat(`{"a":`, deeplyNestedJSONObject) + "1" + strings.Repeat(`}`, deeplyNestedJSONObject),
want: map[string]string{
"json." + strings.Repeat(`a.`, deeplyNestedJSONObject-1) + "a": "1",
},
err: errors.New("max recursion reached while reading json object"),
},
}

func TestReadJSON(t *testing.T) {
for _, tc := range jsonTests {

Check failure on line 160 in internal/bodyprocessors/json_test.go

View workflow job for this annotation

GitHub Actions / lint

missing ',' in composite literal (typecheck)
tt := tc
t.Run(tt.name, func(t *testing.T) {
jsonMap, err := readJSON(strings.NewReader(tt.json))
if err != nil {
t.Error(err)
}
jsonMap, err := readJSON(strings.NewReader(tt.json), maxRecursion)
for k, want := range tt.want {

Check failure on line 164 in internal/bodyprocessors/json_test.go

View workflow job for this annotation

GitHub Actions / lint

missing ',' in composite literal (typecheck)
if err != nil && err.Error() == tt.err.Error() {

Check failure on line 165 in internal/bodyprocessors/json_test.go

View workflow job for this annotation

GitHub Actions / lint

missing ',' in composite literal (typecheck)
fzipi marked this conversation as resolved.
Show resolved Hide resolved
continue

Check failure on line 166 in internal/bodyprocessors/json_test.go

View workflow job for this annotation

GitHub Actions / lint

expected operand, found 'continue' (typecheck)
} else if err != nil {
t.Error(err)

Check failure on line 168 in internal/bodyprocessors/json_test.go

View workflow job for this annotation

GitHub Actions / lint

missing ',' before newline in composite literal (typecheck)
}

Check failure on line 169 in internal/bodyprocessors/json_test.go

View workflow job for this annotation

GitHub Actions / lint

missing ',' before newline in composite literal (typecheck)
fzipi marked this conversation as resolved.
Show resolved Hide resolved
if have, ok := jsonMap[k]; ok {
if want != have {
t.Errorf("key=%s, want %s, have %s", k, want, have)
Expand All @@ -150,7 +189,7 @@
tt := tc
b.Run(tt.name, func(b *testing.B) {
for i := 0; i < b.N; i++ {
_, err := readJSON(strings.NewReader(tt.json))
_, err := readJSON(strings.NewReader(tt.json), maxRecursion)
if err != nil {
b.Error(err)
}
Expand Down
15 changes: 8 additions & 7 deletions internal/corazawaf/transaction.go
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@ type Transaction struct {
ForceRequestBodyVariable bool
RequestBodyAccess bool
RequestBodyLimit int64
RequestBodyJsonDepthLimit int
ForceResponseBodyVariable bool
ResponseBodyAccess bool
ResponseBodyLimit int64
Expand All @@ -78,7 +79,7 @@ type Transaction struct {
HashEnforcement bool

// Stores the last phase that was evaluated
// Used by allow to skip phases
// Used by allow to skip phasesx
lastPhase types.RulePhase

// Handles request body buffers
Expand Down Expand Up @@ -996,9 +997,9 @@ func (tx *Transaction) ProcessRequestBody() (*types.Interruption, error) {
tx.WAF.Rules.Eval(types.PhaseRequestBody, tx)
return tx.interruption, nil
}
mime := ""
mimeType := ""
if m := tx.variables.requestHeaders.Get("content-type"); len(m) > 0 {
mime = m[0]
mimeType = m[0]
}

reader, err := tx.requestBodyBuffer.Reader()
Expand All @@ -1011,7 +1012,7 @@ func (tx *Transaction) ProcessRequestBody() (*types.Interruption, error) {
// Default variables.ReqbodyProcessor values
// XML and JSON must be forced with ctl:requestBodyProcessor=JSON
if tx.ForceRequestBodyVariable {
// We force URLENCODED if mime is x-www... or we have an empty RBP and ForceRequestBodyVariable
// We force URLENCODED if mimeType is x-www... or we have an empty RBP and ForceRequestBodyVariable
if rbp == "" {
rbp = "URLENCODED"
}
Expand All @@ -1035,8 +1036,9 @@ func (tx *Transaction) ProcessRequestBody() (*types.Interruption, error) {
Msg("Attempting to process request body")

if err := bodyprocessor.ProcessRequest(reader, tx.Variables(), plugintypes.BodyProcessorOptions{
Mime: mime,
StoragePath: tx.WAF.UploadDir,
Mime: mimeType,
StoragePath: tx.WAF.UploadDir,
RequestBodyRecursionLimit: tx.WAF.RequestBodyJsonDepthLimit,
}); err != nil {
tx.debugLogger.Error().Err(err).Msg("Failed to process request body")
tx.generateRequestBodyError(err)
Expand Down Expand Up @@ -1262,7 +1264,6 @@ func (tx *Transaction) ProcessResponseBody() (*types.Interruption, error) {
}

tx.debugLogger.Debug().Str("body_processor", bp).Msg("Attempting to process response body")

if err := b.ProcessResponse(reader, tx.Variables(), plugintypes.BodyProcessorOptions{}); err != nil {
tx.debugLogger.Error().Err(err).Msg("Failed to process response body")
tx.generateResponseBodyError(err)
Expand Down
2 changes: 1 addition & 1 deletion internal/corazawaf/transaction_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -1665,7 +1665,7 @@ func TestResponseBodyForceProcessing(t *testing.T) {
if _, err := tx.ProcessRequestBody(); err != nil {
t.Fatal(err)
}
tx.ProcessResponseHeaders(200, "HTTP/1")
tx.ProcessResponseHeaders(200, "HTTP/1.1")
if _, _, err := tx.WriteResponseBody([]byte(`{"key":"value"}`)); err != nil {
t.Fatal(err)
}
Expand Down
25 changes: 20 additions & 5 deletions internal/corazawaf/waf.go
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,12 @@ import (
"github.com/corazawaf/coraza/v3/types"
)

// Default settings
const (
// DefaultRequestBodyJsonDepthLimit is the default limit for the depth of JSON objects in the request body
DefaultRequestBodyJsonDepthLimit = 1024
)

// WAF instance is used to store configurations and rules
// Every web application should have a different WAF instance,
// but you can share an instance if you are ok with sharing
Expand All @@ -44,6 +50,9 @@ type WAF struct {
// Request body page file limit
RequestBodyLimit int64

// Request body JSON recursive depth limit
RequestBodyJsonDepthLimit int

// Request body in memory limit
requestBodyInMemoryLimit *int64

Expand Down Expand Up @@ -176,9 +185,10 @@ func (w *WAF) newTransaction(opts Options) *Transaction {
tx.AuditLogParts = w.AuditLogParts
tx.ForceRequestBodyVariable = false
tx.RequestBodyAccess = w.RequestBodyAccess
tx.RequestBodyLimit = int64(w.RequestBodyLimit)
tx.RequestBodyLimit = w.RequestBodyLimit
tx.RequestBodyJsonDepthLimit = w.RequestBodyJsonDepthLimit
tx.ResponseBodyAccess = w.ResponseBodyAccess
tx.ResponseBodyLimit = int64(w.ResponseBodyLimit)
tx.ResponseBodyLimit = w.ResponseBodyLimit
tx.RuleEngine = w.RuleEngine
tx.HashEngine = false
tx.HashEnforcement = false
Expand All @@ -194,13 +204,13 @@ func (w *WAF) newTransaction(opts Options) *Transaction {
tx.Timestamp = time.Now().UnixNano()
tx.audit = false

// Always non-nil if buffers / collections were already initialized so we don't do any of them
// Always non-nil if buffers / collections were already initialized, so we don't do any of them
// based on the presence of RequestBodyBuffer.
if tx.requestBodyBuffer == nil {
// if no requestBodyInMemoryLimit has been set we default to the requestBodyLimit
var requestBodyInMemoryLimit int64 = w.RequestBodyLimit
var requestBodyInMemoryLimit = w.RequestBodyLimit
if w.requestBodyInMemoryLimit != nil {
requestBodyInMemoryLimit = int64(*w.requestBodyInMemoryLimit)
requestBodyInMemoryLimit = *w.requestBodyInMemoryLimit
}

tx.requestBodyBuffer = NewBodyBuffer(types.BodyBufferOptions{
Expand Down Expand Up @@ -294,6 +304,7 @@ func NewWAF() *WAF {
RequestBodyAccess: false,
RequestBodyLimit: 134217728, // Hard limit equal to _1gb
RequestBodyLimitAction: types.BodyLimitActionReject,
RequestBodyJsonDepthLimit: DefaultRequestBodyJsonDepthLimit,
ResponseBodyAccess: false,
ResponseBodyLimit: 524288, // Hard limit equal to _1gb
auditLogWriter: logWriter,
Expand Down Expand Up @@ -412,5 +423,9 @@ func (w *WAF) Validate() error {
return errors.New("argument limit should be bigger than 0")
}

if w.RequestBodyJsonDepthLimit <= 0 {
return errors.New("request body json depth limit should be bigger than 0")
}

return nil
}
18 changes: 18 additions & 0 deletions internal/seclang/directives.go
Original file line number Diff line number Diff line change
Expand Up @@ -271,6 +271,24 @@ func directiveSecRequestBodyAccess(options *DirectiveOptions) error {
return nil
}

// Description: Configures the maximum JSON recursion depth limit Coraza will accept.
// Default: 1024
// Syntax: SecRequestBodyJsonDepthLimit [LIMIT]
// ---
// Anything over the limit will generate a REQBODY_ERROR in the JSON body processor.
func directiveSecRequestBodyJsonDepthLimit(options *DirectiveOptions) error {
if len(options.Opts) == 0 {
return errEmptyOptions
}

limit, err := strconv.Atoi(options.Opts)
if err != nil {
return err
}
options.WAF.RequestBodyJsonDepthLimit = limit
return nil
}

// Description: Configures the rules engine.
// Syntax: SecRuleEngine On|Off|DetectionOnly
// Default: Off
Expand Down
2 changes: 2 additions & 0 deletions internal/seclang/directivesmap.gen.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading