Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for multiple vulnerability database repositories #7

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add support for multiple vulnerability database repositories #7

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 43 additions & 43 deletions README.md
View file
Open in desktop

Large diffs are not rendered by default.

90 changes: 45 additions & 45 deletions helm/harbor-scanner-trivy/README.md
View file
Open in desktop

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions helm/harbor-scanner-trivy/templates/statefulset.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,9 +78,9 @@ spec:
- name: "SCANNER_TRIVY_OFFLINE_SCAN"
value: {{ .Values.scanner.trivy.offlineScan | quote }}
- name: "SCANNER_TRIVY_DB_REPOSITORY"
value: {{ .Values.scanner.trivy.dbRepository | quote }}
value: {{ .Values.scanner.trivy.dbRepository | join \",\" | quote }}
- name: "SCANNER_TRIVY_JAVA_DB_REPOSITORY"
value: {{ .Values.scanner.trivy.javaDBRepository | quote }}
value: {{ .Values.scanner.trivy.javaDBRepository | join \",\" | quote }}
- name: "SCANNER_TRIVY_GITHUB_TOKEN"
valueFrom:
secretKeyRef:
Expand Down
17 changes: 12 additions & 5 deletions helm/harbor-scanner-trivy/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,14 +74,21 @@ scanner:
## If the flag is enabled you have to manually download the `trivy.db` file and mount it in the
## `/home/scanner/.cache/trivy/db/trivy.db` path (see `cacheDir`).
skipUpdate: false

## If the flag is enabled you have to manually download the `trivy-java.db` file and mount it in the
## `/home/scanner/.cache/trivy/java-db/trivy-java.db` path (see `cacheDir`).
skipJavaDBUpdate: false
# OCI repository to retrieve the trivy vulnerability database from
dbRepository: "ghcr.io/aquasecurity/trivy-db"
# OCI repository to retrieve the Java trivy vulnerability database from
javaDBRepository: "ghcr.io/aquasecurity/trivy-java-db"
# The dbRepository and javaDBRepository flags can take multiple values, improving reliability when downloading databases.
# Databases are downloaded in priority order until one is successful.
# An attempt to download from the next repository is only made if a temporary error is received (e.g. status 429 or 5xx).
#
# OCI repository(ies) to retrieve the trivy vulnerability database in order of priority
dbRepository:
- "mirror.gcr.io/aquasec/trivy-db"
- "ghcr.io/aquasecurity/trivy-db"
# OCI repository(ies) to retrieve the Java trivy vulnerability database in order of priority
javaDBRepository:
- "mirror.gcr.io/aquasec/trivy-java-db"
- "ghcr.io/aquasecurity/trivy-java-db"
# offlineScan the flag to disable external API requests to identify dependencies.
offlineScan: false
## gitHubToken the GitHub access token to download Trivy DB
Expand Down
4 changes: 2 additions & 2 deletions pkg/etc/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,8 @@ type Trivy struct {
IgnorePolicy string `env:"SCANNER_TRIVY_IGNORE_POLICY"`
SkipDBUpdate bool `env:"SCANNER_TRIVY_SKIP_UPDATE" envDefault:"false"`
SkipJavaDBUpdate bool `env:"SCANNER_TRIVY_SKIP_JAVA_DB_UPDATE" envDefault:"false"`
DBRepository string `env:"SCANNER_TRIVY_DB_REPOSITORY" envDefault:"ghcr.io/aquasecurity/trivy-db"`
JavaDBRepository string `env:"SCANNER_TRIVY_JAVA_DB_REPOSITORY" envDefault:"ghcr.io/aquasecurity/trivy-java-db"`
DBRepository []string `env:"SCANNER_TRIVY_DB_REPOSITORY"`
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It can remain as a string and we just document the value should be comma separated URIs?

Copy link
Author

@benji78 benji78 Dec 5, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could do that, but even though we are not changing or using the URIs separately right now, I believe it is clearer and more future proof to use an array of strings.
We could, for example, strip any spaces around the comma (may not be such a good practice though) or change from comma separated string to using one --db-repository or --java-db-repository per URI.

JavaDBRepository []string `env:"SCANNER_TRIVY_JAVA_DB_REPOSITORY"`
Comment on lines -37 to +38
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should I hardcode the default repositories? I was thinking using trivy's default might be better but if they change in future the README documentations would be wrong

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

two days ago a new trivy version has been released: v0.57.1
The default URLs have changed:
aquasecurity/trivy#7679

Copy link
Author

@benji78 benji78 Nov 20, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is indeed what I am referring to and the reason why I waited before creating this PR (and had it as a draft) because I needed to update the documentation.

OfflineScan bool `env:"SCANNER_TRIVY_OFFLINE_SCAN" envDefault:"false"`
GitHubToken string `env:"SCANNER_TRIVY_GITHUB_TOKEN"`
Insecure bool `env:"SCANNER_TRIVY_INSECURE" envDefault:"false"`
Expand Down
34 changes: 22 additions & 12 deletions pkg/etc/config_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,18 +148,20 @@ func TestGetConfig(t *testing.T) {
"SCANNER_API_SERVER_WRITE_TIMEOUT": "2m",
"SCANNER_API_SERVER_IDLE_TIMEOUT": "3m10s",

"SCANNER_TRIVY_CACHE_DIR": "/home/scanner/trivy-cache",
"SCANNER_TRIVY_REPORTS_DIR": "/home/scanner/trivy-reports",
"SCANNER_TRIVY_DEBUG_MODE": "true",
"SCANNER_TRIVY_VULN_TYPE": "os,library",
"SCANNER_TRIVY_SECURITY_CHECKS": "vuln",
"SCANNER_TRIVY_SEVERITY": "CRITICAL",
"SCANNER_TRIVY_IGNORE_UNFIXED": "true",
"SCANNER_TRIVY_INSECURE": "true",
"SCANNER_TRIVY_SKIP_UPDATE": "true",
"SCANNER_TRIVY_OFFLINE_SCAN": "true",
"SCANNER_TRIVY_GITHUB_TOKEN": "<GITHUB_TOKEN>",
"SCANNER_TRIVY_TIMEOUT": "15m30s",
"SCANNER_TRIVY_CACHE_DIR": "/home/scanner/trivy-cache",
"SCANNER_TRIVY_REPORTS_DIR": "/home/scanner/trivy-reports",
"SCANNER_TRIVY_DEBUG_MODE": "true",
"SCANNER_TRIVY_VULN_TYPE": "os,library",
"SCANNER_TRIVY_SECURITY_CHECKS": "vuln",
"SCANNER_TRIVY_SEVERITY": "CRITICAL",
"SCANNER_TRIVY_IGNORE_UNFIXED": "true",
"SCANNER_TRIVY_INSECURE": "true",
"SCANNER_TRIVY_SKIP_UPDATE": "true",
"SCANNER_TRIVY_DB_REPOSITORY": "mirror.gcr.io/aquasec/trivy-db,ghcr.io/aquasecurity/trivy-db",
"SCANNER_TRIVY_JAVA_DB_REPOSITORY": "mirror.gcr.io/aquasec/trivy-java-db,ghcr.io/aquasecurity/trivy-java-db",
"SCANNER_TRIVY_OFFLINE_SCAN": "true",
"SCANNER_TRIVY_GITHUB_TOKEN": "<GITHUB_TOKEN>",
"SCANNER_TRIVY_TIMEOUT": "15m30s",

"SCANNER_STORE_REDIS_NAMESPACE": "store.ns",
"SCANNER_STORE_REDIS_SCAN_JOB_TTL": "2h45m15s",
Expand Down Expand Up @@ -197,6 +199,14 @@ func TestGetConfig(t *testing.T) {
IgnoreUnfixed: true,
SkipDBUpdate: true,
SkipJavaDBUpdate: false,
DBRepository: []string{
"mirror.gcr.io/aquasec/trivy-db",
"ghcr.io/aquasecurity/trivy-db",
},
JavaDBRepository: []string{
"mirror.gcr.io/aquasec/trivy-java-db",
"ghcr.io/aquasecurity/trivy-java-db",
},
OfflineScan: true,
Insecure: true,
GitHubToken: "<GITHUB_TOKEN>",
Expand Down
12 changes: 6 additions & 6 deletions pkg/trivy/wrapper.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -199,16 +199,16 @@ func (w *wrapper) prepareScanCmd(target ScanTarget, outputFile string, opt ScanO
args = append(args, "--offline-scan")
}

if w.config.IgnorePolicy != "" {
args = append(args, "--ignore-policy", w.config.IgnorePolicy)
if len(w.config.DBRepository) > 0 {
args = append(args, "--db-repository", strings.Join(w.config.DBRepository, ","))
}

if w.config.DBRepository != "" {
args = append(args, "--db-repository", w.config.DBRepository)
if len(w.config.JavaDBRepository) > 0 {
args = append(args, "--java-db-repository", strings.Join(w.config.JavaDBRepository, ","))
}

if w.config.JavaDBRepository != "" {
args = append(args, "--java-db-repository", w.config.JavaDBRepository)
if w.config.IgnorePolicy != "" {
args = append(args, "--ignore-policy", w.config.IgnorePolicy)
}

if w.config.DebugMode {
Expand Down
8 changes: 4 additions & 4 deletions pkg/trivy/wrapper_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,8 +115,8 @@ func TestWrapper_Scan(t *testing.T) {
IgnorePolicy: "/home/scanner/opa/policy.rego",
SkipDBUpdate: true,
SkipJavaDBUpdate: true,
DBRepository: "ghcr.io/aquasecurity/trivy-db",
JavaDBRepository: "ghcr.io/aquasecurity/trivy-java-db",
DBRepository: []string{"mirror.gcr.io/aquasec/trivy-db", "ghcr.io/aquasecurity/trivy-db"},
JavaDBRepository: []string{"mirror.gcr.io/aquasec/trivy-java-db", "ghcr.io/aquasecurity/trivy-java-db"},
GitHubToken: "<github_token>",
Insecure: true,
Timeout: 5 * time.Minute,
Expand Down Expand Up @@ -156,9 +156,9 @@ func TestWrapper_Scan(t *testing.T) {
"--skip-db-update",
"--skip-java-db-update",
"--db-repository",
"ghcr.io/aquasecurity/trivy-db",
"mirror.gcr.io/aquasec/trivy-db,ghcr.io/aquasecurity/trivy-db",
"--java-db-repository",
"ghcr.io/aquasecurity/trivy-java-db",
"mirror.gcr.io/aquasec/trivy-java-db,ghcr.io/aquasecurity/trivy-java-db",
"--ignore-policy",
"/home/scanner/opa/policy.rego",
"--debug",
Expand Down
Loading