Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] False positive with hooked_proc_fops event on GKE #2300

Closed
4 of 5 tasks
AlonZivony opened this issue Oct 25, 2022 · 1 comment · Fixed by #2303
Closed
4 of 5 tasks

[BUG] False positive with hooked_proc_fops event on GKE #2300

AlonZivony opened this issue Oct 25, 2022 · 1 comment · Fixed by #2303
Labels

Comments

@AlonZivony
Copy link
Contributor

AlonZivony commented Oct 25, 2022

Prerequisites

  • This affects latest released version.
  • This affects current development tree (origin/HEAD).
  • There isn't an issue describing the bug.

Select one OR another:

  • I'm going to create a PR to solve this (assign to yourself).
  • Someone else should solve this.

Bug description

TLDR

We have 2 problems:

  1. Not always we have all kallsyms symbols, which cause this event to create false-positive detections.
  2. We don't differ between initialized values to uninitialized, which cause in some systems to false positive on uninitialized addresses.

Full Description

Within the GKE environment, we receive many hooked_proc_fops events with supposedly "hooked fops".
Here is an example for a detection resulted from it:

*** Detection ***
Time: 2022-10-23T23:17:44Z
Signature ID: TRC-1020
Signature: File operations hooking on proc filesystem detected
Data: map[Hooked proc file operations:[{struct file_operations pointer hidden} {iterate hidden}]]
Command: kubelet
Hostname: gke-cluster-1-d

After some investigation, it seems that in the GKE environment all the rodata segment symbols are missing, in addition to other symbols.
Normally, the fops struct should be in the rodata segment, so it means we don't have the symbols for the fops structs.
Our event trigger is consisting of 2 checks:

  1. The symbols is not inside the text segment (iow between the __stext and __etext symbols).
  2. The symbol owner is not system (iow the symbols exist and is not owned by a kernel module).

Normally, the fops struct would not pass the first check, but because it is a symbols belonging to the system it will pass the second.
Because the symbols of the rodata segment are missing, the fops struct would not be considered to belong to the system by will be registered as hidden. This will cause the struct to be inserted to the hooked_symbols argument.
I will emphasize that the rodata segement symbols are not included by default in the kallsyms lists, only if the KALLSYMS_ALL configuration is selected.

What is also interesting is that the iterate fop is also considered "hooked" in this case.
After printing the address of the "hooked" fop, we noticed that it points to the address 0 because it is uninitialized (look at the proc_fd_fops struct for example).
In all systems this is the case, so it is interesting that only in this case it is considered "hooked".
After checking it, it seems that normally the address of 0 is registered as some symbol of the system, so it passes the second check of the event. In the GKE environment it is not the case, so the iterate fop also registered as "hooked".

Credit also to @OriGlassman who did the research on this error with me

Context

Relevant information about my setup:

  • Linux version: Ubuntu
  • Linux kernel version: 5.10.123+
  • Tracee version (or commit id of your tree): 0.9.1
@AlonZivony
Copy link
Contributor Author

Also found a vulnerability in libbpfgo in this field which should be fixed for reliability of this event - you can look at the issue I have opened.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant