Author: Sergey Senozhatsky, 2017


MTRACE  
================================================================================  
  
0) What is mtrace  
  
mtrace is a shared library that intercepts and logs glibc/libstdc++ memory  
allocation/release calls. It's based on LD_PRELOAD mechanism.  
  
  
1) How to build  
  
	./autogen.sh  
	./configure --prefix=/path_to_mtrace_src/build  
	make  
	make install  
  
  
The libmtrace.so file will be in build/libs directory and parser tool  
will be in build/bin.  
  
You can specify an alternative --prefix path.  
  
  
2) How to use mtrace  
  
a simple way is to execute something like below  
  
	LD_PRELOAD=/path/libmtrace.so /usr/bin/application  
  
A more appropriate way is to use install_hook.sh script. It will setup  
the tracing and correctly pass all the env var/params to the application.  
  
Usage example:  
  
to install a hook:  
./install_hook.sh install /usr/apps/binary  
  
to uninstall a hook:  
./install_hook.sh uninstall /usr/apps/binary  
  
  
install_hook.sh also can handle an array of targets (applications to trace).  
For example  
./install_hook.sh install /usr/apps/foo /usr/apps/bar /usr/apps/xyz  
./install_hook.sh uninstall /usr/apps/foo /usr/apps/bar /usr/apps/xyz  
  
  
mtrace has several options that control its behavior. I recommend  
MTRACE_ALLOC_MINWMARK/MTRACE_ALLOC_MAXWMARK (see below).  
  
  
  
-  MTRACE_LOG_DIR=path  
  
   create tracing data files under provided directory, instead of stderr.  
   tracing file names have the following naming format:  
  
	path/mtrace-${APPLICATION_NAME}-${PID}  
  
Example:  
  
	*** Trace file name: `tailf /tmp/mtrace-parser-16233'  
  
- MTRACE_HUMAN_READABLE=1  
  
  by default mtrace produces a "compressed" output, which is hard to read.  
  
Example:  
  
[pid:25412] [ts:1489656373.964649] malloc(952) = 0x11ed310 [ts:0.000000]  
#26b43bf1#9#71  
#26b444c5#1#2f5  
#26b435e7#2#f7  
#26b42dbc#3#16c  
#4028ea#4#3a  
#26b39511#5#f1  
#40405a#7#2a  
  
  
  compressed output saves a lot of space and CPU time; compared to full  
  human-readable output. for instance, if app ABC does too many small  
  allocations then the tracing file can be several hundred megabytes  
  in size.  
  
  there is a special application (parser) that processes that output and  
  turns it into a human readable format (HTML report).  
  
- MTRACE_BACKTRACE_DEPTH=NUM  
  
  sometimes we don't care about the entire backtrace (which starts with  
  main()), but care about top NUM frames only. This options lets to set  
  such a limit. Only top NUM frames will be resolved and printed to the  
  output stream.  
  
- MTRACE_REPORTING_MODE=string  
  
  In order to save CPU time and space, mtrace has several options that  
  control output format. By default, mtrace outputs only backtraces of  
  call paths that increased the memory usage (we try to detect changes  
  in  /proc/[pid]/statm file). Memory free events are recorded as event  
  headers only, with out any backtraces, because we, basically, are not  
  so interested in memory free paths.  
  
  so default output (with MTRACE_HUMAN_READABLE=1) looks like this  
  
  
[pid:27429] [ts:1489656883.593174] malloc(80) = 0x184a9e0 [ts:0.000000]  
 # [<0x7fd07bf1>] _nl_intern_locale_data+0x71  
 # [<0x7fd084c5>] _nl_load_locale_from_archive+0x2f5  
 # [<0x7fd075e7>] _nl_find_locale+0xf7  
 # [<0x7fd06dbc>] setlocale+0x16c  
 # [<0x4028ea>] _init+0x3a  
 # [<0x7fcfd511>] __libc_start_main+0xf1  
 # [<0x40405a>] ?+0x40405a  
[pid:27429] [ts:1489656883.593410] malloc(192) = 0x184aa60 [ts:0.000000]  
 # [<0x7fd07bf1>] _nl_intern_locale_data+0x71  
 # [<0x7fd084c5>] _nl_load_locale_from_archive+0x2f5  
 # [<0x7fd075e7>] _nl_find_locale+0xf7  
 # [<0x7fd06dbc>] setlocale+0x16c  
 # [<0x4028ea>] _init+0x3a  
 # [<0x7fcfd511>] __libc_start_main+0xf1  
 # [<0x40405a>] ?+0x40405a  
[pid:27429] [ts:1489656883.602176] free(0x0) [ts:0.000000]  
[pid:27429] [ts:1489656883.602203] free(0x184fb10) [ts:0.000001]  
[pid:27429] [ts:1489656883.602237] free(0x1855d40) [ts:0.000001]  
  
note, there are backtraces for malloc(), but no backtraces for free().  
this saves a lot of CPU time on application that do millions of small  
allocations.  
  
  
- MTRACE_REPORTING_MODE=<string>  can be  
  --     alloc  
  
          report only memory allocation events (all of them) with backtraces.  
          e.g. mmap, malloc, etc.  
  
  --     atop  
  
          in this mode, mtrace will backtrace only events that have set new  
          allocation "record". so basically this will produce a list of  
          increasing in size allocations.  
  
          for example:  
  
          malloc(10)     -- backtrace, set "record" to 10  
          malloc(5)       -- no backtrace.  
          malloc(5)       -- no backtrace.   
          malloc(5)       -- no backtrace.   
          malloc(5)       -- no backtrace.   
          malloc(15)     -- backtrace. because 15 is greater than "10"  
  
         this allows to filter out a lot of noise from the output.  
  
   --    full  
  
         as the name suggests, this is "full" report mode. every event is  
         getting backtraced. it may be quite slow. many malloc() calls are  
         served by glibc internally, and don't result in new pages allocation.  
         so it's mostly noise in the output. for instance,  
  
      ptr = malloc(5)  
      free(ptr)  
      ptre = malloc(5)  
      free(ptr)  
      ....  
  
         these malloc-s will re-use the same page, which has already been  
         allocation and assigned to the process.  
  
  
  
Thus, I, personally, recommend another reporting mode, which is based on  
low/high memory allocation size filtering.  
  
  
- MTRACE_ALLOC_MINWMARK=<num>  
 num is in bytes by default, but can have memory prefix - 100M, 1G.  
  
  
In this mode mtrace will backtrace only memory allocation calls of sizes  
greater than given min watermark. E.g. with MTRACE_ALLOC_MINWMARK=300  
malloc(100) will not be backtraces, while malloc(400) will be.  
  
  
- MTRACE_ALLOC_MAXWMARK=<num>  
 num is in bytes by default, but can have memory prefix - 100M, 1G.  
  
In this mode mtrace will backtrace only memory allocation calls of sizes  
less than given max watermark. E.g. with MTRACE_ALLOC_MINWMARK=300  
malloc(100) will be backtraces, while malloc(400) will not be.  
  
  
It's is possible to provide both min and max watermarks simultaneously,  
so mtrace will backtrace only memory allocation calls of sizes from a  
given range.  
  
  
  
PARSER  
================================================================================  
  
parser is a simple application that decodes given "compressed" mtrace file  
and converts into a human readable HTML report.