Add option to specify URL Prefixes and different listener for expvar

 * Add config option `prefix` to specify URL Prefixes
 * Add config option `expvar` to specify if expvars are exported over HTTP and on what address
 * Add ability to enabled pprof handlers (see `expvar` structure)
 * Update Changelog and Readme

CHANGES.md

@@ -13,8 +13,9 @@ Changes
 
 CHANGELOG
 ---------
-**0.12.4 (WIP)**
+**0.12.4**
  - [Feature] Fuction Defines - allows to do custom metric aliases (thx to @lomik). See [doc/configuration.md](https://github.com/go-graphite/carbonapi/blob/master/doc/configuration.md#define) for config format
+ - [Improvement] New config options that allows to prefix all URLs and to enable /debug/vars on a separate address:port. See [docs/configuration.md](https://github.com/go-graphite/carbonapi/blob/master/doc/configuration.md#expvar) for more information
  - [Improvement] `/render` queries now returns tags in json (as graphite-web do)
  - [Improvement] groupByTags should now support all available aggregation functions (#410)
  - [Fix] Fix panic when using carbonapi\_proto\_v3 and doing tag-related queries (#407)

README.md

@@ -14,12 +14,15 @@ For requirements see **Requirements** section below.
 Installation
 ------------
 
-At this moment we are building packages for CentOS 6, CentOS 7, Ubuntu 14.04 and Ubuntu 16.04. Installation guides are available on packagecloud (see the links below).
+At this moment we are building packages for CentOS 6, CentOS 7, Debian 9, Debian 10, Ubuntu 14.04, Ubuntu 16.04 and Ubuntu 18.04. Installation guides are available on packagecloud (see the links below).
 
 Stable versions: [Stable repo](https://packagecloud.io/go-graphite/stable/install)
 
 Autobuilds (master, might be unstable): [Autobuild repo](https://packagecloud.io/go-graphite/autobuilds/install)
 
+Configuration guides: [docs/configuration.md](https://github.com/go-graphite/carbonapi/blob/master/doc/configuration.md) and [example config](https://github.com/go-graphite/carbonapi/blob/master/cmd/carbonapi/carbonapi.example.yaml).
+
+There are multiple example configurations available for different backends: [prometheus](https://github.com/go-graphite/carbonapi/blob/master/cmd/carbonapi/carbonapi.example.prometheus.yaml), [graphtie-clickhouse](https://github.com/go-graphite/carbonapi/blob/master/cmd/carbonapi/carbonapi.example.clickhouse.yaml), [go-carbon](https://github.com/go-graphite/carbonapi/blob/master/cmd/carbonapi/carbonapi.example.yaml)
 
 General information
 -------------------

cmd/carbonapi/carbonapi.example.yaml

@@ -9,6 +9,14 @@
 # or graphite-clickhouse's http url.
 # Listen address, should always include hostname or ip address and a port.
 listen: "localhost:8081"
+# Specify URL Prefix for all handlers
+prefix: ""
+# Specify if metrics are exported over HTTP and if they are available on the same address or not
+# pprofEnabled controls if extra HTTP Handlers to profile and debug application will be available
+expvar:
+  enabled: true
+  pprofEnabled: false
+  listen: ""
 # Allow extra charsets in metric names. By default only "Latin" is allowed
 # Please note that each unicodeRangeTables will slow down metric parsing a bit
 #   For list of supported tables, see: https://golang.org/src/unicode/tables.go?#L3437
@@ -29,6 +37,12 @@ headersToLog:
   - "X-Dashboard-Id"
   - "X-Grafana-Org-Id"
   - "X-Panel-Id"
+# Specify custom function aliases.
+#  This is example for alias "perMinute(metrics)" that will behave as "perSecond(metric)|scale(60)"
+define:
+  -
+    name: "perMinute"
+    template: "perSecond({{.argString}})|scale(60)"
 # Max concurrent requests to CarbonZipper
 concurency: 1000
 cache:

cmd/carbonapi/config/config.go

@@ -41,6 +41,12 @@ type Define struct {
 	Template string `mapstructure:"template"`
 }
 
+type ExpvarConfig struct {
+	Listen       string `mapstructure:"listen"`
+	Enabled      bool   `mapstructure:"enabled"`
+	PProfEnabled bool   `mapstructure:"pprofEnabled"`
+}
+
 type ConfigType struct {
 	ExtrapolateExperiment      bool               `mapstructure:"extrapolateExperiment"`
 	Logger                     []zapwriter.Config `mapstructure:"logger"`
@@ -68,6 +74,8 @@ type ConfigType struct {
 	HeadersToPass              []string           `mapstructure:"headersToPass"`
 	HeadersToLog               []string           `mapstructure:"headersToLog"`
 	Define                     []Define           `mapstructure:"define"`
+	Prefix                     string             `mapstructure:"prefix"`
+	Expvar                     ExpvarConfig       `mapstructure:"expvar"`
 
 	QueryCache cache.BytesCache `mapstructure:"-" json:"-"`
 	FindCache  cache.BytesCache `mapstructure:"-" json:"-"`
@@ -131,4 +139,10 @@ var Config = ConfigType{
 	},
 	ExpireDelaySec:             10 * 60,
 	GraphiteWeb09Compatibility: false,
+	Prefix:                     "",
+	Expvar: ExpvarConfig{
+		Listen:       "",
+		Enabled:      true,
+		PProfEnabled: false,
+	},
 }

cmd/carbonapi/config/init.go

@@ -91,7 +91,13 @@ func SetUpConfig(logger *zap.Logger, BuildVersion string) {
 			newStruct.ColorList = nil
 			newStruct.YDivisors = nil
 			sub := graphTemplatesViper.Sub(k)
-			sub.Unmarshal(&newStruct)
+			err = sub.Unmarshal(&newStruct)
+			if err != nil {
+				logger.Error("failed to parse graphTemplates config, settings will be ignored",
+					zap.String("graphTemplate_path", Config.GraphTemplates),
+					zap.Error(err),
+				)
+			}
 			if newStruct.ColorList == nil || len(newStruct.ColorList) == 0 {
 				newStruct.ColorList = make([]string, len(png.DefaultParams.ColorList))
 				copy(newStruct.ColorList, png.DefaultParams.ColorList)

cmd/carbonapi/http/init.go

@@ -1,34 +1,49 @@
 package http
 
 import (
+	"expvar"
 	"net/http"
+	"net/http/pprof"
 
 	"github.com/dgryski/httputil"
+	"github.com/go-graphite/carbonapi/cmd/carbonapi/config"
 	"github.com/go-graphite/carbonapi/util/ctx"
 )
 
 func InitHandlers(headersToPass, headersToLog []string) *http.ServeMux {
-	r := http.DefaultServeMux
-	r.HandleFunc("/render/", httputil.TrackConnections(httputil.TimeHandler(enrichContextWithHeaders(headersToPass, headersToLog, ctx.ParseCtx(renderHandler, ctx.HeaderUUIDAPI)), bucketRequestTimes)))
-	r.HandleFunc("/render", httputil.TrackConnections(httputil.TimeHandler(enrichContextWithHeaders(headersToPass, headersToLog, ctx.ParseCtx(renderHandler, ctx.HeaderUUIDAPI)), bucketRequestTimes)))
+	r := http.NewServeMux()
+	r.HandleFunc(config.Config.Prefix+"/render/", httputil.TrackConnections(httputil.TimeHandler(enrichContextWithHeaders(headersToPass, headersToLog, ctx.ParseCtx(renderHandler, ctx.HeaderUUIDAPI)), bucketRequestTimes)))
+	r.HandleFunc(config.Config.Prefix+"/render", httputil.TrackConnections(httputil.TimeHandler(enrichContextWithHeaders(headersToPass, headersToLog, ctx.ParseCtx(renderHandler, ctx.HeaderUUIDAPI)), bucketRequestTimes)))
 
-	r.HandleFunc("/metrics/find/", httputil.TrackConnections(httputil.TimeHandler(enrichContextWithHeaders(headersToPass, headersToLog, ctx.ParseCtx(findHandler, ctx.HeaderUUIDAPI)), bucketRequestTimes)))
-	r.HandleFunc("/metrics/find", httputil.TrackConnections(httputil.TimeHandler(enrichContextWithHeaders(headersToPass, headersToLog, ctx.ParseCtx(findHandler, ctx.HeaderUUIDAPI)), bucketRequestTimes)))
+	r.HandleFunc(config.Config.Prefix+"/metrics/find/", httputil.TrackConnections(httputil.TimeHandler(enrichContextWithHeaders(headersToPass, headersToLog, ctx.ParseCtx(findHandler, ctx.HeaderUUIDAPI)), bucketRequestTimes)))
+	r.HandleFunc(config.Config.Prefix+"/metrics/find", httputil.TrackConnections(httputil.TimeHandler(enrichContextWithHeaders(headersToPass, headersToLog, ctx.ParseCtx(findHandler, ctx.HeaderUUIDAPI)), bucketRequestTimes)))
 
-	r.HandleFunc("/info/", httputil.TrackConnections(httputil.TimeHandler(enrichContextWithHeaders(headersToPass, headersToLog, ctx.ParseCtx(infoHandler, ctx.HeaderUUIDAPI)), bucketRequestTimes)))
-	r.HandleFunc("/info", httputil.TrackConnections(httputil.TimeHandler(enrichContextWithHeaders(headersToPass, headersToLog, ctx.ParseCtx(infoHandler, ctx.HeaderUUIDAPI)), bucketRequestTimes)))
+	r.HandleFunc(config.Config.Prefix+"/info/", httputil.TrackConnections(httputil.TimeHandler(enrichContextWithHeaders(headersToPass, headersToLog, ctx.ParseCtx(infoHandler, ctx.HeaderUUIDAPI)), bucketRequestTimes)))
+	r.HandleFunc(config.Config.Prefix+"/info", httputil.TrackConnections(httputil.TimeHandler(enrichContextWithHeaders(headersToPass, headersToLog, ctx.ParseCtx(infoHandler, ctx.HeaderUUIDAPI)), bucketRequestTimes)))
 
-	r.HandleFunc("/lb_check", lbcheckHandler)
+	r.HandleFunc(config.Config.Prefix+"/lb_check", lbcheckHandler)
 
-	r.HandleFunc("/version", versionHandler)
-	r.HandleFunc("/version/", versionHandler)
+	r.HandleFunc(config.Config.Prefix+"/version", versionHandler)
+	r.HandleFunc(config.Config.Prefix+"/version/", versionHandler)
 
-	r.HandleFunc("/functions", enrichContextWithHeaders(headersToPass, headersToLog, functionsHandler))
-	r.HandleFunc("/functions/", enrichContextWithHeaders(headersToPass, headersToLog, functionsHandler))
+	r.HandleFunc(config.Config.Prefix+"/functions", enrichContextWithHeaders(headersToPass, headersToLog, functionsHandler))
+	r.HandleFunc(config.Config.Prefix+"/functions/", enrichContextWithHeaders(headersToPass, headersToLog, functionsHandler))
 
-	r.HandleFunc("/tags", enrichContextWithHeaders(headersToPass, headersToLog, tagHandler))
-	r.HandleFunc("/tags/", enrichContextWithHeaders(headersToPass, headersToLog, tagHandler))
+	r.HandleFunc(config.Config.Prefix+"/tags", enrichContextWithHeaders(headersToPass, headersToLog, tagHandler))
+	r.HandleFunc(config.Config.Prefix+"/tags/", enrichContextWithHeaders(headersToPass, headersToLog, tagHandler))
 
-	r.HandleFunc("/", enrichContextWithHeaders(headersToPass, headersToLog, usageHandler))
+	r.HandleFunc(config.Config.Prefix+"/", enrichContextWithHeaders(headersToPass, headersToLog, usageHandler))
+
+	if config.Config.Expvar.Enabled {
+		if config.Config.Expvar.Listen == "" || config.Config.Expvar.Listen == config.Config.Listen {
+			r.HandleFunc(config.Config.Prefix+"/debug/vars", expvar.Handler().ServeHTTP)
+			if config.Config.Expvar.PProfEnabled {
+				r.HandleFunc(config.Config.Prefix+"/debug/pprof/heap", pprof.Index)
+				r.HandleFunc(config.Config.Prefix+"/debug/pprof/profile", pprof.Profile)
+				r.HandleFunc(config.Config.Prefix+"/debug/pprof/symbol", pprof.Symbol)
+				r.HandleFunc(config.Config.Prefix+"/debug/pprof/trace", pprof.Trace)
+			}
+		}
+	}
 	return r
 }

cmd/carbonapi/main.go

@@ -1,10 +1,13 @@
 package main
 
 import (
+	"expvar"
 	"flag"
 	"log"
 	"net/http"
+	"net/http/pprof"
 	_ "net/http/pprof"
+	"sync"
 
 	"github.com/facebookgo/grace/gracehttp"
 	"github.com/go-graphite/carbonapi/cmd/carbonapi/config"
@@ -46,14 +49,60 @@ func main() {
 	handler = handlers.CORS()(handler)
 	handler = handlers.ProxyHeaders(handler)
 
-	err = gracehttp.Serve(&http.Server{
-		Addr:    config.Config.Listen,
-		Handler: handler,
-	})
+	wg := sync.WaitGroup{}
+	if config.Config.Expvar.Enabled {
+		if config.Config.Expvar.Listen != "" || config.Config.Expvar.Listen != config.Config.Listen {
+			r := http.NewServeMux()
+			r.HandleFunc(config.Config.Prefix+"/debug/vars", expvar.Handler().ServeHTTP)
+			if config.Config.Expvar.PProfEnabled {
+				r.HandleFunc(config.Config.Prefix+"/debug/pprof/heap", pprof.Index)
+				r.HandleFunc(config.Config.Prefix+"/debug/pprof/profile", pprof.Profile)
+				r.HandleFunc(config.Config.Prefix+"/debug/pprof/symbol", pprof.Symbol)
+				r.HandleFunc(config.Config.Prefix+"/debug/pprof/trace", pprof.Trace)
+			}
 
-	if err != nil {
-		logger.Fatal("gracehttp failed",
-			zap.Error(err),
-		)
+			handler := handlers.CompressHandler(r)
+			handler = handlers.CORS()(handler)
+			handler = handlers.ProxyHeaders(handler)
+
+			logger.Info("expvar handler will listen on a separate address/port",
+				zap.String("expvar_listen", config.Config.Expvar.Listen),
+				zap.Bool("pprof_enabled", config.Config.Expvar.PProfEnabled),
+			)
+
+			wg.Add(1)
+			go func() {
+				err = gracehttp.Serve(&http.Server{
+					Addr:    config.Config.Expvar.Listen,
+					Handler: handler,
+				})
+
+				if err != nil {
+					logger.Fatal("gracehttp failed",
+						zap.Error(err),
+					)
+				}
+
+				wg.Done()
+			}()
+		}
 	}
+
+	wg.Add(1)
+	go func() {
+		err = gracehttp.Serve(&http.Server{
+			Addr:    config.Config.Listen,
+			Handler: handler,
+		})
+
+		if err != nil {
+			logger.Fatal("gracehttp failed",
+				zap.Error(err),
+			)
+		}
+
+		wg.Done()
+	}()
+
+	wg.Wait()
 }

doc/configuration.md

@@ -4,44 +4,48 @@ Table of Contents
 * [General configuration for carbonapi](#general-configuration-for-carbonapi)
   * [listen](#listen)
     * [Example:](#example)
-  * [headersToPass](#headerstopass)
+  * [prefix](#prefix)
     * [Example:](#example-1)
-  * [headersToLog](#headerstolog)
+  * [headersToPass](#headerstopass)
     * [Example:](#example-2)
-  * [headersToLog](#define)
+  * [headersToLog](#headerstolog)
     * [Example:](#example-3)
+  * [headersToLog](#define)
+    * [Example:](#example-4)
   * [unicodeRangeTables](#unicoderangetables)
-    * [Example](#example-4)
-  * [cache](#cache)
     * [Example](#example-5)
-  * [cpus](#cpus)
+  * [cache](#cache)
     * [Example](#example-6)
-  * [tz](#tz)
+  * [cpus](#cpus)
     * [Example](#example-7)
-  * [functionsConfig](#functionsconfig)
+  * [tz](#tz)
     * [Example](#example-8)
-  * [graphite](#graphite)
+  * [functionsConfig](#functionsconfig)
     * [Example](#example-9)
-  * [pidFile](#pidfile)
+  * [graphite](#graphite)
     * [Example](#example-10)
-  * [graphTemplates](#graphtemplates)
+  * [pidFile](#pidfile)
     * [Example](#example-11)
-  * [defaultColors](#defaultcolors)
+  * [graphTemplates](#graphtemplates)
     * [Example](#example-12)
-  * [logger](#logger)
+  * [defaultColors](#defaultcolors)
     * [Example](#example-13)
+  * [expvar](#expvar)
+    * [Example](#example-14)
+  * [logger](#logger)
+    * [Example](#example-15)
 * [Carbonzipper configuration](#carbonzipper-configuration)
   * [concurency](#concurency)
-    * [Example](#example-14)
+    * [Example](#example-16)
   * [maxBatchSize](#maxbatchsize)
-    * [Example](#example-15)
+    * [Example](#example-17)
   * [idleConnections](#idleconnections)
   * [upstreams](#upstreams)
-    * [Example](#example-16)
+    * [Example](#example-18)
       * [For go\-carbon and prometheus](#for-go-carbon-and-prometheus)
       * [For graphite\-clickhouse](#for-graphite-clickhouse)
   * [expireDelaySec](#expiredelaysec)
-    * [Example](#example-17)
+    * [Example](#example-19)
 
 # General configuration for carbonapi
 
@@ -65,6 +69,20 @@ This will make it available on all IPv4 addresses, port 8080:
 listen: "0.0.0.0:8080"
 ```
 
+***
+## prefix
+
+Specify prefix for all URLs. Might be useful when you cannot afford to listen on different port.
+
+Default: None
+
+### Example:
+This will make carbonapi handlers accessible on `/graphite`, e.x. `http://localhost:8080/render` will become `http://localhost:8080/graphite/render` 
+
+```yaml
+prefix: "graphite"
+```
+
 ***
 ## headersToPass
 
@@ -111,7 +129,7 @@ Defines are done by templating this custom aliases to known set of functions.
 
 Templating is done by utilizing golang text template language.
 
-Supported values:
+Supported variables:
  - argString - argument as a string. E.x. in query `myDefine(foo, bar, baz)`, argString will be `foo, bar, baz`
  - args - indexed array of arguments. E.x. in case of `myDefine(foo, bar)`, `index .args 0` will be first argument, `index .args 1` will be second
  - kwargs - key-value arguments (map). This is required to support cases like `myDefine(foo, bar=baz)`, in this case `index .args 0` will contain `foo`, and `index .kwargs "bar"` will contain `baz`
@@ -285,6 +303,33 @@ defaultColors:
       "darkblue": "002173"
 ```
 
+***
+## expvar
+
+Controls whether expvar (contains internal metrics, config, etc) is enabled and if it's accessible on a separate address:port.
+Also allows to enable pprof handlers (useful for profiling and debugging).
+
+Please note, that exposing pprof handlers to untrusted network is *dangerous* and might lead to data leak.
+
+Exposing expvars to untrusted network is not recommended as it might give 3rd party unnecessary amount of data about your infrastructure.
+
+### Example
+This describes current defaults: expvar enabled, pprof handlers disabled, listen on the same address-port as main application.
+```yaml
+expvar:
+      enabled: true
+      pprofEnabled: false
+      listen: ""
+```
+
+This is useful to enable debugging and to move all related handlers and add exposed only on localhost, port 7070.
+```yaml
+expvar:
+      enabled: true
+      pprofEnabled: true
+      listen: "localhost:7070"
+```
+
 ***
 ## logger