SOLR-16974: Ref-guide documentation

solr/solr-ref-guide/modules/deployment-guide/pages/circuit-breakers.adoc

@@ -32,8 +32,25 @@ Setting the `shards.tolerant=true` parameter on requests can help with graceful
 circuit breaker thresholds are reached on some nodes. See the <<shards.tolerant Parameter>> for details.
 
 == Circuit Breaker Configurations
-All circuit breaker configurations are listed as independent `<circuitBreaker>` entries in `solrconfig.xml` as shown below.
-A circuit breaker can register itself to trip for query requests and/or update requests. By default only search requests are affected. A user may register multiple circuit breakers of the same type with different thresholds for each request type.
+Circuit breakers can be configured globally for the entire node, or for each collection individually, or a combination. Per-collection circit breakers take precedence over global circuit breakers.
+
+A circuit breaker can register itself to trip for query requests and/or update requests. A user may register multiple circuit breakers of the same type with different thresholds for each request type.
+
+=== Global Circuit Breakers
+Circuit breakers can be configured globally using environment variables, e.g. in `solr.in.sh`, or system properties. The variables available are:
+
+[options="header"]
+|===
+|Name |Environment Variable Name |System Property Name
+|JVM Heap Usage |`SOLR_CIRCUITBREAKER_QUERY_MEM`, `SOLR_CIRCUITBREAKER_UPDATE_MEM` |`solr.circuitbreaker.query.mem`, `solr.circuitbreaker.update.mem`
+|System CPU Usage |`SOLR_CIRCUITBREAKER_QUERY_CPU`, `SOLR_CIRCUITBREAKER_UPDATE_CPU` |`solr.circuitbreaker.query.cpu`, `solr.circuitbreaker.update.cpu`
+|System Load Average |`SOLR_CIRCUITBREAKER_QUERY_LOADAVG`, `SOLR_CIRCUITBREAKER_UPDATE_LOADAVG` |`solr.circuitbreaker.query.loadavg`, `solr.circuitbreaker.update.loadavg`
+|===
+
+For example, you can enable a global CPU circuit breaker that rejects update requests when above 95% CPU load, by setting the following environment variable: `SOLR_CIRCUITBREAKER_UPDATE_CPU=95`.
+
+=== Per Collection Circuit Breakers
+Circuit breakers are configured as independent `<circuitBreaker>` entries in `solrconfig.xml` as shown in the below examples. By default, only search requests are affected.
 
 == Currently Supported Circuit Breakers
 
@@ -52,13 +69,20 @@ The main configuration for this circuit breaker is controlling the threshold per
 
 To enable and configure the JVM heap usage based circuit breaker, add the following:
 
+.Per collection in `solrconfig.xml`
 [source,xml]
 ----
 <circuitBreaker class="org.apache.solr.util.circuitbreaker.MemoryCircuitBreaker">
  <double name="threshold">75</double>
 </circuitBreaker>
 ----
 
+.Global in `solr.in.sh`
+[source,bash]
+----
+SOLR_CIRCUITBREAKER_QUERY_HEAP=75
+----
+
 The `threshold` is defined as a percentage of the max heap allocated to the JVM.
 
 For the circuit breaker configuration, a value of "0" maps to 0% usage and a value of "100" maps to 100% usage.
@@ -81,13 +105,20 @@ disabled and log an error message. An alternative can then be to use the <<syste
 
 To enable and configure the CPU utilization based circuit breaker:
 
+.Per collection in `solrconfig.xml`
 [source,xml]
 ----
 <circuitBreaker class="org.apache.solr.util.circuitbreaker.CPUCircuitBreaker">
  <double  name="threshold">75</double>
 </circuitBreaker>
 ----
 
+.Global in `solr.in.sh`
+[source,bash]
+----
+SOLR_CIRCUITBREAKER_QUERY_CPU=75
+----
+
 The triggering threshold is defined in percent CPU usage. A value of "0" maps to 0% usage
 and a value of "100" maps to 100% usage. The example above will trip when the CPU usage is
 equal to or greater than 75%.
@@ -103,13 +134,20 @@ https://en.wikipedia.org/wiki/Load_(computing)[Wikipedia page for Load],
 
 To enable and configure the CPU utilization based circuit breaker:
 
+.Per collection in `solrconfig.xml`
 [source,xml]
 ----
 <circuitBreaker class="org.apache.solr.util.circuitbreaker.LoadAverageCircuitBreaker">
  <double  name="threshold">8.0</double>
 </circuitBreaker>
 ----
 
+.Global in `solr.in.sh`
+[source,bash]
+----
+SOLR_CIRCUITBREAKER_QUERY_LOADAVG=75
+----
+
 The triggering threshold is a floating point number matching load average.
 The example circuit breaker above will trip when the load average is equal to or greater than 8.0.
 
@@ -118,6 +156,7 @@ The example circuit breaker above will trip when the load average is equal to or
 In this example we will prevent update requests above 80% CPU load, and prevent query requests above 95% CPU load. Supported request types are `query` and `update`.
 This would prevent expensive bulk updates from impacting search. Note also the support for short-form class name.
 
+.Per collection in `solrconfig.xml`
 [source,xml]
 ----
 <config>
@@ -137,6 +176,13 @@ This would prevent expensive bulk updates from impacting search. Note also the s
 </config>
 ----
 
+.Global in `solr.in.sh`
+[source,bash]
+----
+SOLR_CIRCUITBREAKER_QUERY_CPU=75
+SOLR_CIRCUITBREAKER_UPDATE_CPU=75
+----
+
 == Performance Considerations
 
 While JVM or CPU circuit breakers do not add any noticeable overhead per request, having too many circuit breakers checked for a single request can cause a performance overhead.

solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc

@@ -71,6 +71,9 @@ Due to changes in Lucene 9, that isn't possible any more.
 === Curator upgraded to 5.5.0 and requires Zookeeper 3.5.x or higher
 xref:upgrade-notes:major-changes-in-solr-9.html#solr-8-2[Solr 8.2 recommended using Zookeeper 3.5.5] and now with Curator 5.5.0 requires https://curator.apache.org/docs/breaking-changes/[Zookeeper 3.5.x or higher]. This primarily affects users of `hadoop-auth`, but usage of Curator could affect other parts of Solr.
 
+=== Global Circuit Breakers
+* Circuit breakers can now be configured globally, not only per collection. See xref:deployment-guide:circuit-breakers.adoc[Configuring Circuit Breakers] for more information.
+
 == Solr 9.4
 === The Built-In Config Sets
 * The build in ConfigSets (`_default` and `sample_techproducts_configs`), now use a default `autoSoftCommit` time of 3 seconds,