Skip to content

Commit

Permalink
add timeline and several fixes
Browse files Browse the repository at this point in the history
  • Loading branch information
@skonto
skonto committed Nov 14, 2023
1 parent f920642 commit 02b5cf6
Show file tree
Hide file tree
Showing 2 changed files with 155 additions and 53 deletions.
208 changes: 155 additions & 53 deletions blog/docs/articles/demystifying-activator-on-path.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,19 @@

**Author: [Stavros Kontopoulos](https://twitter.com/s_kontopoulos), Principal Software Engineer @ RedHat**

**Date: 2023-10-11**
**Date: 2023-11-14**

_In this blog post you will learn how to recognize when activator is on the data path and what it triggers that behavior._

The activator acts as a component on the data path to enable traffic buffering when a service is scaled-to-zero.
One lesser known feature of activator is, that it can act as a request buffer that handles back-pressure with the goal to not overload a Knative service.
For this, a Knative service can define how much traffic it can handle using [annotations](https://knative.dev/docs/serving/autoscaling/autoscaling-targets/#configuring-targets).
The autoscaler component will use this information to calculate the amount of pods needed to handle the incoming traffic for a specific Knative service
When serving traffic a knative service can operate in two modes: proxy mode and serve mode.

In detail when serving traffic, a Knative service can operate in two modes: proxy mode and serve mode.
When in proxy mode, Activator is on the data path (which means the incoming requests are routed through the Activator component), and it will stay on the path until certain conditions (more on this later) are met.
When these conditions are met, Activator is removed from the data path, and the service transitions to serve mode.
However, it was not always like that when a service scales from/to zero, the activator is added by default to the data path.
For example, when a service scales from/to zero, the activator is added by default to the data path.
This default setting often confuses users for reasons we will see next as it is possible that activator will not be removed unless enough capacity is available.
This is intended as one of the Activator's roles is to offer backpressure capabilities so that a Knative service is not overloaded by incoming traffic.

Expand All @@ -22,9 +23,9 @@ This is intended as one of the Activator's roles is to offer backpressure capabi
The default pod autoscaler in Knative (KPA) is a sophisticated algorithm that uses metrics from pods to make scaling decisions.
Let's see in detail what happens when a new Knative service is created.

Once the user creates a new service the corresponding Knative reconciler creates a Knative Configuration and a Knative Route for that service.
Once the user creates a new service the corresponding Knative reconciler creates a Knative `Configuration` and a Knative `Route` for that service.
Then the Configuration reconciler creates a `Revision` resource and the reconciler for the latter will create a Pod Autoscaler(PA) resource along with the K8s deployment for the service.
The Route reconciler will create the ingress resource that will be picked up by the Knative net-* components responsible for managing traffic locally in the cluster and externally to the cluster.
The Route reconciler will create the `Ingress` resource that will be picked up by the Knative net-* components responsible for managing traffic locally in the cluster and externally to the cluster.

Now, the creation of the PA triggers the KPA reconciler which goes through certain steps in order to setup an autoscaling configuration for the revision:

Expand All @@ -36,18 +37,49 @@ sets up a pod scaler via the multi-scaler component. The pod scaler calculates a
- calls a scale method that decides the number of wanted pods and also updates the K8s raw deployment that corresponds to the revision.

- creates/updates a ServerlessService (SKS) that holds info about the operation mode (proxy or serve) and stores the number of activators that should be used in proxy mode.
The number of activators depends on the capacity that needs to be covered.
That SKS create/update event triggers a reconciliation for the SKS from its specific controller that creates the required public and private K8s services so traffic can be routed to the K8s deployment.
Also in proxy mode that controller will pick up the number of activators and configure an equal number of endpoints for the revision's [public service](https://github.com/knative/serving/blob/main/docs/scaling/SYSTEM.md#data-flow-examples).
This in combination with the networking setup done by the net-* components is the end-to-end networking setup that needs to happen for a ksvc to be ready to serve traffic.
The number of activators specified in the SKS depends on the capacity that needs to be covered.

- updates the PA and reports the active and wanted pods in its status

!!! note

The SKS create/update event above triggers a reconciliation for the SKS from its specific controller that creates the required public and private K8s services so traffic can be routed to the raw K8s deployment.
Also in proxy mode that SKS controller will pick up the number of activators and configure an equal number of endpoints for the revision's [public service](https://github.com/knative/serving/blob/main/docs/scaling/SYSTEM.md#data-flow-examples).
This in combination with the networking setup done by the net-* components (driven by the Ingress resource) is roughly the end-to-end networking setup that needs to happen for a ksvc to be ready to serve traffic.


## Capacity and Operation Modes in Practice

As described earlier Activator will be removed if enough capacity is available and there is an invariant that needs to hold, that is EBC (excess burst capacity)>0, where EBC = TotalCapacity - ObservedInPanicMode - TargetBurstCapacity(TBC).
As described earlier Activator will be removed if enough capacity is available. Let's see how this capacity is calculated
but before that let's introduce two concepts: the `panic` and `stable` windows.
The `panic` window is the time duration where we don't have enough capacity to serve the traffic. It happens usually with a sudden burst of traffic.
The condition that describes when to enter the panic mode and start the panic window is:

```
dppc/readyPodsCount >= spec.PanicThreshold
```
where
```
dppc := math.Ceil(observedPanicValue / spec.TargetValue)
```
The target value is the utilization in terms of concurrency and that is 0.7*(revision_target).
0.7 is the utilization factor for each replica and when reached we need to scale out.

**Note:** if the KPA metrics Requests Per Second(RPS) is used then the utilization factor is 0.75.

The `observedPanicValue` is the calculated average value seen during the panic window for the concurrency metric.
The panic threshold is configurable (default 2) and expresses the ratio of desired versus available pods.

Let's see an example of a service that has a target concurrency of 10 and tbc=10:
After we enter panic mode in order to exit we need to have enough capacity for a period that is equal to the stable window size.

To quantify the idea of enough capacity to deal with bursts of traffic we introduce the notion of the Excess Burst Capacity(EBC) that needs to be >=0.
It is defined as:

```
EBC = TotalCapacity - ObservedPanicValue - TargetBurstCapacity(TBC).
```

Let's see an example of how these are calculated in practice. Here is a service that has a target concurrency of 10 and tbc=10:

```
apiVersion: serving.knative.dev/v1
Expand All @@ -65,7 +97,16 @@ spec:
- image: ghcr.io/knative/autoscale-go:latest
```

Initially when the ksvc was deployed there was no traffic and one pod is created by default for verification reasons.
The scenario we are going to demonstrate is deploye the ksvc, let it scale down to zero and then send traffic for 10 minutes.
We then collect the logs from the autoscaler and visualize the EBC, ready pods and panic mode over time.
The graphs are shown next.

![Excess burst capacity over time](/blog/articles/images/ebc.png)
![Ready pods over time](/blog/articles/images/readypods.png)
![Panic mode over time](/blog/articles/images/panic.png)


Let's describe inde tail what we see above. Initially when the ksvc is deployed there is no traffic and one pod is created by default for verification reasons.

Until the pod is up we have:
```bash
Expand All @@ -85,11 +126,16 @@ NAME MODE ACTIVATORS SERVICENAME PRIVATESERVICENAM
autoscale-go-00001 Serve 2 autoscale-go-00001 autoscale-go-00001-private True
```

The reason why we are in Serve mode is that because EBC=0. In the logs we get:
The reason why we are in Serve mode is that because EBC=0. When you enable debug logs, in the logs you get:


```bash
{"severity":"DEBUG","timestamp":"2023-10-10T15:29:37.241575214Z","logger":"autoscaler","caller":"scaling/autoscaler.go:286","message":"PodCount=1 Total1PodCapacity=10.000 ObsStableValue=0.000 ObsPanicValue=0.000 TargetBC=10.000 ExcessBC=0.000","commit":"f1617ef","knative.dev/key":"default/autoscale-go-00001"}
```
"severity": "DEBUG",
"timestamp": "2023-10-10T15:29:37.241575214Z",
"logger": "autoscaler",
"caller": "scaling/autoscaler.go:286",
"message": "PodCount=1 Total1PodCapacity=10.000 ObsStableValue=0.000 ObsPanicValue=0.000 TargetBC=10.000 ExcessBC=0.000",
"commit": "f1617ef",
"knative.dev/key": "default/autoscale-go-00001"
```

EBC = 10 - 0 - 10 = 0
Expand All @@ -104,12 +150,6 @@ NAME MODE ACTIVATORS SERVICENAME PRIVATESERVICENAM
autoscale-go-00001 Proxy 2 autoscale-go-00001 autoscale-go-00001-private Unknown NoHealthyBackends
```

When you enable debug logs, you can see this also in the autoscaler logs. In this case we go directly to:

```
{"severity":"DEBUG","timestamp":"2023-10-10T15:29:37.241523364Z","logger":"autoscaler","caller":"scaling/autoscaler.go:247","message":"Operating in stable mode.","commit":"f1617ef","knative.dev/key":"default/autoscale-go-00001"}
```

Let's send some traffic (experiment was run on Minikube):

```bash
Expand All @@ -119,29 +159,54 @@ hey -z 600s -c 20 -q 1 -host "autoscale-go.default.example.com" "http://192.168.
Initially activator when get a request in it sends stats to the autoscaler which tries to scale from zero based on some initial scale (default 1):

```
{"severity":"DEBUG","timestamp":"2023-10-10T15:32:56.178498172Z","logger":"autoscaler.stats-websocket-server","caller":"statserver/server.go:193","message":"Received stat message: {Key:default/autoscale-go-00001 Stat:{PodName:activator-59dff6d45c-9rdxh AverageConcurrentRequests:1 AverageProxiedConcurrentRequests:0 RequestCount:1 ProxiedRequestCount:0 ProcessUptime:0 Timestamp:0}}","commit":"f1617ef","address":":8080"}
{"severity":"DEBUG","timestamp":"2023-10-10T15:32:56.178733422Z","logger":"autoscaler","caller":"statforwarder/processor.go:64","message":"Accept stat as owner of bucket autoscaler-bucket-00-of-01","commit":"f1617ef","bucket":"autoscaler-bucket-00-of-01","knative.dev/key":"default/autoscale-go-00001"}
"severity": "DEBUG",
"timestamp": "2023-10-10T15:32:56.178498172Z",
"logger": "autoscaler.stats-websocket-server",
"caller": "statserver/server.go:193",
"message": "Received stat message: {Key:default/autoscale-go-00001 Stat:{PodName:activator-59dff6d45c-9rdxh AverageConcurrentRequests:1 AverageProxiedConcurrentRequests:0 RequestCount:1 ProxiedRequestCount:0 ProcessUptime:0 Timestamp:0}}",
"commit": "f1617ef",
"address": ":8080"
"severity": "DEBUG",
"timestamp": "2023-10-10T15:32:56.178733422Z",
"logger": "autoscaler",
"caller": "statforwarder/processor.go:64",
"message": "Accept stat as owner of bucket autoscaler-bucket-00-of-01",
"commit": "f1617ef",
"bucket": "autoscaler-bucket-00-of-01",
"knative.dev/key": "default/autoscale-go-00001"
```

The autoscaler enters panic mode since we don't have enough capacity, EBS is 10*0 -1 -10 = -11
```
{"severity":"DEBUG","timestamp":"2023-10-10T15:32:56.178920551Z","logger":"autoscaler","caller":"scaling/autoscaler.go:286","message":"PodCount=0 Total1PodCapacity=10.000 ObsStableValue=1.000 ObsPanicValue=1.000 TargetBC=10.000 ExcessBC=-11.000","commit":"f1617ef","knative.dev/key":"default/autoscale-go-00001"}
```
"severity": "DEBUG",
"timestamp": "2023-10-10T15:32:56.178920551Z",
"logger": "autoscaler",
"caller": "scaling/autoscaler.go:286",
"message": "PodCount=0 Total1PodCapacity=10.000 ObsStableValue=1.000 ObsPanicValue=1.000 TargetBC=10.000 ExcessBC=-11.000",
"commit": "f1617ef",
"knative.dev/key": "default/autoscale-go-00001"
"severity": "INFO",
"timestamp": "2023-10-10T15:32:57.24099875Z",
"logger": "autoscaler",
"caller": "scaling/autoscaler.go:215",
"message": "PANICKING.",
"commit": "f1617ef",
"knative.dev/key": "default/autoscale-go-00001"
```
Later on as traffic continues we get proper statistics from activator closer to the rate:


```
{
"severity":"DEBUG",
"timestamp":"2023-10-10T15:32:56.949001622Z",
"logger":"autoscaler.stats-websocket-server",
"caller":"statserver/server.go:193",
"message":"Received stat message: {Key:default/autoscale-go-00001 Stat:{PodName:activator-59dff6d45c-9rdxh AverageConcurrentRequests:18.873756322609804 AverageProxiedConcurrentRequests:0 RequestCount:19 ProxiedRequestCount:0 ProcessUptime:0 Timestamp:0}}",
"commit":"f1617ef",
"address":":8080"
}
{
"severity":"INFO",
"timestamp":"2023-10-10T15:32:56.432854252Z",
"logger":"autoscaler",
Expand All @@ -152,59 +217,96 @@ Later on as traffic continues we get proper statistics from activator closer to
"knative.dev/kind":"autoscaling.internal.knative.dev.PodAutoscaler",
"knative.dev/traceid":"7988492e-eea3-4d19-bf5a-8762cf5ff8eb",
"knative.dev/key":"default/autoscale-go-00001"
}
{
"severity":"DEBUG",
"timestamp":"2023-10-10T15:32:57.241052566Z",
"logger":"autoscaler",
"caller":"scaling/autoscaler.go:286",
"message":"PodCount=0 Total1PodCapacity=10.000 ObsStableValue=19.874 ObsPanicValue=19.874 TargetBC=10.000 ExcessBC=-30.000",
"commit":"f1617ef",
"knative.dev/key":"default/autoscale-go-00001"
}
```

Since the pod is not up yet: EBS = 0*10 - floor(19.874) - 10 = -30


Given the new statistics kpa decides to scale to 3 pods.
Given the new statistics kpa decides to scale to 3 pods at some point.

```
{"severity":"INFO","timestamp":"2023-10-10T15:32:57.241421042Z","logger":"autoscaler","caller":"kpa/scaler.go:370","message":"Scaling from 1 to 3","commit":"f1617ef","knative.dev/controller":"knative.dev.serving.pkg.reconciler.autoscaling.kpa.Reconciler","knative.dev/kind":"autoscaling.internal.knative.dev.PodAutoscaler","knative.dev/traceid":"6dcf87c9-15d8-41d3-95ae-5ca9b3d90705","knative.dev/key":"default/autoscale-go-00001"}
"severity": "INFO",
"timestamp": "2023-10-10T15:32:57.241421042Z",
"logger": "autoscaler",
"caller": "kpa/scaler.go:370",
"message": "Scaling from 1 to 3",
"commit": "f1617ef",
"knative.dev/controller": "knative.dev.serving.pkg.reconciler.autoscaling.kpa.Reconciler",
"knative.dev/kind": "autoscaling.internal.knative.dev.PodAutoscaler",
"knative.dev/traceid": "6dcf87c9-15d8-41d3-95ae-5ca9b3d90705",
"knative.dev/key": "default/autoscale-go-00001"
```

But let's see why is this is the case. The log above comes from the multi-scaler which reports a scaled result that contains EBS as reported above and a desired pod count for different windows.

Roughly the final desired number is (there is more logic that covers corner cases and checking against min/max scale limits):
Roughly the final desired number is (there is more logic that covers corner cases and checking against min/max scale limits)
derived from the dppc we saw earlier.

In this case the target value is 0.7*10=10. So we have for example for the panic window: dppc=ceil(19.874/7)=3

As metrics get stabilized and revision is scaled enough we have:

```
dspc := math.Ceil(observedStableValue / spec.TargetValue)
dppc := math.Ceil(observedPanicValue / spec.TargetValue)
"severity": "INFO",
"timestamp": "2023-10-10T15:33:01.320912032Z",
"logger": "autoscaler",
"caller": "kpa/kpa.go:158",
"message": "SKS should be in Serve mode: want = 3, ebc = 0, #act's = 2 PA Inactive? = false",
"commit": "f1617ef",
"knative.dev/controller": "knative.dev.serving.pkg.reconciler.autoscaling.kpa.Reconciler",
"knative.dev/kind": "autoscaling.internal.knative.dev.PodAutoscaler",
"knative.dev/traceid": "f0d22038-130a-4560-bd67-2751ecf3975d",
"knative.dev/key": "default/autoscale-go-00001"
"severity": "DEBUG",
"timestamp": "2023-10-10T15:33:03.24101879Z",
"logger": "autoscaler",
"caller": "scaling/autoscaler.go:286",
"message": "PodCount=3 Total1PodCapacity=10.000 ObsStableValue=16.976 ObsPanicValue=15.792 TargetBC=10.000 ExcessBC=4.000",
"commit": "f1617ef",
"knative.dev/key": "default/autoscale-go-00001"
```

EBS = 3*10 - floor(15.792) - 10 = 4

The target value is the utilization in terms of concurrency and that is is 0.7*(revision_target).
In this case this is 7. So we have for example for the panic window: ceil(19.874/7)=3
Then when we reach the required pod count and metrics are stable we get EBC=3*10 - floor(19.968) - 10=0:

**Note:** if RPS is used then the utilization factor is 0.75.
```
"severity": "DEBUG",
"timestamp": "2023-10-10T15:33:59.24118625Z",
"logger": "autoscaler",
"caller": "scaling/autoscaler.go:286",
"message": "PodCount=3 Total1PodCapacity=10.000 ObsStableValue=19.602 ObsPanicValue=19.968 TargetBC=10.000 ExcessBC=0.000",
"commit": "f1617ef",
"knative.dev/key": "default/autoscale-go-00001"
```

Later on when revision is scaled we have:
A few seconds later, one minute after we get in panicking mode we get to stable mode (un-panicking):

```
{"severity":"INFO","timestamp":"2023-10-10T15:33:01.320912032Z","logger":"autoscaler","caller":"kpa/kpa.go:158","message":"SKS should be in Serve mode: want = 3, ebc = 0, #act's = 2 PA Inactive? = false","commit":"f1617ef","knative.dev/controller":"knative.dev.serving.pkg.reconciler.autoscaling.kpa.Reconciler","knative.dev/kind":"autoscaling.internal.knative.dev.PodAutoscaler","knative.dev/traceid":"f0d22038-130a-4560-bd67-2751ecf3975d","knative.dev/key":"default/autoscale-go-00001"}
{"severity":"DEBUG","timestamp":"2023-10-10T15:33:03.24101879Z","logger":"autoscaler","caller":"scaling/autoscaler.go:286","message":"PodCount=3 Total1PodCapacity=10.000 ObsStableValue=16.976 ObsPanicValue=15.792 TargetBC=10.000 ExcessBC=4.000","commit":"f1617ef","knative.dev/key":"default/autoscale-go-00001"}
"severity": "INFO",
"timestamp": "2023-10-10T15:34:01.240916706Z",
"logger": "autoscaler",
"caller": "scaling/autoscaler.go:223",
"message": "Un-panicking.",
"commit": "f1617ef",
"knative.dev/key": "default/autoscale-go-00001"
```

EBS = 3*10 - floor(15.792) - 10 = 4
The sks also transitions to Serve mode as we have enough capacity until traffic stops and deployment is scaled back to zero (activator is removed from path).
For the experiment above since we have stable traffic for almost 10 minutes we don't observe any changes as soon as we have enough pods ready.
Note that when traffic goes down and until we adjust the pod count, for some short period of time, we have more ebc than we need.

Later on the sks transitions to Serve mode as we have enough capacity until traffic stops and deployment is scaled back to zero.
The major events are also depected in the timeline bellow:

The above are shown visually next with graphs describing ebc, ready pods and
![Excess burst capacity over time](/blog/articles/images/ebc.png)
![Ready pods over time](/blog/articles/images/readypods.png)
![Panic mode over time](/blog/articles/images/panic.png)
![timeline](/blog/articles/images/timeline.png)

### Conclusion

Expand Down
Binary file added blog/docs/articles/images/timeline.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 02b5cf6

Please sign in to comment.