From d3ac121e6596b2956eab3c3a57b780e9a147b7a2 Mon Sep 17 00:00:00 2001 From: Michal Fupso Date: Mon, 25 Nov 2024 14:34:07 -0800 Subject: [PATCH] Add loadbalancer docs --- .../configuring/advertise-service-ips.mdx | 40 ++-- calico/networking/index.mdx | 1 + .../networking/ipam/service-loadbalancer.mdx | 209 ++++++++++++++++++ calico/reference/calicoctl/ipam/check.mdx | 21 +- calico/reference/resources/ippool.mdx | 25 +-- .../resources/kubecontrollersconfig.mdx | 10 + sidebars-calico.js | 1 + 7 files changed, 268 insertions(+), 39 deletions(-) create mode 100644 calico/networking/ipam/service-loadbalancer.mdx diff --git a/calico/networking/configuring/advertise-service-ips.mdx b/calico/networking/configuring/advertise-service-ips.mdx index f5860c256d..693e8d584d 100644 --- a/calico/networking/configuring/advertise-service-ips.mdx +++ b/calico/networking/configuring/advertise-service-ips.mdx @@ -199,28 +199,30 @@ The following steps will configure $[prodname] to advertise Service `status.Load For help see, [BGP configuration resource](../../reference/resources/bgpconfig.mdx). -Service LoadBalancer address allocation is outside the current scope of $[prodname], but can be implemented with an external controller. -You can build your own, or use a third-party implementation like the MetalLB project. +$[prodname] IPAM can manage address allocation for Service LoadBalancer with LoadBalancerController. To enable $[prodname] IPAM to start managing addresses to Service LoadBalancer set up an IPPool for LoadBalancer. -To install the MetalLB controller for allocating addresses, perform the following steps. - -1. Follow [the MetalLB documentation](https://metallb.universe.tf/installation/#installation-by-manifest) to install the `metallb-system/controller` resources. - - However, do not install the `metallb-system/speaker` component. The speaker component also attempts to establish BGP sessions on the node, and will conflict with Calico. +```bash +kubectl apply -f - < + diff --git a/calico/networking/ipam/service-loadbalancer.mdx b/calico/networking/ipam/service-loadbalancer.mdx new file mode 100644 index 0000000000..f488522832 --- /dev/null +++ b/calico/networking/ipam/service-loadbalancer.mdx @@ -0,0 +1,209 @@ +--- +description: Create LoadBalancer +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# LoadBalancer IP address management + +## Understanding $[prodname] LoadBalancer IP address management + +You might want to utilize Service LoadBalancer in your cluster to provide stable long-lasting IP address to access your deployed application. +$[prodname] can help you with providing and managing IP addresses to your Service LoadBalancers in your cluster. $[prodname] comes with LoadBalancer IPAM deployed as part of $[prodname] Kube-controllers. + +## Configure $[prodname] IP address management for Service LoadBalancer + +### Before you begin... + +Ensure that you have a cluster with $[prodname] installed, enabled bgp, external node configured as bgp peer and $[prodname] kube-controllers configured. + +For this example we will use LoadBalancer kube-controller assignIPs set to AllServices. To verify your LoadBalancer kube-controller settings you can check the output of the following command. + +```bash +kubectl get kubecontrollersconfiguration default -o "jsonpath={.spec.controllers.loadBalancer}" +``` + +If the output is not ```{"assignIPs":"AllServices"}``` you can patch your LoadBalancer kube-controller configuration by running the following command. + +```bash +kubectl patch kubecontrollersconfiguration default --patch '{"spec": {"controllers":{"loadBalancer":{"AssignIPs": "AllService"}}}}' +``` + +In this example we will use nginx deployment to connect our Service LoadBalancer to. If you wish to follow along you can create the deployment with the following configuration. + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: nginx +spec: + selector: + matchLabels: + app: nginx + replicas: 4 + template: + metadata: + labels: + app: nginx + spec: + containers: + - name: nginx + image: nginx + ports: + - containerPort: 80 +``` + +### Adding IP Pool for Service LoadBalancer + +For $[prodname] to be able to manage Service LoadBalancer IP addresses we need to create an IP Pool specifically to be used by LoadBalancer kube-controller. $[prodname] will then use that IP Pool to assign addresses from. + +Bellow is example IP Pool + +```yaml +apiVersion: projectcalico.org/v3 +kind: IPPool +metadata: + name: loadbalancerIPPool +spec: + cidr: 192.210.0.0/20 + blockSize: 24 + natOutgoing: true + disabled: false + assignmentMode: Automatic + allowedUses: + - LoadBalancer +``` + +:::note + +You can create multiple IP Pools with allowedUses LoadBalancer as long as there are no CIDR conflicts. By setting assignmentMode to Manual you can reserve the IP Pool for manual assignments only. Explore more about [manual assignment](#loadbalancer-ip-address-assignment) + +::: + +### Creating Service LoadBalancer + +We will create a Service LoadBalancer that connects to our nginx deployment. Calico IPAM will automatically detect that a new Service LoadBalancer has been created and assign an IP address from available IP Pool. + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: nginx-loadbalancer +spec: + selector: + app: nginx + ports: + - port: 80 + targetPort: 80 + name: default + type: LoadBalancer +``` + +After creating the Service LoadBalancer you can verify that IP address has been assigned. To do this you can run + +```bash +kubectl describe service nginx-loadbalancer +``` + +In the output you can see that LoadBalancer Ingress IP address has been assigned, take a note of this IP address. + +```bash +Name: nginx-loadbalancer +Namespace: default +Labels: +Annotations: +Selector: app=nginx +Type: LoadBalancer +IP Family Policy: SingleStack +IP Families: IPv4 +IP: 10.99.228.103 +IPs: 10.99.228.103 +LoadBalancer Ingress: 192.210.10.1 (VIP) +Port: default 80/TCP +TargetPort: 80/TCP +NodePort: default 32002/TCP +Endpoints: 192.168.165.134:80,192.168.21.198:80,192.168.238.133:80 + 1 more... +Session Affinity: None +External Traffic Policy: Cluster +Internal Traffic Policy: Cluster +Events: +``` + +### Advertise Service LoadBalancer IP address + +In order to be able to reach our application via the Service LoadBalancer Ingress IP address we have to advertise the LoadBalancer CIDR. You can find out more information at [Advertising Kubernetes service IP addresses](../../networking/configuring/advertise-service-ips.mdx) + +You can patch the bgp configuration to advertise the IP Pool CIDR + +```bash +calicoctl patch bgpconfig default --patch '{"spec": {"serviceLoadBalancerIPs": [{"cidr": "192.210.0.0/20"}]}}' +``` + +We should now be able to access our application behind the Service LoadBalancer by the IP address assigned by $[prodname] from our external bgp peer. + +Run the following command to check that our nginx deployment is reachable: + +```bash +curl 192.210.10.1 +``` + +If your output contains the following, you have successfully set up Service LoadBalancer with $[prodname] + +``` +Welcome to nginx! +If you see this page, the nginx web server is successfully installed and +working. Further configuration is required. +``` + +## LoadBalancer IP address assignment + +There are cases where you would like to be more specific about what IP address $[prodname] assigns to your LoadBalancer. With annotations, you can specify how IP address should be assigned to that Service. The annotations can be added and removed as needed during the lifetime of the Service. When you remove an annotation $[prodname] will check if the assigned IP is still valid and potentially assign a new one. + +### Specify IP Pool + +#### IPv4 Pool + +Annotate the Service with projectcalico.org/ipv4pools to assign IP address from the selected IP Pools. You can specify multiple IP Pools in this annotation, $[prodname] will pick an available IP address to assign. + +``` +"projectcalico.org/ipv4pools": '["loadBalancerIPv4Pool"]' +``` + +#### IPv6 Pool + +Annotate the Service with projectcalico.org/ipv6pools to assign IP address from the selected IP Pools. You can specify multiple IP Pools in this annotation, $[prodname] will pick an available IP address to assign. + +``` +"projectcalico.org/ipv6pools": '["loadBalancerIPv6Pool"]' +``` + +:::note + +In dual-stack environment you don't have to specify both annotations. If IP Pool was not specified for IP family, $[prodname] will automatically assign available IP address from available IP Pool + +::: + +### Specifying IP address + +Annotate the Service with projectcalico.org/loadBalancerIPs to assign specific IP address. The address must be available otherwise $[prodname] will not be able to assign the address. Currently you can only specify one IPv4 and one IPv6 address at the same time. + +``` +"projectcalico.org/loadBalancerIPs": '["x.x.x.x"]' +``` + +:::note + +If Service contains "projectcalico.org/loadBalancerIPs" annotation and either "projectcalico.org/ipv6pools" or "projectcalico.org/ipv4pools" are present, $[prodname] will favour projectcalico.org/loadBalancerIPs annotation and try to assign that IP address. There is no fall back to an IP Pool if the specified address is not available. + +::: + +### Manage LoadBalancer kube-controller assignment mode + +In certain cases you might not wish for $[prodname] to automatically assign IP addresses to your Service LoadBalancers. This can be useful in case you have multiple Service IPAM solutions in your cluster. +LoadBalancer kube-controller is able to operate in two distinct modes ```Automatic``` in which $[prodname] assigns IP address to each Service LoadBalancer in your cluster and ```RequestedServicesOnly``` in which $[prodname] only assigns IP addresses to Service LoadBalancer with annotations mentioned above. +You can change the mode at any point, but note that when switching to ```RequestedServicesOnly``` will unassign any addresses from Services that do not contain the above annotations. + +```bash +kubectl patch kubecontrollersconfiguration default --patch '{"spec": {"controllers":{"loadBalancer":{"AssignIPs": "RequestedServicesOnly"}}}}' +``` \ No newline at end of file diff --git a/calico/reference/calicoctl/ipam/check.mdx b/calico/reference/calicoctl/ipam/check.mdx index c48cfa5959..01dade7540 100644 --- a/calico/reference/calicoctl/ipam/check.mdx +++ b/calico/reference/calicoctl/ipam/check.mdx @@ -17,18 +17,25 @@ Usage: calicoctl ipam check [--config=] [--show-all-ips] [--show-problem-ips] [-o ] Options: - -h --help Show this screen. - -o --output= Path to output report file. - --show-all-ips Print all IPs that are checked. - --show-problem-ips Print all IPs that are leaked or not allocated properly. - -c --config= Path to the file containing connection configuration in - YAML or JSON format. - [default: /etc/calico/calicoctl.cfg] + -h --help Show this screen. + -o --output= Path to output report file. + --show-all-ips Print all IPs that are checked. + --show-problem-ips Print all IPs that are leaked or not allocated properly. + -c --config= Path to the file containing connection configuration in + YAML or JSON format. + [default: /etc/calico/calicoctl.cfg] + --kubeconfig= Path to Kubeconfig file Description: The ipam check command checks the integrity of the IPAM datastructures against Kubernetes. ``` +:::note + +When using calicoctl in etcd mode, Kubeconfig path is required. It can be specified either as env KUBECONFIG variable or by using the --kubeconfig parameter. + +::: + ### Examples Example workflow for checking consistency and releasing leaked addresses. diff --git a/calico/reference/resources/ippool.mdx b/calico/reference/resources/ippool.mdx index 77ce70618c..e181800b37 100644 --- a/calico/reference/resources/ippool.mdx +++ b/calico/reference/resources/ippool.mdx @@ -25,6 +25,7 @@ spec: allowedUses: - Workload - Tunnel + assignmentMode: Automatic ``` ## IP pool definition @@ -43,11 +44,12 @@ spec: | blockSize | The CIDR size of allocation blocks used by this pool. Blocks are allocated on demand to hosts and are used to aggregate routes. The value can only be set when the pool is created. | 20 to 32 (inclusive) for IPv4 and 116 to 128 (inclusive) for IPv6 | int | `26` for IPv4 pools and `122` for IPv6 pools. | | ipipMode | The mode defining when IPIP will be used. Cannot be set at the same time as `vxlanMode`. | Always, CrossSubnet, Never | string | `Never` | | vxlanMode | The mode defining when VXLAN will be used. Cannot be set at the same time as `ipipMode`. | Always, CrossSubnet, Never | string | `Never` | -| natOutgoing | When enabled, packets sent from $[prodname] networked containers in this pool to destinations outside of any Calico IP pools will be masqueraded. | true, false | boolean | `false` | -| disabled | When set to true, $[prodname] IPAM will not assign addresses from this pool. | true, false | boolean | `false` | +| natOutgoing | When enabled, packets sent from $[prodname] networked containers in this pool to destinations outside of any Calico IP pools will be masqueraded. | true, false | boolean | `false` | +| disabled | When set to true, $[prodname] IPAM will not assign addresses from this pool. | true, false | boolean | `false` | | disableBGPExport _(since v3.21.0)_ | Disable exporting routes from this IP Pool’s CIDR over BGP. | true, false | boolean | `false` | -| nodeSelector | Selects the nodes that $[prodname] IPAM should assign addresses from this pool to. | | [selector](#node-selector) | all() | -| allowedUses _(since v3.21.0)_ | Controls whether the pool will be used for automatic assignments of certain types. See [below](#allowed-uses). | Workload, Tunnel | list of strings | `["Workload", "Tunnel"]` | +| nodeSelector | Selects the nodes that $[prodname] IPAM should assign addresses from this pool to. | | [selector](#node-selector) | all() | +| allowedUses _(since v3.21.0)_ | Controls whether the pool will be used for automatic assignments of certain types. See [below](#allowed-uses). | Workload, Tunnel, LoadBalancer | list of strings | `["Workload", "Tunnel"]` | +| assignmentMode | Controls whether the pool will be used for automatic assignments or only if requested manualy | Automatic, Manual | strings | `Automatic` | :::note @@ -62,6 +64,8 @@ IP pool by creating blocks of incorrect size. When automatically assigning IP addresses to workloads, only pools with "Workload" in their `allowedUses` field are consulted. Similarly, when assigning IPs for tunnel devices, only "Tunnel" pools are eligible. +LoadBalancer option for `allowedUses` field cannot be used in conjunction with either Tunnel or Workload. + If the `allowedUses` field is not specified, it defaults to `["Workload", "Tunnel"]` for compatibility with older versions of Calico. It is not possible to specify a pool with no allowed uses. @@ -115,20 +119,15 @@ Increasing the block size from the default (e.g., using `24` for IPv4 to give 25 Reducing the block size from the default (e.g., using `28` for IPv4 to give 16 addresses per block) means more blocks per host and therefore potentially more routes. This can be beneficial if it allows the blocks to be more fairly distributed amongst the hosts. +### Assignment Mode + +Determines if the IP Pool should be used by Calico IPAM for automatic IP assignments. With `Automatic` assignmentMode Calico IPAM will automatically assign IP addresses from this pool. By setting assignmentMode to `Manual` Calico IPAM will not assign IP addresses from this pool unless specified by the user. + ### Node Selector For details on configuring IP pool node selectors, please read the [Assign IP addresses based on topology guide.](../../networking/ipam/assign-ip-addresses-topology.mdx). -:::tip - -To prevent an IP pool from being used automatically by $[prodname] IPAM, while still allowing -it to be used manually for static assignments, set the `IPPool`'s `nodeSelector` to `!all()`. Since the selector -matches no nodes, the IPPool will not be used automatically and, unlike setting `disabled: true`, it can still be -used for manual assignments. - -::: - #### Selector reference diff --git a/calico/reference/resources/kubecontrollersconfig.mdx b/calico/reference/resources/kubecontrollersconfig.mdx index b2b38dea76..246ecc802d 100644 --- a/calico/reference/resources/kubecontrollersconfig.mdx +++ b/calico/reference/resources/kubecontrollersconfig.mdx @@ -33,6 +33,8 @@ spec: reconcilerPeriod: 5m namespace: reconcilerPeriod: 5m + loadbalancer: + assignIPs: AllServices ``` ## Kubernetes controllers configuration definition @@ -117,6 +119,14 @@ $[prodname] datastore. | ---------------- | ---------------------------------------------------------------- | --------------------------------- | ------- | | reconcilerPeriod | Period to perform reconciliation with the $[prodname] datastore | [Duration string][parse-duration] | 5m | +### LoadBalancerController + +The loadBalancer controller manages IPAM for Service LoadBalancer + +| Field | Description | Accepted Values | Schema | Default | +| ---------------- | ---------------------------------------------------------------- | ---------------------------------------- | ------ | ----------- | +| assignIPs | Mode in which LoadBalancer controller operates | AllServices, RequestedServicesOnly | String | AllServices | + ## Supported operations | Datastore type | Create | Delete (Global `default`) | Update | Get/List | Notes | diff --git a/sidebars-calico.js b/sidebars-calico.js index a75074ddc6..078889c25f 100644 --- a/sidebars-calico.js +++ b/sidebars-calico.js @@ -279,6 +279,7 @@ module.exports = { 'networking/ipam/migrate-pools', 'networking/ipam/change-block-size', 'networking/ipam/legacy-firewalls', + 'networking/ipam/service-loadbalancer' ], }, {