什么是全链路灰度？
Aliware
在发布应用的过程中，我们通常希望用少量特定流量来验证新版本的发布是否正常，以保障整体稳定性。这个过程被称为灰度发布。关于灰度发布，我们通过逐步增加发布的范围，来验证新版本的稳定性。如果新版本出现问题，我们也能及时发现，控制影响范围，保障整体的稳定性。
-
逐步增加发布的影响范围，拒绝一次性全部发布。 -
阶段性的发布过程，可以通过金丝雀发布方式小心验证，以验证新版本的稳定性； -
可暂停、可回滚、可继续、可自动化状态流转，以便灵活地控制发布过程并确保稳定性；
据调研数据 70% 的线上问题都是由于变更导致，我们常说安全生产三板斧，可灰度、可观测、可回滚，也是为了控制变更带来的风险与影响面。通过采用灰度发布的方式，我们能够更加稳健地发布新版本，避免因发布过程中出现的问题而带来的损失。
Istio 全链路灰度技术解析
Aliware
如何通过 Istio 实现全链路灰度能力，想必大家都很关心这个话题，今天我们就详细谈一下基于 istio 实现全链路灰度能力的几个关键技术细节。
为什么选择 Kruise Rollout？
Aliware
Kruise + Istio 全链路灰度实践
Aliware
谈完技术实现的细节，下面我们就开始基于 Kruise Rollout 跟 Istio 的全链路灰度能力实践。
apiVersion: v1kind: Servicemetadata:name: mockanamespace: e2elabels:app: mockaservice: mockaspec:ports:- port: 8000name: httpselector:app: mocka---apiVersion: apps/v1kind: Deploymentmetadata:name: mocka-basenamespace: e2elabels:app: mockaspec:replicas: 1selector:matchLabels:app: mockatemplate:metadata:labels:app: mockaversion: basespec:containers:- name: defaultimage: registry.cn-beijing.aliyuncs.com/aliacs-app-catalog/go-http-sample:1.0imagePullPolicy: Alwaysenv:- name: versionvalue: base- name: appvalue: mocka- name: upstream_urlvalue: "http://mockb:8000/"ports:- containerPort: 8000---apiVersion: v1kind: Servicemetadata:name: mockbnamespace: e2elabels:app: mockbservice: mockbspec:ports:- port: 8000name: httpselector:app: mockb---apiVersion: apps/v1kind: Deploymentmetadata:name: mockb-basenamespace: e2elabels:app: mockbspec:replicas: 1selector:matchLabels:app: mockbtemplate:metadata:labels:app: mockbversion: basespec:containers:- name: defaultimage: registry.cn-beijing.aliyuncs.com/aliacs-app-catalog/go-http-sample:1.0imagePullPolicy: Alwaysenv:- name: versionvalue: base- name: appvalue: mockbports:- containerPort: 8000
服务的 header 部分处理代码如下所示：
// All URLs will be handled by this functionm.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {requestId := r.Header.Get("my-request-id")fmt.Printf("receive request: my-request-id: %s\n", requestId)response := fmt.Sprintf("-> %s(version: %s, ip: %s)", app, version, ip)if url != "" {// 新请求只发送my-request-idcontent := doReq(url, requestId)response = response + content}w.Write([]byte(response))})
之后部署 Istio Gateway、DestinationRule 和 VirtualService 配置路由和外部访问：
apiVersion: networking.istio.io/v1beta1kind: Gatewaymetadata:name: gatewaynamespace: e2espec:selector:istio: ingressgatewayservers:- hosts:- "*"port:name: httpnumber: 80protocol: HTTP---apiVersion: networking.istio.io/v1beta1kind: DestinationRulemetadata:name: dr-mockanamespace: e2espec:host: mockatrafficPolicy:loadBalancer:simple: ROUND_ROBINsubsets:- labels:version: basename: version-base---apiVersion: networking.istio.io/v1beta1kind: DestinationRulemetadata:name: dr-mockbnamespace: e2espec:host: mockbtrafficPolicy:loadBalancer:simple: ROUND_ROBINsubsets:- labels:version: basename: version-base---apiVersion: networking.istio.io/v1beta1kind: VirtualServicemetadata:name: vs-mockanamespace: e2espec:gateways:- simple-gatewayhosts:- "*"http:- route:- destination:host: mockasubset: version-base---apiVersion: networking.istio.io/v1beta1kind: VirtualServicemetadata:name: vs-mockbnamespace: e2espec:hosts:- mockbhttp:- route:- destination:host: mockbsubset: version-base
部署完成后整个结构如下图所示：
kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].nodePort}'kubectl get po -l istio=ingressgateway -n istio-system -o jsonpath='{.items[0].status.hostIP}'
此时运行 curl http://GATEWAY_IP:PORT 可以得到如下结果：
-> mocka(version: base, ip: 10.244.1.36)-> mockb(version: base, ip: 10.244.1.37)
-
添加了匹配 my-request-id=canary 的 header 规则，包含指定 header 的流量会走灰度环境 -
为发布的 pod 添加了 label istio.service.tag=gray 以及 version=canary
⚠️pod label 的添加规则是什么？
apiVersion: rollouts.kruise.io/v1alpha1kind: Rolloutmetadata:name: rollouts-anamespace: demoannotations:rollouts.kruise.io/rolling-style: canaryrollouts.kruise.io/trafficrouting: mocka-trspec:disabled: falseobjectRef:workloadRef:apiVersion: apps/v1kind: Deploymentname: mocka-basestrategy:canary:steps:- replicas: 1patchPodTemplateMetadata:labels:istio.service.tag: grayversion: canary---apiVersion: rollouts.kruise.io/v1alpha1kind: TrafficRoutingmetadata:name: mocka-trnamespace: demospec:strategy:matches:- headers:- type: Exactname: my-request-idvalue: canaryobjectRef:- service: mockacustomNetworkRefs:- apiVersion: networking.istio.io/v1alpha3kind: VirtualServicename: vs-mocka- apiVersion: networking.istio.io/v1beta1kind: DestinationRulename: dr-mocka---apiVersion: rollouts.kruise.io/v1alpha1kind: Rolloutmetadata:name: rollouts-bnamespace: demoannotations:rollouts.kruise.io/rolling-style: canaryrollouts.kruise.io/trafficrouting: mockb-trspec:disabled: falseobjectRef:workloadRef:apiVersion: apps/v1kind: Deploymentname: mockb-basestrategy:canary:steps:- replicas: 1patchPodTemplateMetadata:labels:istio.service.tag: grayversion: canary---apiVersion: rollouts.kruise.io/v1alpha1kind: TrafficRoutingmetadata:name: mockb-trnamespace: demospec:strategy:matches:- headers:- type: Exactname: my-request-idvalue: canaryobjectRef:- service: mockbcustomNetworkRefs:- apiVersion: networking.istio.io/v1alpha3kind: VirtualServicename: vs-mockb- apiVersion: networking.istio.io/v1beta1kind: DestinationRulename: dr-mockb
apiVersion: networking.istio.io/v1beta1kind: VirtualServicemetadata:annotations:kubectl.kubernetes.io/last-applied-configuration: |{"apiVersion":"networking.istio.io/v1beta1","kind":"VirtualService","metadata":{"annotations":{},"name":"vs-mocka","namespace":"demo"},"spec":{"gateways":["simple-gateway"],"hosts":["*"],"http":[{"route":[{"destination":{"host":"mocka","subset":"version-base"}}]}]}}rollouts.kruise.io/origin-spec-configuration: '{"spec":{"gateways":["simple-gateway"],"hosts":["*"],"http":[{"route":[{"destination":{"host":"mocka","subset":"version-base"}}]}]},"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"networking.istio.io/v1beta1\",\"kind\":\"VirtualService\",\"metadata\":{\"annotations\":{},\"name\":\"vs-mocka\",\"namespace\":\"demo\"},\"spec\":{\"gateways\":[\"simple-gateway\"],\"hosts\":[\"*\"],\"http\":[{\"route\":[{\"destination\":{\"host\":\"mocka\",\"subset\":\"version-base\"}}]}]}}\n"}}'creationTimestamp: "2023-09-12T07:49:15Z"generation: 40name: vs-mockanamespace: demoresourceVersion: "98670"uid: c7da3a99-789c-4f1e-93a4-caaee41cbe06spec:gateways:- simple-gatewayhosts:- '*'http:# -- lua脚本自动添加的规则- match:- headers:my-request-id:exact: canaryroute:- destination:host: mockasubset: canary# --- route:- destination:host: mockasubset: version-base---apiVersion: networking.istio.io/v1beta1kind: VirtualServicemetadata:annotations:kubectl.kubernetes.io/last-applied-configuration: |{"apiVersion":"networking.istio.io/v1beta1","kind":"VirtualService","metadata":{"annotations":{},"name":"vs-mockb","namespace":"demo"},"spec":{"hosts":["mockb"],"http":[{"route":[{"destination":{"host":"mockb","subset":"version-base"}}]}]}}rollouts.kruise.io/origin-spec-configuration: '{"spec":{"hosts":["mockb"],"http":[{"route":[{"destination":{"host":"mockb","subset":"version-base"}}]}]},"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"networking.istio.io/v1beta1\",\"kind\":\"VirtualService\",\"metadata\":{\"annotations\":{},\"name\":\"vs-mockb\",\"namespace\":\"demo\"},\"spec\":{\"hosts\":[\"mockb\"],\"http\":[{\"route\":[{\"destination\":{\"host\":\"mockb\",\"subset\":\"version-base\"}}]}]}}\n"}}'creationTimestamp: "2023-09-12T07:49:16Z"generation: 40name: vs-mockbnamespace: demoresourceVersion: "98677"uid: 7c96ee2b-96ce-48e4-ba6d-cf94171ed854spec:hosts:- mockbhttp:# -- lua脚本自动添加的规则- match:- headers:my-request-id:exact: canaryroute:- destination:host: mockbsubset: canary# --- route:- destination:host: mockbsubset: version-base---apiVersion: networking.istio.io/v1beta1kind: DestinationRulemetadata:annotations:kubectl.kubernetes.io/last-applied-configuration: |{"apiVersion":"networking.istio.io/v1beta1","kind":"DestinationRule","metadata":{"annotations":{},"name":"dr-mocka","namespace":"demo"},"spec":{"host":"mocka","subsets":[{"labels":{"version":"base"},"name":"version-base"}],"trafficPolicy":{"loadBalancer":{"simple":"ROUND_ROBIN"}}}}rollouts.kruise.io/origin-spec-configuration: '{"spec":{"host":"mocka","subsets":[{"labels":{"version":"base"},"name":"version-base"}],"trafficPolicy":{"loadBalancer":{"simple":"ROUND_ROBIN"}}},"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"networking.istio.io/v1beta1\",\"kind\":\"DestinationRule\",\"metadata\":{\"annotations\":{},\"name\":\"dr-mocka\",\"namespace\":\"demo\"},\"spec\":{\"host\":\"mocka\",\"subsets\":[{\"labels\":{\"version\":\"base\"},\"name\":\"version-base\"}],\"trafficPolicy\":{\"loadBalancer\":{\"simple\":\"ROUND_ROBIN\"}}}}\n"}}'creationTimestamp: "2023-09-12T07:49:15Z"generation: 12name: dr-mockanamespace: demoresourceVersion: "98672"uid: a6f49044-e889-473c-b188-edbdb8ee347fspec:host: mockasubsets:- labels:version: basename: version-base# -- lua脚本自动添加的规则- labels:istio.service.tag: grayname: canary# --trafficPolicy:loadBalancer:simple: ROUND_ROBIN---apiVersion: networking.istio.io/v1beta1kind: DestinationRulemetadata:annotations:kubectl.kubernetes.io/last-applied-configuration: |{"apiVersion":"networking.istio.io/v1beta1","kind":"DestinationRule","metadata":{"annotations":{},"name":"dr-mockb","namespace":"demo"},"spec":{"host":"mockb","subsets":[{"labels":{"version":"base"},"name":"version-base"}],"trafficPolicy":{"loadBalancer":{"simple":"ROUND_ROBIN"}}}}rollouts.kruise.io/origin-spec-configuration: '{"spec":{"host":"mockb","subsets":[{"labels":{"version":"base"},"name":"version-base"}],"trafficPolicy":{"loadBalancer":{"simple":"ROUND_ROBIN"}}},"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"networking.istio.io/v1beta1\",\"kind\":\"DestinationRule\",\"metadata\":{\"annotations\":{},\"name\":\"dr-mockb\",\"namespace\":\"demo\"},\"spec\":{\"host\":\"mockb\",\"subsets\":[{\"labels\":{\"version\":\"base\"},\"name\":\"version-base\"}],\"trafficPolicy\":{\"loadBalancer\":{\"simple\":\"ROUND_ROBIN\"}}}}\n"}}'creationTimestamp: "2023-09-12T07:49:15Z"generation: 12name: dr-mockbnamespace: demoresourceVersion: "98678"uid: 4bd0f6c5-efa1-4558-9e31-6f7615c21f9aspec:host: mockbsubsets:- labels:version: basename: version-base# -- lua脚本自动添加的规则- labels:istio.service.tag: grayname: canary# --trafficPolicy:loadBalancer:simple: ROUND_ROBIN
此时运行 curl http://GATEWAY_IP:PORT 可以得到如下结果，所有流量均通过 base 版本服务。
-> mocka(version: base, ip: 10.244.1.36)-> mockb(version: base, ip: 10.244.1.37)此时运行 curl -H http://GATEWAY_IP:PORT -Hmy-request-id:canary 可以得到如下结果，所有流量均通过 canary 版本服务。
-> mocka(version: canary, ip: 10.244.1.41)-> mockb(version: canary, ip: 10.244.1.42)此时整个服务的流量可以表示为下图，包含 headermy-request-id=canary 的流量全部走灰度环境，其它流量全部走基线环境：
apiVersion: networking.istio.io/v1alpha3kind: EnvoyFiltermetadata:name: http-request-labelling-according-sourcenamespace: istio-systemspec:workloadSelector:labels:app: istio-ingressgatewayconfigPatches:- applyTo: HTTP_FILTERmatch:context: GATEWAYlistener:filterChain:filter:name: "envoy.filters.network.http_connection_manager"subFilter:name: "envoy.filters.http.router"patch:operation: INSERT_BEFOREvalue:name: envoy.luatyped_config:"@type": "type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua"inlineCode: |function envoy_on_request(request_handle)local header = "agent"headers = request_handle:headers()version = headers:get(header)if (version ~= nil) thenif (version == "pc") thenheaders:add("my-request-id","canary")elseheaders:add("my-request-id","base")endelseheaders:add("my-request-id","base")endend
保持其它配置不变，此时运行 curl http://GATEWAY_IP:PORT 可以得到如下结果，所有流量均通过 base 版本服务。
-> mocka(version: base, ip: 10.244.1.36)-> mockb(version: base, ip: 10.244.1.37)此时运行 curl -H http://GATEWAY_IP:PORT -Hagent:pc 可以得到如下结果，EnvoyFilter 自动为该流量添加 headermy-request-id=canary，因此所有流量均通过新版本服务。
-> mocka(version: canary, ip: 10.244.1.41)-> mockb(version: canary, ip: 10.244.1.42)完整的全链路灰度方案
Aliware
Kruise Rollout 除了支持开源的 Istio 实现全链路灰度外，还支持与 MSE 实现完整的全链路灰度方案，我们可以通过如下操作文档[2]快速实现体系化的全链路灰度能力。
-
加入社区 Slack channel (English) -
加入社区钉钉群：搜索群号 23330762 (Chinese) -
加入社区微信群（新）：添加用户 openkruise 并让机器人拉你入群 (Chinese)
[1] Kruise Rollout
https://openkruise.io/rollouts/introduction
[2] 操作文档