K3s 配置 Envoy Gateway:结合 Cloudflare Tunnel 与 Tailscale 实现内外网服务隔离

status
Published
type
Post
slug
k3s-envoy-gateway-cloudflare-tunnel-and-tailscale
date
Sep 25, 2026
tags
K8s
Config
Cloudflare
Note
summary
笔记记录了在 K3s 中通过 Envoy Gateway 实现服务内外网隔离的运维配置过程,包括利用 Tailscale 接入 LoadBalancer 暴露私有服务、通过 Cloudflare Tunnel 暴露公开服务,并结合 cert-manager 自动签发 Let's Encrypt TLS 证书以及配合 external-dns 实现 Cloudflare DNS 记录的自动化管理。
这几天重装迁移了一台线路较普通的服务器,正好当实验环境试试 Envoy Gateway,顺便记录下大致配置过程。
在 AI Agent 大行其道的今天,这种操作方式多少显得有点“古法运维”。AI 固然能几秒钟把服务拉起来,但如果不细致审视底层的组装细节,配置次数多了时间长了,整个就容易变成“黑盒”。按需配置、理清关键组件的流量走向,这种清晰的掌控感还是不错的。(也可能是”登味儿/守旧派“,或者说单纯烧不起 Token 🤣

配置概览

目标效果:在同一个 K3s 集群内,将公开服务与私有服务从入口层进行彻底隔离:
  • 私有服务:通过 Envoy Gateway 暴露为 LoadBalancer 类型的 Service,配合集群内 K3s 自带的 ServiceLB(原 Klipper LoadBalancer),借助 Tailscale 节点网络进行内网访问。
  • 公开服务:通过 Envoy Gateway 暴露为 ClusterIP 类型的 Service,流量仅在集群内部流转,前端由部署在集群内的 cloudflared 容器以 Cloudflare Tunnel 方式反向代理出去。
[GatewayClass: envoy-gateway] (控制面控制器) │ ├──> [Gateway: envoy-gateway-tailscale] ──> Service类型: LoadBalancer (Tailscale 内网接入) │ └──> [Gateway: envoy-gateway-cloudflare] ──> Service类型: ClusterIP (Cloudflare Tunnel 接入)
此外,配合 cert-manager 完成 Let's Encrypt TLS 证书自动签发,并利用 external-dns 根据网关类型自动同步 DNS 记录(Tailscale 自动建 A 记录,Cloudflare Tunnel 自动建 CNAME 记录)。

K3s 配置安装

基础的系统配置和 Tailscale 安装此处略去不表。
配置文件 /etc/rancher/k3s/config.yaml
write-kubeconfig-mode: "0600" # token: "" # 强随机字符串 (openssl rand -hex 16) token-file: /etc/rancher/k3s/token bind-address: "0.0.0.0" node-ip: "<Tailscale-IP>" node-external-ip: "<Tailscale-IP>" tls-san: - "127.0.0.1" - "localhost" - "<Tailscale-IP>" - "<HostName>" - "<Tailscale-MagicDNS>" - "<个人域名>" disable: - traefik kube-proxy-arg: - "proxy-mode=nftables" # 基础的 kubelet 参数就放在这,官方建议是使用专门的 kubelet 配置文件 kubelet-arg: - "max-pods=110" - "image-gc-high-threshold=80" - "image-gc-low-threshold=60" - "container-log-max-size=10Mi" - "container-log-max-files=3" - "eviction-hard=memory.available<300Mi,nodefs.available<2Gi" - "serialize-image-pulls=false" # 需调整内核参数 protect-kernel-defaults: true secrets-encryption: true
token-file 生成
mkdir -p /etc/rancher/k3s TOKEN_FILE="/etc/rancher/k3s/token" if [[ ! -f "${TOKEN_FILE}" ]]; then if command -v openssl >/dev/null 2>&1; then openssl rand -hex 16 > "${TOKEN_FILE}" else head -c 32 /dev/urandom \ | base64 \ | tr -dc 'a-zA-Z0-9' \ | head -c 32 > "${TOKEN_FILE}" fi chmod 600 "${TOKEN_FILE}" else log "检测到已有 token,跳过生成" fi
内核参数配置
cat >/etc/sysctl.d/99-k3s-kubelet.conf <<EOF # Kubernetes network forwarding net.ipv4.ip_forward = 1 net.ipv6.conf.all.forwarding = 1 # Bridge netfilter net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 # Protect kernel defaults requirements vm.overcommit_memory = 1 kernel.panic = 10 kernel.panic_on_oops = 1 EOF sysctl --system >/dev/null
安装命令 curl -sfL https://get.k3s.io | sh -
notion image
面板用的是 Radar 项目,本机启动,CLI + Web 形式,提供 MCP Server

Envoy 网关部署

直接用 Helm 安装 Envoy Gateway,可配置项:Envoy GatewayEnvoy GatewayGateway Helm Chart
# 安装 helm install eg oci://docker.io/envoyproxy/gateway-helm --version v1.9.1 -n envoy-gateway-system --create-namespace --set config.envoyGateway.logging.format=json # 等待 Envoy Gateway 启动就绪 kubectl wait --timeout=5m -n envoy-gateway-system deployment/envoy-gateway --for=condition=Available
notion image
控制面安装好了,接下来就是 GatewayClass、Gateway、HTTPRoute 以及数据面的网络/路由发布。
创建 Gateway Class
apiVersion: gateway.networking.k8s.io/v1 kind: GatewayClass metadata: name: eg spec: controllerName: gateway.envoyproxy.io/gatewayclass-controller
创建 EnvoyProxy 配置,覆写 Envoy Gateway Controller 之后生成的 Service 名称,
apiVersion: gateway.envoyproxy.io/v1alpha1 kind: EnvoyProxy metadata: name: cloudflare-proxy-config namespace: envoy-gateway-system spec: provider: type: Kubernetes kubernetes: envoyService: type: ClusterIP name: envoy-svc-fixed
创建 Gateway,其中 Cloudflare Tunnel 对应引入上面的 Envoy 自定义覆写配置,Tailscale 不用
apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: cloudflare-gateway namespace: envoy-gateway-system spec: gatewayClassName: eg listeners: - name: http protocol: HTTP port: 80 allowedRoutes: namespaces: from: All infrastructure: parametersRef: group: gateway.envoyproxy.io kind: EnvoyProxy name: cloudflare-proxy-config --- apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: tailscale-gateway namespace: envoy-gateway-system spec: gatewayClassName: eg listeners: - name: http protocol: HTTP port: 80 allowedRoutes: namespaces: from: All
  • Tailscale Gateway:Service 为默认的 LoadBalancer。K3s 的 ServiceLB 检测到后,会在节点启动 Pod 进行端口转发与流量透传。
  • Cloudflare Gateway:引入自定义 EnvoyProxy 配置,指定 Service 为 ClusterIP,仅限 K3s 内网访问。

Tailscale 链路测试

用 Kubernetes 的 httpbin 测试服务
Deployment + Service
apiVersion: apps/v1 kind: Deployment metadata: name: httpbin namespace: default spec: replicas: 1 selector: matchLabels: app: httpbin template: metadata: labels: app: httpbin spec: containers: - name: httpbin image: kennethreitz/httpbin ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: httpbin namespace: default spec: selector: app: httpbin hostnames: - httpbin.okhk.net ports: - port: 80 targetPort: 80
HTTPRoute
apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: httpbin namespace: default spec: parentRefs: - name: tailscale-gateway namespace: envoy-gateway-system sectionName: http rules: - matches: - path: type: PathPrefix value: / backendRefs: - name: httpbin port: 80
本地可以通过修改 hosts 或在内网 DNS 中配置域名指向 Tailscale 节点 IP,即可成功访问。
notion image

Cloudflare Tunnel 接入

上面验证了 Tailscale 的接入,下面开始接入 Cloudflare Tunnel,将其运行在 K3s 内部,直接对接 Envoy。
Namespace
apiVersion: v1 kind: Namespace metadata: name: cloudflare
Secret
在 Cloudflare Zero Trust 处创建得到 Tunnel Token,或者调用 API 也行,不做赘述
apiVersion: v1 kind: Secret metadata: name: cf-tunnel-secret namespace: cloudflare type: Opaque stringData: tunnel-token: "<Tunnel Token>"
Deployment
apiVersion: apps/v1 kind: Deployment metadata: name: cloudflared namespace: cloudflare labels: app.kubernetes.io/name: cloudflared spec: replicas: 1 revisionHistoryLimit: 2 selector: matchLabels: app.kubernetes.io/name: cloudflared template: metadata: labels: app.kubernetes.io/name: cloudflared spec: containers: - name: cloudflared image: cloudflare/cloudflared:2026.9.3 imagePullPolicy: IfNotPresent args: - tunnel - --output - json - --loglevel - debug - --metrics - 0.0.0.0:2000 - --no-autoupdate - run env: - name: TUNNEL_TOKEN valueFrom: secretKeyRef: name: cf-tunnel-secret key: tunnel-token ports: - name: metrics containerPort: 2000 protocol: TCP resources: requests: cpu: 10m memory: 32Mi limits: cpu: 100m memory: 128Mi livenessProbe: httpGet: path: /ready port: metrics initialDelaySeconds: 15 periodSeconds: 30 timeoutSeconds: 5 failureThreshold: 3 readinessProbe: httpGet: path: /ready port: metrics initialDelaySeconds: 5 periodSeconds: 10 timeoutSeconds: 3 failureThreshold: 3 securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true runAsNonRoot: true runAsUser: 65532 runAsGroup: 65532 capabilities: drop: - ALL seccompProfile: type: RuntimeDefault volumeMounts: - name: tmp mountPath: /tmp volumes: - name: tmp emptyDir: {}
部署完成后,Cloudflare Dashboard 显示隧道连接已建立。配置一条 Published application routes ,Service 指向 K3s 中 Envoy Gateway Service。也就是:envoy-svc-fixed.envoy-gateway-system。此时公网流量经过 Cloudflare CDN 边缘节点,由 Tunnel 穿透注入集群,再传递给 Envoy Gateway 进行路由分发。
notion image
notion image
notion image
至此 Tailscale 和 Cloudflare Tunnel 访问 K3s 服务的基本链路已经测试通过。

cert-manager 自动签发 Let's Encrypt 证书

Helm 部署
helm install \ cert-manager oci://quay.io/jetstack/charts/cert-manager \ --version v1.21.2 \ --namespace cert-manager \ --create-namespace \ --set crds.enabled=true
Cloudflare API Key 创建
登录 Cloudflare Dashboard,进入 My Profile -> API Tokens,创建 Token: • Permissions: ◦ Zone - DNS - Edit ◦ Zone - Zone - Read • Zone Resources:选择包含您域名的 Zone(或 Include - All zones)
将 Token 存储为 Kubernetes Secret,后续要使用 ClusterIssuer ,必须存储在 cert-manager 相同 namespace 下
kubectl create secret generic cloudflare-api-token-secret \ --namespace cert-manager \ --from-literal=api-token=<CLOUDFLARE_API_TOKEN>
配置 cert-manager ClusterIssuer 开启 DNS-01 Challenge 认证
apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt-cloudflare-dns spec: acme: # 邮箱,用于接收 LetsEncrypt 证书到期提醒 email: your-email@example.com server: https://acme-v02.api.letsencrypt.org/directory privateKeySecretRef: name: letsencrypt-cloudflare-account-key solvers: - dns01: cloudflare: apiTokenSecretRef: name: cloudflare-api-token-secret key: api-token
申请证书(Envoy Gateway Namespace 下)
apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: okhk-tls namespace: envoy-gateway-system spec: secretName: okhk-tls-secret issuerRef: name: letsencrypt-cloudflare-dns kind: ClusterIssuer dnsNames: - "*.okhk.net"
在 Envoy Gateway 的 Listener 中引用挂载 TLS 证书
apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: envoy-gateway namespace: envoy-gateway-system spec: gatewayClassName: eg listeners: - name: http protocol: HTTP port: 80 allowedRoutes: namespaces: from: All - name: https protocol: HTTPS port: 443 hostname: "*.okhk.net" tls: mode: Terminate certificateRefs: - kind: Secret name: okhk-tls-secret allowedRoutes: namespaces: from: All infrastructure: parametersRef: group: gateway.envoyproxy.io kind: EnvoyProxy name: custom-proxy-config
至此 TLS 证书配置完成,后续域名只需要配置 Tailscale 的 A 记录或者 Cloudflare Tunnel 的 CNAME 记录即可完成私有服务和公开服务的接入。

external-dns 配置

为了避免手动维护 DNS 记录,通过部署 external-dns 实现 HTTPRoute 与 DNS 的自动同步:
helm repo add external-dns https://kubernetes-sigs.github.io/external-dns/ helm repo update helm install external-dns external-dns/external-dns \ -namespace external-dns \ --create-namespace \ -f external-dns-values.yaml
domainFilters: - okhk.net env: - name: CF_API_TOKEN valueFrom: secretKeyRef: key: apiKey name: external-dns extraArgs: - --cloudflare-dns-records-per-page=5000 - --cloudflare-record-comment=provisioned by external-dns - --events - --min-event-sync-interval=5s - --batch-change-size=200 - --batch-change-interval=1s - --metrics-address=:7979 - --events-emit=RecordError interval: 1m logFormat: json logLevel: debug policy: upsert provider: name: cloudflare registry: txt secretConfiguration: data: apiKey: enabled: true mountPath: /etc/secrets sources: - gateway-httproute txtOwnerId: edns txtPrefix: _externaldns.
  • Tailscale 路由:external-dns 默认解析 Gateway 的外部 IP 并生成 A 记录,正好契合 Tailscale LoadBalancer 的 IP 地址。
  • Cloudflare Tunnel 路由:在 Gateway 资源上添加注解 external-dns.kubernetes.io/target 指向 Tunnel CNAME,并在 HTTPRoute 上添加注解以自动开启 Cloudflare Proxied:
Gateway 中配置 CNAME 记录指向目标:external-dns.kubernetes.io/target
apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: annotations: external-dns.kubernetes.io/target: <Tunnel-ID>.cfargotunnel.com name: cloudflare-gateway namespace: envoy-gateway-system spec: gatewayClassName: eg infrastructure: parametersRef: group: gateway.envoyproxy.io kind: EnvoyProxy name: cf-proxy-config listeners: - allowedRoutes: namespaces: from: All name: http port: 80 protocol: HTTP - allowedRoutes: namespaces: from: All hostname: "*.okhk.net" name: https port: 443 protocol: HTTPS tls: certificateRefs: - kind: Secret name: okhk-tls-secret mode: Terminate --- apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: tailscale-gateway namespace: envoy-gateway-system spec: gatewayClassName: eg listeners: - allowedRoutes: namespaces: from: All hostname: "*.okhk.net" name: https port: 443 protocol: HTTPS tls: certificateRefs: - kind: Secret name: okhk-tls-secret mode: Terminate
HTTPRoute 中配置开启 Cloudflare 代理: external-dns.kubernetes.io/cloudflare-proxied
metadata: annotations: external-dns.kubernetes.io/cloudflare-proxied: "true"
下图中 RemoteAddr10.42.0.28 和 10.42.0.29 分别是两个 Gateway 实例的 Pod IP,可见确实是两个 Gateway 分别转发的。
notion image
notion image
完成上述架构配置后,日后新增服务只需编写对应的 HTTPRoute 并绑定到不同的 Gateway,即可无缝完成私有服务与公开服务的隔离发布。
记录的有些杂,本来想用 AI 跑一轮优化下行文结构,看了下还是算了,就先到这。