浏览 422
Kubernetes 已成为容器化应用
拓扑结构
所用硬件
所用软件
●
●
●
●
●
●
收集的指标
●
●
●
测试方法
●
●
●
测试 RPS
taskset -c 0-21,44-65 wrk -t 44 -c 1000 -d 180s https://host.example.com:443/1kb.bin测试 TPS
taskset -c 0-21,44-65 wrk -t 44 -c 1000 -d 180s -H ‘Connection: Close’ https://host.example.com:443
●
●
测试 HTTP 吞吐量
taskset -c 0-21,44-65 wrk -t 44 -c 1000 -d 180s https://host.example.com:443/1mb.bin性能分析
HTTP 请求的 RPS
| CPU | RPS |
|---|---|
| 1 | 36,647 |
| 2 | 74,192 |
| 4 | 148,936 |
| 8 | 300,625 |
| 16 | 342,651 |
| 24 | 342,785 |
HTTPS 请求的 RPS
| CPU | RPS |
|---|---|
| 1 | 28,640 |
| 2 | 58,041 |
| 4 | 117,255 |
| 8 | 236,703 |
| 16 | 341,232 |
| 24 | 342,785 |
TPS
1 | ||
2 | ||
4 | ||
8 | ||
吞吐量
| CPU | 吞吐量(Gbps) |
|---|---|
| 1 | 1.91 |
| 2 | 4.78 |
| 4 | 8.72 |
| 8 | 8.64 |
| 16 | 8.80 |
| 24 | 8.80 |
测试技术细节
$ kubectl get nodes --show-labels
●
●
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: nginx-ingress
namespace: nginx-ingress
spec:
selector:
matchLabels:
app: nginx-ingress
template:
metadata:
labels:
app: nginx-ingress
spec:
serviceAccountName: nginx-ingress
nodeSelector:
kubernetes.io/hostname: npq3
hostNetwork: true
containers:
- image: nginx/nginx-ingress:edge
imagePullPolicy: IfNotPresent
name: nginx-ingress
ports:
- name: http
containerPort: 80
hostPort: 80
- name: https
containerPort: 443
hostPort: 443
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
args:
- -nginx-configmaps=$(POD_NAMESPACE)/nginx-config
- -default-server-tls-secret=$(POD_NAMESPACE)/default-server-secret使用Kubernetes 服务配置以暴露 NGINX Ingress Controller
apiVersion: v1
kind: Service
metadata:
name: nginx-ingress
namespace: nginx-ingress
spec:
externalTrafficPolicy: Local
ports:
- port: 80
targetPort: 80
protocol: TCP
name: http
- port: 443
targetPort: 443
protocol: TCP
name: https
externalIPs:
- 10.10.16.10
selector:
app: nginx-ingress用于 NGINX Ingress Controller的 Kubernetes ConfigMap
●
●
●
kind: ConfigMap
apiVersion: v1
metadata:
name: nginx-config
namespace: nginx-ingress
data:
worker-processes: "24"
worker-connections: "100000"
worker-rlimit-nofile: "102400"
worker-cpu-affinity: "auto 111111111111111111111111"
keepalive: "200"
main-template: |
user nginx;
worker_processes {{.WorkerProcesses}};
{{- if .WorkerRlimitNofile}}
worker_rlimit_nofile {{.WorkerRlimitNofile}};{{end}}
{{- if .WorkerCPUAffinity}}
worker_cpu_affinity {{.WorkerCPUAffinity}};{{end}}
{{- if .WorkerShutdownTimeout}}
worker_shutdown_timeout {{.WorkerShutdownTimeout}};{{end}}
daemon off;
error_log /var/log/nginx/error.log {{.ErrorLogLevel}};
pid /var/run/nginx.pid;
{{- if .MainSnippets}}
{{range $value := .MainSnippets}}
{{$value}}{{end}}
{{- end}}
events {
worker_connections {{.WorkerConnections}};
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
...
sendfile on;
access_log off;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 315;
keepalive_requests 10000000;
#gzip on;
...
}用于后端的 Kubernetes DaemonSet
网络优化
后端 DaemonSet 部署
apiVersion: v1
data:
app.conf: "server {\n listen 80;\nlocation / {\n root /usr/share/nginx/bin;
\n }\n}\n"
kind: ConfigMap
metadata:
name: app-conf
namespace: default
---
apiVersion: v1
data:
nginx.conf: |+
user nginx;
worker_processes 44;
worker_rlimit_nofile 102400;
worker_cpu_affinity auto 0000000000000000000000111111111111111111111100000000000000000000001111111111111111111111;
error_log /var/log/nginx/error.log notice;
pid /var/run/nginx.pid;
events {
worker_connections 100000;
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
sendfile on;
tcp_nopush on;
tcp_nodelay on;
access_log off;
include /etc/nginx/conf.d/*.conf;
}
---
apiVersion: v1
data:
1kb.bin: "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"
kind: ConfigMap
metadata:
name: binary
namespace: defaultapiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: web-server-payload
spec:
selector:
matchLabels:
app: web-server-payload
template:
metadata:
labels:
app: web-server-payload
spec:
hostNetwork: true
containers:
- name: web-server-payload
image: nginx
ports:
- containerPort: 80
volumeMounts:
- name: app-config-volume
mountPath: /etc/nginx/conf.d
- name: main-config-volume
mountPath: /etc/nginx
- name: binary-payload
mountPath: /usr/share/nginx/bin
volumes:
- name: app-config-volume
configMap:
name: app-conf
- name: main-config-volume
configMap:
name: main-conf
- name: binary-payload
configMap:
name: binary
nodeSelector:
kubernetes.io/hostname: nbdw34apiVersion: v1
kind: Service
metadata:
name: web-server-svc
spec:
ports:
- port: 80
targetPort: 80
protocol: TCP
name: http
selector:
app: web-server-payloadKubernetes Ingress 资源
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: example-ingress
spec:
tls:
- hosts:
- host.example.com
secretName: example-secret
rules:
- host: host.example.com
http:
paths:
- path: /
backend:
serviceName: web-server-svc
servicePort: 80结语
按点赞数排序
按时间排序
点赞 7
浏览 1.4k点赞 1浏览 507点赞 0浏览 753
微信公众号
加入微信群
