Skip to content

CPU manager with custom MachineConfigPool in OCP 4.x

How to apply the CPU manager to only a subset of worker nodes

Create a custom MachineConfigPool named worker-cpu-manager.

worker-cpu-manager.yaml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
  name: worker-cpu-manager
  labels:
    custom-kubelet: cpumanager-enabled
spec:
  machineConfigSelector:
    matchExpressions:
      - {key: machineconfiguration.openshift.io/role, operator: In, values: [worker,worker-cpu-manager]}
  nodeSelector:
    matchLabels:
      node-role.kubernetes.io/worker-cpu-manager: ""
  paused: false

Apply the pool:

1
oc apply -f worker-cpu-manager.yaml

Create the cpumanager-kubelet.yaml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
  name: cpumanager-enabled
spec:
  machineConfigPoolSelector:
    matchLabels:
      custom-kubelet: cpumanager-enabled
  kubeletConfig:
     cpuManagerPolicy: static
     cpuManagerReconcilePeriod: 5s

1
oc apply -f cpumanager-kubelet.yaml

Change worker node to worker-cpu-manager role:

1
oc label node <node> node-role.kubernetes.io/worker-cpu-manager=

Verify:

1
2
3
4
5
6
7
# oc get nodes
NAME                             STATUS   ROLES                       AGE     VERSION
openshift-master-0.example.com   Ready    master                      4h3m    v1.17.1+1aa1c48
openshift-master-1.example.com   Ready    master                      4h2m    v1.17.1+1aa1c48
openshift-master-2.example.com   Ready    master                      3h56m   v1.17.1+1aa1c48
openshift-worker-0.example.com   Ready    worker                      18m     v1.17.1+1aa1c48
openshift-worker-1.example.com   Ready    worker,worker-cpu-manager   18m     v1.17.1+1aa1c48

1
2
3
4
oc get machineconfig
(...)
rendered-worker-cpu-manager-bc48d7bf24df726f468b357482032845         8af4f709c4ba9c0afff3408ecc99c8fce61dd314   2.2.0             87s
rendered-worker-cpu-manager-ca0a09ddea41402490e1c39a138cd44e         8af4f709c4ba9c0afff3408ecc99c8fce61dd314   2.2.0             18m
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
[root@openshift-jumpserver-0 cpuman]# diff <(oc get machineconfig rendered-worker-cpu-manager-bc48d7bf24df726f468b357482032845 -o yaml)  <(oc get machineconfig rendered-worker-cpu-manager-ca0a09ddea41402490e1c39a138cd44e -o yaml)
6c6
<   creationTimestamp: "2020-06-26T16:40:44Z"
---
>   creationTimestamp: "2020-06-26T16:24:01Z"
8c8
<   name: rendered-worker-cpu-manager-bc48d7bf24df726f468b357482032845
---
>   name: rendered-worker-cpu-manager-ca0a09ddea41402490e1c39a138cd44e
16,18c16,18
<   resourceVersion: "451269"
<   selfLink: /apis/machineconfiguration.openshift.io/v1/machineconfigs/rendered-worker-cpu-manager-bc48d7bf24df726f468b357482032845
<   uid: bf950ef8-7270-4699-a113-dfe6c6c1d4fb
---
>   resourceVersion: "445466"
>   selfLink: /apis/machineconfiguration.openshift.io/v1/machineconfigs/rendered-worker-cpu-manager-ca0a09ddea41402490e1c39a138cd44e
>   uid: 74acdcd1-d195-48a5-8689-cecca137605c
163,168d162
<           verification: {}
<         filesystem: root
<         mode: 420
<         path: /etc/kubernetes/kubelet.conf
<       - contents:
<           source: data:text/plain,%7B%odcy%2(...)

Compare unmodified node to node with CPU manager:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
[root@openshift-jumpserver-0 cpuman]# oc debug node/openshift-worker-0.example.com
Starting pod/openshift-worker-0examplecom-debug ...
To use host binaries, run `chroot /host`
Pod IP: 192.168.123.215
If you don't see a command prompt, try pressing enter.
sh-4.2# cat /host/etc/kubernetes/kubelet.conf | grep cpuManager
sh-4.2# exit
exit

Removing debug pod ...

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
[root@openshift-jumpserver-0 cpuman]# oc debug node/openshift-worker-1.example.com
Starting pod/openshift-worker-1examplecom-debug ...
To use host binaries, run `chroot /host`
Pod IP: 192.168.123.204
If you don't see a command prompt, try pressing enter.
sh-4.2# cat /host/etc/kubernetes/kubelet.conf | grep cpuManager
{"kind":"KubeletConfiguration","apiVersion":"kubelet.config.k8s.io/v1beta1","staticPodPath":"/etc/kubernetes/manifests","syncFrequency":"0s","fileCheckFrequency":"0s","httpCheckFrequency":"0s","rotateCertificates":true,"serverTLSBootstrap":true,"authentication":{"x509":{"clientCAFile":"/etc/kubernetes/kubelet-ca.crt"},"webhook":{"cacheTTL":"0s"},"anonymous":{"enabled":false}},"authorization":{"webhook":{"cacheAuthorizedTTL":"0s","cacheUnauthorizedTTL":"0s"}},"clusterDomain":"cluster.local","clusterDNS":["172.30.0.10"],"streamingConnectionIdleTimeout":"0s","nodeStatusUpdateFrequency":"0s","nodeStatusReportFrequency":"0s","imageMinimumGCAge":"0s","volumeStatsAggPeriod":"0s","systemCgroups":"/system.slice","cgroupRoot":"/","cgroupDriver":"systemd","cpuManagerPolicy":"static","cpuManagerReconcilePeriod":"5s","runtimeRequestTimeout":"0s","maxPods":250,"kubeAPIQPS":50,"kubeAPIBurst":100,"serializeImagePulls":false,"evictionPressureTransitionPeriod":"0s","featureGates":{"LegacyNodeRoleBehavior":false,"NodeDisruptionExclusion":true,"RotateKubeletServerCertificate":true,"SCTPSupport":true,"ServiceNodeExclusion":true,"SupportPodPidsLimit":true},"containerLogMaxSize":"50Mi","systemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}}
sh-4.2# exit
exit

Removing debug pod ...

Spawn 2 pods on the same worker, one with and one without CPU manager enabled:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
[root@openshift-jumpserver-0 cpuman]# cat cpumanager-pod.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: cpumanager
spec:
  containers:
  - name: cpumanager
    image: gcr.io/google_containers/pause-amd64:3.0
    resources:
      requests:
        cpu: 1
        memory: "1G"
      limits:
        cpu: 1
        memory: "1G"
  nodeSelector:
    cpumanager: "true"
[root@openshift-jumpserver-0 cpuman]# cat non-cpumanager-pod.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: noncpumanager
spec:
  containers:
  - name: cpumanager
    image: gcr.io/google_containers/pause-amd64:3.0
[root@openshift-jumpserver-0 cpuman]# oc get pods -o wide
NAME            READY   STATUS    RESTARTS   AGE     IP           NODE                             NOMINATED NODE   READINESS GATES
cpumanager      1/1     Running   0          11m     172.24.2.4   openshift-worker-1.example.com   <none>           <none>
noncpumanager   1/1     Running   0          8m10s   172.24.2.5   openshift-worker-1.example.com   <none>           <none>

Verify CPU pinning for the pods' CPUs:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
[root@openshift-jumpserver-0 cpuman]# oc debug node/openshift-worker-1.example.com
Starting pod/openshift-worker-1examplecom-debug ...
To use host binaries, run `chroot /host`
systemctlPod IP: 192.168.123.204
If you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host
sh-4.4# ps aux | grep pause
root       39072  0.0  0.0   1028     4 ?        Ss   16:55   0:00 /pause
root       47750  0.0  0.0   1028     4 ?        Ss   16:58   0:00 /pause
root       64183  0.0  0.0   9180   960 ?        S+   17:03   0:00 grep pause
sh-4.4# cat /proc/39072/status | grep -i cpu
Cpus_allowed:   00,00100000
Cpus_allowed_list:  20
sh-4.4# cat /proc/47750/status | grep -i cpu
Cpus_allowed:   ff,ffefffff
Cpus_allowed_list:  0-19,21-39

Alternatively:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
sh-4.4# systemctl status 39072
 crio-ff43bfb551a274eb9e9040510753db6271ef27c13c203fe69e87aad5f7d49f17.scope - libcontainer container ff43bfb551a274eb9e9040510753db6271ef27c13c203fe69e87aad5f7d49f17
   Loaded: loaded (/run/systemd/transient/crio-ff43bfb551a274eb9e9040510753db6271ef27c13c203fe69e87aad5f7d49f17.scope; transient)
Transient: yes
   Active: active (running) since Fri 2020-06-26 16:55:34 UTC; 15min ago
    Tasks: 1 (limit: 1024)
   Memory: 1.4M (limit: 953.6M)
      CPU: 33ms
   CGroup: /kubepods.slice/kubepods-podc405f7cf_b2d1_49c4_bd4c_09f01a0b8e2d.slice/crio-ff43bfb551a274eb9e9040510753db6271ef27c13c203fe69e87aad5f7d49f17.scope
           └─39072 /pause

Jun 26 16:55:34 openshift-worker-1.example.com systemd[1]: Started libcontainer container ff43bfb551a274eb9e9040510753db6271ef27c13c203fe69e87aad5f7d49f17.
sh-4.4# systemctl status 47750
 crio-473fd6bdac78596e0dbcf0d33dc11e11285725b4cf4e55410ce70dbdf088d260.scope - libcontainer container 473fd6bdac78596e0dbcf0d33dc11e11285725b4cf4e55410ce70dbdf088d260
   Loaded: loaded (/run/systemd/transient/crio-473fd6bdac78596e0dbcf0d33dc11e11285725b4cf4e55410ce70dbdf088d260.scope; transient)
Transient: yes
   Active: active (running) since Fri 2020-06-26 16:58:32 UTC; 13min ago
    Tasks: 1 (limit: 1024)
   Memory: 1.0M
      CPU: 35ms
   CGroup: /kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b4059d0_8030_461a_b192_b5820f6c1119.slice/crio-473fd6bdac78596e0dbcf0d33dc11e11285725b4cf4e55410ce70dbdf088d260.scope
           └─47750 /pause

Jun 26 16:58:32 openshift-worker-1.example.com systemd[1]: Started libcontainer container 473fd6bdac78596e0dbcf0d33dc11e11285725b4cf4e55410ce70dbdf088d260.
sh-4.4# cat /sys/fs/cgroup/   
blkio/            cpu,cpuacct/      cpuset/           freezer/          memory/           net_cls,net_prio/ perf_event/       rdma/             
cpu/              cpuacct/          devices/          hugetlb/          net_cls/          net_prio/         pids/             systemd/          
sh-4.4# cat /sys/fs/cgroup/cpuset//kubepods.slice/kubepods-podc405f7cf_b2d1_49c4_bd4c_09f01a0b8e2d.slice/crio-ff43bfb551a274eb9e9040510753db6271ef27c13c203fe69e87aad5f7d49f17.scope/cpuset.cpus
20
sh-4.4# cat /sys/fs/cgroup/cpuset//kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b4059d0_8030_461a_b192_b5820f6c1119.slice/crio-473fd6bdac78596e0dbcf0d33dc11e11285725b4cf4e55410ce70dbdf088d260.scope/cpuset.pus
0-19,21-39

Exploring resource limits

Let's spawn more pods than the hypervisor has CPUs:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
[root@openshift-jumpserver-0 cpuman]# cat cpumanager-pod-generated-name.yaml 
apiVersion: v1
kind: Pod
metadata:
  generateName: cpumanager-
spec:
  containers:
  - name: cpumanager
    image: gcr.io/google_containers/pause-amd64:3.0
    resources:
      requests:
        cpu: 1
        memory: "1G"
      limits:
        cpu: 1
        memory: "1G"
  nodeSelector:
    cpumanager: "true"

1
for i in {0..45}; do oc create -f cpumanager-pod-generated-name.yaml ; done
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
[root@openshift-jumpserver-0 cpuman]# oc get pods | grep Pending
cpumanager-8rtpt   0/1     Pending   0          4m23s
cpumanager-bw848   0/1     Pending   0          4m24s
cpumanager-cm2st   0/1     Pending   0          4m24s
cpumanager-cwj82   0/1     Pending   0          4m23s
cpumanager-r5krw   0/1     Pending   0          4m24s
[root@openshift-jumpserver-0 cpuman]# oc describe node openshift-worker-1.example.com
Name:               openshift-worker-1.example.com
Roles:              worker-cpu-manager
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    cpumanager=true
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=openshift-worker-1.example.com
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/worker-cpu-manager=
                    node.openshift.io/os_id=rhcos
Annotations:        k8s.ovn.org/l3-gateway-config:
                      {"default":{"mode":"local","interface-id":"br-local_openshift-worker-1.example.com","mac-address":"8e:c7:71:4d:0f:49","ip-addresses":["169...
                    k8s.ovn.org/node-chassis-id: 5a8880a6-5b50-4be5-9d84-f195bbc306a2
                    k8s.ovn.org/node-join-subnets: {"default":"100.64.3.0/29"}
                    k8s.ovn.org/node-mgmt-port-mac-address: 9a:b7:cd:ab:bf:b9
                    k8s.ovn.org/node-subnets: {"default":"172.24.2.0/23"}
                    machineconfiguration.openshift.io/currentConfig: rendered-worker-cpu-manager-bc48d7bf24df726f468b357482032845
                    machineconfiguration.openshift.io/desiredConfig: rendered-worker-cpu-manager-bc48d7bf24df726f468b357482032845
                    machineconfiguration.openshift.io/reason: 
                    machineconfiguration.openshift.io/state: Done
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Thu, 25 Jun 2020 12:03:55 -0400
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  openshift-worker-1.example.com
  AcquireTime:     <unset>
  RenewTime:       Fri, 26 Jun 2020 14:14:05 -0400
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Fri, 26 Jun 2020 14:10:56 -0400   Fri, 26 Jun 2020 12:43:22 -0400   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Fri, 26 Jun 2020 14:10:56 -0400   Fri, 26 Jun 2020 12:43:22 -0400   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Fri, 26 Jun 2020 14:10:56 -0400   Fri, 26 Jun 2020 12:43:22 -0400   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Fri, 26 Jun 2020 14:10:56 -0400   Fri, 26 Jun 2020 12:43:33 -0400   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:  192.168.123.204
  Hostname:    openshift-worker-1.example.com
Capacity:
  cpu:                40
  ephemeral-storage:  584946668Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             131924236Ki
  pods:               250
Allocatable:
  cpu:                39500m
  ephemeral-storage:  538013106513
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             130773260Ki
  pods:               250
System Info:
  Machine ID:                             21668e85e1264ac78ea115b2fe79408e
  System UUID:                            4c4c4544-004b-5a10-8050-cac04f484832
  Boot ID:                                f20ee17e-226f-4f34-b9ce-965043215c2d
  Kernel Version:                         4.18.0-147.8.1.el8_1.x86_64
  OS Image:                               Red Hat Enterprise Linux CoreOS 44.81.202005250830-0 (Ootpa)
  Operating System:                       linux
  Architecture:                           amd64
  Container Runtime Version:              cri-o://1.17.4-12.dev.rhaos4.4.git2be4d9c.el8
  Kubelet Version:                        v1.17.1
  Kube-Proxy Version:                     v1.17.1
Non-terminated Pods:                      (49 in total)
  Namespace                               Name                                    CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE
  ---------                               ----                                    ------------  ----------  ---------------  -------------  ---
  default                                 cpumanager                              1 (2%)        1 (2%)      1G (0%)          1G (0%)        78m
  default                                 cpumanager-2548b                        1 (2%)        1 (2%)      1G (0%)          1G (0%)        7m41s
  default                                 cpumanager-44nxg                        1 (2%)        1 (2%)      1G (0%)          1G (0%)        4m29s
  default                                 cpumanager-5zx5v                        1 (2%)        1 (2%)      1G (0%)          1G (0%)        4m30s
  default                                 cpumanager-72spc                        1 (2%)        1 (2%)      1G (0%)          1G (0%)        4m28s
  default                                 cpumanager-9754w                        1 (2%)        1 (2%)      1G (0%)          1G (0%)        4m29s
  default                                 cpumanager-cff7t                        1 (2%)        1 (2%)      1G (0%)          1G (0%)        8m6s
  default                                 cpumanager-djhz9                        1 (2%)        1 (2%)      1G (0%)          1G (0%)        4m27s
  default                                 cpumanager-dl7lf                        1 (2%)        1 (2%)      1G (0%)          1G (0%)        7m42s
  default                                 cpumanager-dnz7k                        1 (2%)        1 (2%)      1G (0%)          1G (0%)        4m28s
  default                                 cpumanager-drhf8                        1 (2%)        1 (2%)      1G (0%)          1G (0%)        4m27s
  default                                 cpumanager-ff7h5                        1 (2%)        1 (2%)      1G (0%)          1G (0%)        4m30s
  default                                 cpumanager-fh77s                        1 (2%)        1 (2%)      1G (0%)          1G (0%)        7m40s
  default                                 cpumanager-fsptg                        1 (2%)        1 (2%)      1G (0%)          1G (0%)        4m30s
  default                                 cpumanager-h74fq                        1 (2%)        1 (2%)      1G (0%)          1G (0%)        7m38s
  default                                 cpumanager-hb72w                        1 (2%)        1 (2%)      1G (0%)          1G (0%)        4m28s
  default                                 cpumanager-hbhz2                        1 (2%)        1 (2%)      1G (0%)          1G (0%)        4m29s
  default                                 cpumanager-j7smp                        1 (2%)        1 (2%)      1G (0%)          1G (0%)        4m28s
  default                                 cpumanager-jlwwn                        1 (2%)        1 (2%)      1G (0%)          1G (0%)        4m30s
  default                                 cpumanager-jmdfk                        1 (2%)        1 (2%)      1G (0%)          1G (0%)        4m30s
  default                                 cpumanager-kvfkp                        1 (2%)        1 (2%)      1G (0%)          1G (0%)        7m45s
  default                                 cpumanager-kx4ft                        1 (2%)        1 (2%)      1G (0%)          1G (0%)        4m28s
  default                                 cpumanager-kz6zg                        1 (2%)        1 (2%)      1G (0%)          1G (0%)        7m43s
  default                                 cpumanager-m4qrn                        1 (2%)        1 (2%)      1G (0%)          1G (0%)        4m31s
  default                                 cpumanager-mzqqs                        1 (2%)        1 (2%)      1G (0%)          1G (0%)        4m30s
  default                                 cpumanager-n2tt5                        1 (2%)        1 (2%)      1G (0%)          1G (0%)        4m28s
  default                                 cpumanager-n79js                        1 (2%)        1 (2%)      1G (0%)          1G (0%)        4m31s
  default                                 cpumanager-qwjkf                        1 (2%)        1 (2%)      1G (0%)          1G (0%)        4m27s
  default                                 cpumanager-sn658                        1 (2%)        1 (2%)      1G (0%)          1G (0%)        4m29s
  default                                 cpumanager-tc7q6                        1 (2%)        1 (2%)      1G (0%)          1G (0%)        4m29s
  default                                 cpumanager-vss6b                        1 (2%)        1 (2%)      1G (0%)          1G (0%)        4m29s
  default                                 cpumanager-w846n                        1 (2%)        1 (2%)      1G (0%)          1G (0%)        4m31s
  default                                 cpumanager-wchfq                        1 (2%)        1 (2%)      1G (0%)          1G (0%)        4m30s
  default                                 cpumanager-whwxr                        1 (2%)        1 (2%)      1G (0%)          1G (0%)        4m28s
  default                                 cpumanager-wkhw8                        1 (2%)        1 (2%)      1G (0%)          1G (0%)        4m31s
  default                                 cpumanager-wtr5z                        1 (2%)        1 (2%)      1G (0%)          1G (0%)        4m31s
  default                                 cpumanager-z7zgl                        1 (2%)        1 (2%)      1G (0%)          1G (0%)        4m28s
  default                                 cpumanager-zckx8                        1 (2%)        1 (2%)      1G (0%)          1G (0%)        4m30s
  default                                 cpumanager-zsnwv                        1 (2%)        1 (2%)      1G (0%)          1G (0%)        4m27s
  default                                 noncpumanager                           0 (0%)        0 (0%)      0 (0%)           0 (0%)         75m
  openshift-cluster-node-tuning-operator  tuned-tkhrp                             10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         26h
  openshift-dns                           dns-default-q8hww                       110m (0%)     0 (0%)      70Mi (0%)        512Mi (0%)     26h
  openshift-image-registry                node-ca-sfjgq                           10m (0%)      0 (0%)      10Mi (0%)        0 (0%)         26h
  openshift-machine-config-operator       machine-config-daemon-8nw2s             40m (0%)      0 (0%)      100Mi (0%)       0 (0%)         26h
  openshift-marketplace                   certified-operators-74d989c4dd-w6l7f    10m (0%)      0 (0%)      100Mi (0%)       0 (0%)         50m
  openshift-monitoring                    node-exporter-nlgr5                     9m (0%)       0 (0%)      210Mi (0%)       0 (0%)         26h
  openshift-multus                        multus-w2sgc                            10m (0%)      0 (0%)      150Mi (0%)       0 (0%)         26h
  openshift-ovn-kubernetes                ovnkube-node-pg57x                      200m (0%)     0 (0%)      600Mi (0%)       0 (0%)         26h
  openshift-ovn-kubernetes                ovs-node-ktvtm                          100m (0%)     0 (0%)      300Mi (0%)       0 (0%)         26h
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests           Limits
  --------           --------           ------
  cpu                39499m (99%)       39 (98%)
  memory             40667235840 (30%)  39536870912 (29%)
  ephemeral-storage  0 (0%)             0 (0%)
Events:
  Type     Reason                   Age                  From                                     Message
  ----     ------                   ----                 ----                                     -------
  Normal   NodeNotSchedulable       110m                 kubelet, openshift-worker-1.example.com  Node openshift-worker-1.example.com status is now: NodeNotSchedulable
  Normal   Starting                 106m                 kubelet, openshift-worker-1.example.com  Starting kubelet.
  Normal   NodeHasSufficientMemory  106m (x2 over 106m)  kubelet, openshift-worker-1.example.com  Node openshift-worker-1.example.com status is now: NodeHasSufficientMemory
  Normal   NodeHasNoDiskPressure    106m (x2 over 106m)  kubelet, openshift-worker-1.example.com  Node openshift-worker-1.example.com status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID     106m (x2 over 106m)  kubelet, openshift-worker-1.example.com  Node openshift-worker-1.example.com status is now: NodeHasSufficientPID
  Warning  Rebooted                 106m                 kubelet, openshift-worker-1.example.com  Node openshift-worker-1.example.com has been rebooted, boot id: 7df19e71-4ebc-4a66-94e8-1e475d11e095
  Normal   NodeNotReady             106m                 kubelet, openshift-worker-1.example.com  Node openshift-worker-1.example.com status is now: NodeNotReady
  Normal   NodeNotSchedulable       106m                 kubelet, openshift-worker-1.example.com  Node openshift-worker-1.example.com status is now: NodeNotSchedulable
  Normal   NodeAllocatableEnforced  106m                 kubelet, openshift-worker-1.example.com  Updated Node Allocatable limit across pods
  Normal   NodeReady                106m                 kubelet, openshift-worker-1.example.com  Node openshift-worker-1.example.com status is now: NodeReady
  Normal   NodeSchedulable          100m                 kubelet, openshift-worker-1.example.com  Node openshift-worker-1.example.com status is now: NodeSchedulable
  Normal   Starting                 90m                  kubelet, openshift-worker-1.example.com  Starting kubelet.
  Normal   NodeHasSufficientMemory  90m (x2 over 90m)    kubelet, openshift-worker-1.example.com  Node openshift-worker-1.example.com status is now: NodeHasSufficientMemory
  Normal   NodeHasNoDiskPressure    90m (x2 over 90m)    kubelet, openshift-worker-1.example.com  Node openshift-worker-1.example.com status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID     90m (x2 over 90m)    kubelet, openshift-worker-1.example.com  Node openshift-worker-1.example.com status is now: NodeHasSufficientPID
  Warning  Rebooted                 90m                  kubelet, openshift-worker-1.example.com  Node openshift-worker-1.example.com has been rebooted, boot id: f20ee17e-226f-4f34-b9ce-965043215c2d
  Normal   NodeNotReady             90m                  kubelet, openshift-worker-1.example.com  Node openshift-worker-1.example.com status is now: NodeNotReady
  Normal   NodeNotSchedulable       90m                  kubelet, openshift-worker-1.example.com  Node openshift-worker-1.example.com status is now: NodeNotSchedulable
  Normal   NodeAllocatableEnforced  90m                  kubelet, openshift-worker-1.example.com  Updated Node Allocatable limit across pods
  Normal   NodeReady                90m                  kubelet, openshift-worker-1.example.com  Node openshift-worker-1.example.com status is now: NodeReady
  Normal   NodeSchedulable          85m                  kubelet, openshift-worker-1.example.com  Node openshift-worker-1.example.com status is now: NodeSchedulable

Verify processes and cgroup limits on the worker node:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
[root@openshift-jumpserver-0 cpuman]# oc debug node/openshift-worker-1.example.com
Starting pod/openshift-worker-1examplecom-debug ...
To use host binaries, run `chroot /host`
Pod IP: 192.168.123.204
If you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host
sh-4.4# ps aux | grep pause
root       39072  0.0  0.0   1028     4 ?        Ss   16:55   0:00 /pause
root       47750  0.0  0.0   1028     4 ?        Ss   16:58   0:00 /pause
root      274649  0.0  0.0   1028     4 ?        Ss   18:06   0:00 /pause
root      276349  0.0  0.0   1028     4 ?        Ss   18:06   0:00 /pause
root      276526  0.0  0.0   1028     4 ?        Ss   18:06   0:00 /pause
root      276685  0.0  0.0   1028     4 ?        Ss   18:06   0:00 /pause
root      276824  0.0  0.0   1028     4 ?        Ss   18:06   0:00 /pause
root      276988  0.0  0.0   1028     4 ?        Ss   18:06   0:00 /pause
root      277279  0.0  0.0   1028     4 ?        Ss   18:06   0:00 /pause
root      291552  0.0  0.0   1028     4 ?        Ss   18:09   0:00 /pause
root      291674  0.0  0.0   1028     4 ?        Ss   18:09   0:00 /pause
root      291783  0.0  0.0   1028     4 ?        Ss   18:09   0:00 /pause
root      291900  0.0  0.0   1028     4 ?        Ss   18:09   0:00 /pause
root      292072  0.0  0.0   1028     4 ?        Ss   18:09   0:00 /pause
root      292145  0.0  0.0   1028     4 ?        Ss   18:09   0:00 /pause
root      292191  0.0  0.0   1028     4 ?        Ss   18:09   0:00 /pause
root      292638  0.0  0.0   1028     4 ?        Ss   18:09   0:00 /pause
root      292639  0.0  0.0   1028     4 ?        Ss   18:09   0:00 /pause
root      292693  0.0  0.0   1028     4 ?        Ss   18:09   0:00 /pause
root      292941  0.0  0.0   1028     4 ?        Ss   18:09   0:00 /pause
root      292949  0.0  0.0   1028     4 ?        Ss   18:09   0:00 /pause
root      292971  0.0  0.0   1028     4 ?        Ss   18:09   0:00 /pause
root      292986  0.0  0.0   1028     4 ?        Ss   18:09   0:00 /pause
root      293300  0.0  0.0   1028     4 ?        Ss   18:09   0:00 /pause
root      293422  0.0  0.0   1028     4 ?        Ss   18:09   0:00 /pause
root      293528  0.0  0.0   1028     4 ?        Ss   18:09   0:00 /pause
root      293615  0.0  0.0   1028     4 ?        Ss   18:09   0:00 /pause
root      293678  0.0  0.0   1028     4 ?        Ss   18:09   0:00 /pause
root      293823  0.0  0.0   1028     4 ?        Ss   18:09   0:00 /pause
root      293909  0.0  0.0   1028     4 ?        Ss   18:09   0:00 /pause
root      293951  0.0  0.0   1028     4 ?        Ss   18:09   0:00 /pause
root      293965  0.0  0.0   1028     4 ?        Ss   18:09   0:00 /pause
root      293983  0.0  0.0   1028     4 ?        Ss   18:09   0:00 /pause
root      294087  0.0  0.0   1028     4 ?        Ss   18:09   0:00 /pause
root      294164  0.0  0.0   1028     4 ?        Ss   18:09   0:00 /pause
root      294184  0.0  0.0   1028     4 ?        Ss   18:09   0:00 /pause
root      294239  0.0  0.0   1028     4 ?        Ss   18:09   0:00 /pause
root      294330  0.0  0.0   1028     4 ?        Ss   18:09   0:00 /pause
root      294352  0.0  0.0   1028     4 ?        Ss   18:09   0:00 /pause
root      294426  0.0  0.0   1028     4 ?        Ss   18:09   0:00 /pause
root      301832  0.0  0.0   9180   964 ?        R+   18:11   0:00 grep pause
sh-4.4# systemctl status 47750
 crio-473fd6bdac78596e0dbcf0d33dc11e11285725b4cf4e55410ce70dbdf088d260.scope - libcontainer container 473fd6bdac78596e0dbcf0d33dc11e11285725b4cf4e55410ce70dbdf088d260
   Loaded: loaded (/run/systemd/transient/crio-473fd6bdac78596e0dbcf0d33dc11e11285725b4cf4e55410ce70dbdf088d260.scope; transient)
Transient: yes
   Active: active (running) since Fri 2020-06-26 16:58:32 UTC; 1h 12min ago
    Tasks: 1 (limit: 1024)
   Memory: 1.0M
      CPU: 35ms
   CGroup: /kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b4059d0_8030_461a_b192_b5820f6c1119.slice/crio-473fd6bdac78596e0dbcf0d33dc11e11285725b4cf4e55410ce70dbdf088d260.scope
           └─47750 /pause

Jun 26 16:58:32 openshift-worker-1.example.com systemd[1]: Started libcontainer container 473fd6bdac78596e0dbcf0d33dc11e11285725b4cf4e55410ce70dbdf088d260.
sh-4.4# cat /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b4059d0_8030_461a_b192_b5820f6c1119.slice/crio-473fd6bdac78596e0dbcf0d33dc11e11285725b4cf4e55410ce70dbdf088d260.scope/cpuset.cpus
0
sh-4.4# exit
exit
sh-4.2# exit
exit

Removing debug pod ...

Default host limits:

1
2
3
4
5
6
sh-4.4# cat /etc/kubernetes/kubelet.conf | jq '.systemReserved'
{
  "cpu": "500m",
  "ephemeral-storage": "1Gi",
  "memory": "1Gi"
}

Applying custom limits

Let's now reserve 10 CPUs for the host.

cpumanager-kubeletconfig.yaml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
  name: cpumanager-enabled
spec:
  machineConfigPoolSelector:
    matchLabels:
      custom-kubelet: cpumanager-enabled
  kubeletConfig:
     cpuManagerPolicy: static
     cpuManagerReconcilePeriod: 5s
     systemReserved:
       cpu: "10"
       memory: "1Gi"
       ephemeral-storage: "10Gi"

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
[root@openshift-jumpserver-0 cpuman]# oc apply -f cpumanager-kubeletconfig.yaml 
kubeletconfig.machineconfiguration.openshift.io/cpumanager-enabled configured
[root@openshift-jumpserver-0 cpuman]# oc get -o yaml -f cpumanager-kubeletconfig.yaml 
apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"machineconfiguration.openshift.io/v1","kind":"KubeletConfig","metadata":{"annotations":{},"name":"cpumanager-enabled"},"spec":{"kubeletConfig":{"cpuManagerPolicy":"static","cpuManagerReconcilePeriod":"5s","systemReserved":{"cpu":"10","ephemeral-storage":"10Gi","memory":"1Gi"}},"machineConfigPoolSelector":{"matchLabels":{"custom-kubelet":"cpumanager-enabled"}}}}
  creationTimestamp: "2020-06-26T18:29:07Z"
  finalizers:
  - b5ac419d-573d-428c-adc1-f7b5bedb27e3
  - 21fa7ce4-49fa-470d-b4ed-f0972dbdd040
  generation: 2
  name: cpumanager-enabled
  resourceVersion: "484854"
  selfLink: /apis/machineconfiguration.openshift.io/v1/kubeletconfigs/cpumanager-enabled
  uid: 75d54d68-7b96-4b9e-8c9a-db3820b2629b
spec:
  kubeletConfig:
    cpuManagerPolicy: static
    cpuManagerReconcilePeriod: 5s
    systemReserved:
      cpu: "10"
      ephemeral-storage: 10Gi
      memory: 1Gi
  machineConfigPoolSelector:
    matchLabels:
      custom-kubelet: cpumanager-enabled
status:
  conditions:
  - lastTransitionTime: "2020-06-26T18:29:07Z"
    message: Success
    status: "True"
    type: Success
  - lastTransitionTime: "2020-06-26T18:33:14Z"
    message: Success
    status: "True"
    type: Success
[root@openshift-jumpserver-0 cpuman]# oc get nodes
NAME                             STATUS                     ROLES                AGE   VERSION
openshift-master-0.example.com   Ready                      master               27h   v1.17.1
openshift-master-1.example.com   Ready                      master               27h   v1.17.1
openshift-master-2.example.com   Ready                      master               27h   v1.17.1
openshift-worker-0.example.com   Ready                      worker               26h   v1.17.1
openshift-worker-1.example.com   Ready,SchedulingDisabled   worker-cpu-manager   26h   v1.17.1
[root@openshift-jumpserver-0 cpuman]# oc get machineconfig | grep worker-cpu-manager
99-worker-cpu-manager-ab32b145-ada3-4f96-adf9-1fb8388ba183-kubelet   8af4f709c4ba9c0afff3408ecc99c8fce61dd314   2.2.0             4m42s
rendered-worker-cpu-manager-171f3675f101028b058ffe27d6344bb2         8af4f709c4ba9c0afff3408ecc99c8fce61dd314   2.2.0             30s
rendered-worker-cpu-manager-bc48d7bf24df726f468b357482032845         8af4f709c4ba9c0afff3408ecc99c8fce61dd314   2.2.0             113m
rendered-worker-cpu-manager-ca0a09ddea41402490e1c39a138cd44e         8af4f709c4ba9c0afff3408ecc99c8fce61dd314   2.2.0             129m
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
[root@openshift-jumpserver-0 cpuman]# for i in {0..35}; do oc create -f cpumanager-pod-generated-name.yaml ; done
pod/cpumanager-msf8c created
pod/cpumanager-2kwws created
pod/cpumanager-7mx5k created
pod/cpumanager-bq7sf created
pod/cpumanager-86kkl created
pod/cpumanager-z7h5f created
pod/cpumanager-c5pd4 created
pod/cpumanager-sr6jt created
pod/cpumanager-dp7ck created
pod/cpumanager-n4ffv created
pod/cpumanager-5q77h created
pod/cpumanager-zxbj2 created
pod/cpumanager-7f8fw created
pod/cpumanager-j5kdw created
pod/cpumanager-fhfmw created
pod/cpumanager-hz5r8 created
pod/cpumanager-44xst created
pod/cpumanager-r6h98 created
pod/cpumanager-mjsfq created
pod/cpumanager-dh897 created
pod/cpumanager-s445q created
pod/cpumanager-brwcz created
pod/cpumanager-mnbrg created
pod/cpumanager-56vmv created
pod/cpumanager-kt5fg created
pod/cpumanager-fscpd created
pod/cpumanager-hsjzn created
pod/cpumanager-cvsth created
pod/cpumanager-j7wlv created
pod/cpumanager-z87tp created
pod/cpumanager-xggjm created
pod/cpumanager-jg646 created
pod/cpumanager-ml7kg created
pod/cpumanager-qwhjm created
pod/cpumanager-m6jw8 created
pod/cpumanager-f8qsh created
[root@openshift-jumpserver-0 cpuman]# oc get pods | grep Runn
cpumanager-2kwws   1/1     Running   0          15s
cpumanager-44xst   1/1     Running   0          13s
cpumanager-56vmv   1/1     Running   0          12s
cpumanager-5q77h   1/1     Running   0          14s
cpumanager-7f8fw   1/1     Running   0          14s
cpumanager-7mx5k   1/1     Running   0          15s
cpumanager-86kkl   1/1     Running   0          15s
cpumanager-8rtpt   1/1     Running   0          32m
cpumanager-bq7sf   1/1     Running   0          15s
cpumanager-brwcz   1/1     Running   0          13s
cpumanager-bw848   1/1     Running   0          32m
cpumanager-c5pd4   1/1     Running   0          14s
cpumanager-cm2st   1/1     Running   0          32m
cpumanager-cwj82   1/1     Running   0          32m
cpumanager-dh897   1/1     Running   0          13s
cpumanager-dp7ck   1/1     Running   0          14s
cpumanager-fhfmw   1/1     Running   0          13s
cpumanager-hz5r8   1/1     Running   0          13s
cpumanager-j5kdw   1/1     Running   0          14s
cpumanager-mjsfq   1/1     Running   0          13s
cpumanager-mnbrg   1/1     Running   0          12s
cpumanager-msf8c   1/1     Running   0          15s
cpumanager-n4ffv   1/1     Running   0          14s
cpumanager-r5krw   1/1     Running   0          32m
cpumanager-r6h98   1/1     Running   0          13s
cpumanager-s445q   1/1     Running   0          13s
cpumanager-sr6jt   1/1     Running   0          14s
cpumanager-z7h5f   1/1     Running   0          15s
cpumanager-zxbj2   1/1     Running   0          14s
[root@openshift-jumpserver-0 cpuman]# oc get pods | grep Runn | wc -l
29

Verify host limits:

1
2
3
4
5
6
sh-4.4# cat /etc/kubernetes/kubelet.conf | jq '.systemReserved'
{
  "cpu": "10",
  "ephemeral-storage": "10Gi",
  "memory": "1Gi"
}

Reserving 0 CPUs for the host

This will not work - cpumanager-kubeletconfig.yaml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
  name: cpumanager-enabled
spec:
  machineConfigPoolSelector:
    matchLabels:
      custom-kubelet: cpumanager-enabled
  kubeletConfig:
     cpuManagerPolicy: static
     cpuManagerReconcilePeriod: 5s
     systemReserved:
       cpu: "0"
       memory: "1Gi"
       ephemeral-storage: "10Gi"

Upon worker restart, the kubelet will not be able to start, reporting:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
[root@openshift-worker-1 ~]# Jun 26 19:05:10 openshift-worker-1.example.com systemd[1]: Stopped Kubernetes Kubelet.
Jun 26 19:05:10 openshift-worker-1.example.com systemd[1]: kubelet.service: Consumed 115ms CPU time
Jun 26 19:05:10 openshift-worker-1.example.com systemd[1]: Starting Kubernetes Kubelet...
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421694    5492 flags.go:33] FLAG: --add-dir-header="false"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421726    5492 flags.go:33] FLAG: --address="0.0.0.0"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421741    5492 flags.go:33] FLAG: --allowed-unsafe-sysctls="[]"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421748    5492 flags.go:33] FLAG: --alsologtostderr="false"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421752    5492 flags.go:33] FLAG: --anonymous-auth="true"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421756    5492 flags.go:33] FLAG: --application-metrics-count-limit="100"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421759    5492 flags.go:33] FLAG: --authentication-token-webhook="false"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421763    5492 flags.go:33] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421767    5492 flags.go:33] FLAG: --authorization-mode="AlwaysAllow"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421771    5492 flags.go:33] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421775    5492 flags.go:33] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421778    5492 flags.go:33] FLAG: --azure-container-registry-config=""
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421781    5492 flags.go:33] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421787    5492 flags.go:33] FLAG: --bootstrap-checkpoint-path=""
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421790    5492 flags.go:33] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421793    5492 flags.go:33] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421797    5492 flags.go:33] FLAG: --cgroup-driver="cgroupfs"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421800    5492 flags.go:33] FLAG: --cgroup-root=""
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421803    5492 flags.go:33] FLAG: --cgroups-per-qos="true"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421806    5492 flags.go:33] FLAG: --chaos-chance="0"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421811    5492 flags.go:33] FLAG: --client-ca-file=""
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421814    5492 flags.go:33] FLAG: --cloud-config=""
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421817    5492 flags.go:33] FLAG: --cloud-provider=""
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421820    5492 flags.go:33] FLAG: --cluster-dns="[]"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421824    5492 flags.go:33] FLAG: --cluster-domain=""
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421828    5492 flags.go:33] FLAG: --cni-bin-dir="/opt/cni/bin"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421831    5492 flags.go:33] FLAG: --cni-cache-dir="/var/lib/cni/cache"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421834    5492 flags.go:33] FLAG: --cni-conf-dir="/etc/cni/net.d"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421838    5492 flags.go:33] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421842    5492 flags.go:33] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421845    5492 flags.go:33] FLAG: --container-log-max-files="5"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421849    5492 flags.go:33] FLAG: --container-log-max-size="10Mi"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421852    5492 flags.go:33] FLAG: --container-runtime="remote"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421865    5492 flags.go:33] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421868    5492 flags.go:33] FLAG: --containerd="/run/containerd/containerd.sock"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421872    5492 flags.go:33] FLAG: --contention-profiling="false"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421876    5492 flags.go:33] FLAG: --cpu-cfs-quota="true"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421879    5492 flags.go:33] FLAG: --cpu-cfs-quota-period="100ms"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421883    5492 flags.go:33] FLAG: --cpu-manager-policy="none"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421886    5492 flags.go:33] FLAG: --cpu-manager-reconcile-period="10s"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421889    5492 flags.go:33] FLAG: --docker="unix:///var/run/docker.sock"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421893    5492 flags.go:33] FLAG: --docker-endpoint="unix:///var/run/docker.sock"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421896    5492 flags.go:33] FLAG: --docker-env-metadata-whitelist=""
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421899    5492 flags.go:33] FLAG: --docker-only="false"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421902    5492 flags.go:33] FLAG: --docker-root="/var/lib/docker"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421905    5492 flags.go:33] FLAG: --docker-tls="false"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421908    5492 flags.go:33] FLAG: --docker-tls-ca="ca.pem"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421911    5492 flags.go:33] FLAG: --docker-tls-cert="cert.pem"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421914    5492 flags.go:33] FLAG: --docker-tls-key="key.pem"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421917    5492 flags.go:33] FLAG: --dynamic-config-dir=""
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421921    5492 flags.go:33] FLAG: --enable-cadvisor-json-endpoints="true"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421924    5492 flags.go:33] FLAG: --enable-controller-attach-detach="true"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421929    5492 flags.go:33] FLAG: --enable-debugging-handlers="true"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421932    5492 flags.go:33] FLAG: --enable-load-reader="false"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421951    5492 flags.go:33] FLAG: --enable-server="true"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421954    5492 flags.go:33] FLAG: --enforce-node-allocatable="[pods]"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421958    5492 flags.go:33] FLAG: --event-burst="10"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421961    5492 flags.go:33] FLAG: --event-qps="5"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421963    5492 flags.go:33] FLAG: --event-storage-age-limit="default=0"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421966    5492 flags.go:33] FLAG: --event-storage-event-limit="default=0"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421969    5492 flags.go:33] FLAG: --eviction-hard="imagefs.available<15%,memory.available<100Mi,nodefs.available<10%,nodefs.inodesFree<5%"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421979    5492 flags.go:33] FLAG: --eviction-max-pod-grace-period="0"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421982    5492 flags.go:33] FLAG: --eviction-minimum-reclaim=""
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421986    5492 flags.go:33] FLAG: --eviction-pressure-transition-period="5m0s"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.421990    5492 flags.go:33] FLAG: --eviction-soft=""
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422000    5492 flags.go:33] FLAG: --eviction-soft-grace-period=""
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422003    5492 flags.go:33] FLAG: --exit-on-lock-contention="false"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422006    5492 flags.go:33] FLAG: --experimental-allocatable-ignore-eviction="false"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422009    5492 flags.go:33] FLAG: --experimental-bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422012    5492 flags.go:33] FLAG: --experimental-check-node-capabilities-before-mount="false"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422015    5492 flags.go:33] FLAG: --experimental-dockershim="false"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422018    5492 flags.go:33] FLAG: --experimental-dockershim-root-directory="/var/lib/dockershim"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422021    5492 flags.go:33] FLAG: --experimental-kernel-memcg-notification="false"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422024    5492 flags.go:33] FLAG: --experimental-mounter-path=""
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422027    5492 flags.go:33] FLAG: --fail-swap-on="true"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422030    5492 flags.go:33] FLAG: --feature-gates=""
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422034    5492 flags.go:33] FLAG: --file-check-frequency="20s"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422037    5492 flags.go:33] FLAG: --global-housekeeping-interval="1m0s"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422040    5492 flags.go:33] FLAG: --hairpin-mode="promiscuous-bridge"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422043    5492 flags.go:33] FLAG: --healthz-bind-address="127.0.0.1"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422046    5492 flags.go:33] FLAG: --healthz-port="10248"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422049    5492 flags.go:33] FLAG: --help="false"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422052    5492 flags.go:33] FLAG: --hostname-override=""
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422055    5492 flags.go:33] FLAG: --housekeeping-interval="10s"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422058    5492 flags.go:33] FLAG: --http-check-frequency="20s"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422060    5492 flags.go:33] FLAG: --image-gc-high-threshold="85"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422063    5492 flags.go:33] FLAG: --image-gc-low-threshold="80"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422066    5492 flags.go:33] FLAG: --image-pull-progress-deadline="1m0s"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422069    5492 flags.go:33] FLAG: --image-service-endpoint=""
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422072    5492 flags.go:33] FLAG: --iptables-drop-bit="15"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422075    5492 flags.go:33] FLAG: --iptables-masquerade-bit="14"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422078    5492 flags.go:33] FLAG: --keep-terminated-pod-volumes="false"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422081    5492 flags.go:33] FLAG: --kube-api-burst="10"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422084    5492 flags.go:33] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422087    5492 flags.go:33] FLAG: --kube-api-qps="5"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422090    5492 flags.go:33] FLAG: --kube-reserved=""
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422093    5492 flags.go:33] FLAG: --kube-reserved-cgroup=""
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422103    5492 flags.go:33] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422106    5492 flags.go:33] FLAG: --kubelet-cgroups=""
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422109    5492 flags.go:33] FLAG: --lock-file=""
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422112    5492 flags.go:33] FLAG: --log-backtrace-at=":0"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422115    5492 flags.go:33] FLAG: --log-cadvisor-usage="false"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422118    5492 flags.go:33] FLAG: --log-dir=""
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422121    5492 flags.go:33] FLAG: --log-file=""
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422124    5492 flags.go:33] FLAG: --log-file-max-size="1800"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422127    5492 flags.go:33] FLAG: --log-flush-frequency="5s"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422130    5492 flags.go:33] FLAG: --logtostderr="true"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422133    5492 flags.go:33] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422136    5492 flags.go:33] FLAG: --make-iptables-util-chains="true"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422139    5492 flags.go:33] FLAG: --manifest-url=""
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422142    5492 flags.go:33] FLAG: --manifest-url-header=""
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422146    5492 flags.go:33] FLAG: --master-service-namespace="default"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422149    5492 flags.go:33] FLAG: --max-open-files="1000000"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422153    5492 flags.go:33] FLAG: --max-pods="110"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422156    5492 flags.go:33] FLAG: --maximum-dead-containers="-1"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422159    5492 flags.go:33] FLAG: --maximum-dead-containers-per-container="1"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422162    5492 flags.go:33] FLAG: --minimum-container-ttl-duration="6m0s"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422165    5492 flags.go:33] FLAG: --minimum-image-ttl-duration="2m0s"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422168    5492 flags.go:33] FLAG: --network-plugin=""
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422170    5492 flags.go:33] FLAG: --network-plugin-mtu="0"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422173    5492 flags.go:33] FLAG: --node-ip=""
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422176    5492 flags.go:33] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhcos"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422182    5492 flags.go:33] FLAG: --node-status-max-images="50"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422185    5492 flags.go:33] FLAG: --node-status-update-frequency="10s"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422188    5492 flags.go:33] FLAG: --non-masquerade-cidr="10.0.0.0/8"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422191    5492 flags.go:33] FLAG: --oom-score-adj="-999"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422194    5492 flags.go:33] FLAG: --pod-cidr=""
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422196    5492 flags.go:33] FLAG: --pod-infra-container-image="k8s.gcr.io/pause:3.1"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422200    5492 flags.go:33] FLAG: --pod-manifest-path=""
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422211    5492 flags.go:33] FLAG: --pod-max-pids="-1"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422214    5492 flags.go:33] FLAG: --pods-per-core="0"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422217    5492 flags.go:33] FLAG: --port="10250"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422219    5492 flags.go:33] FLAG: --protect-kernel-defaults="false"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422222    5492 flags.go:33] FLAG: --provider-id=""
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422225    5492 flags.go:33] FLAG: --qos-reserved=""
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422228    5492 flags.go:33] FLAG: --read-only-port="10255"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422231    5492 flags.go:33] FLAG: --really-crash-for-testing="false"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422234    5492 flags.go:33] FLAG: --redirect-container-streaming="false"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422237    5492 flags.go:33] FLAG: --register-node="true"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422240    5492 flags.go:33] FLAG: --register-schedulable="true"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422243    5492 flags.go:33] FLAG: --register-with-taints=""
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422246    5492 flags.go:33] FLAG: --registry-burst="10"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422249    5492 flags.go:33] FLAG: --registry-qps="5"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422252    5492 flags.go:33] FLAG: --reserved-cpus=""
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422255    5492 flags.go:33] FLAG: --resolv-conf="/etc/resolv.conf"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422258    5492 flags.go:33] FLAG: --root-dir="/var/lib/kubelet"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422261    5492 flags.go:33] FLAG: --rotate-certificates="false"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422264    5492 flags.go:33] FLAG: --rotate-server-certificates="false"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422266    5492 flags.go:33] FLAG: --runonce="false"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422269    5492 flags.go:33] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422272    5492 flags.go:33] FLAG: --runtime-request-timeout="2m0s"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422275    5492 flags.go:33] FLAG: --seccomp-profile-root="/var/lib/kubelet/seccomp"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422278    5492 flags.go:33] FLAG: --serialize-image-pulls="true"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422281    5492 flags.go:33] FLAG: --skip-headers="false"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422284    5492 flags.go:33] FLAG: --skip-log-headers="false"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422287    5492 flags.go:33] FLAG: --stderrthreshold="2"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422290    5492 flags.go:33] FLAG: --storage-driver-buffer-duration="1m0s"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422293    5492 flags.go:33] FLAG: --storage-driver-db="cadvisor"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422296    5492 flags.go:33] FLAG: --storage-driver-host="localhost:8086"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422299    5492 flags.go:33] FLAG: --storage-driver-password="root"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422301    5492 flags.go:33] FLAG: --storage-driver-secure="false"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422312    5492 flags.go:33] FLAG: --storage-driver-table="stats"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422315    5492 flags.go:33] FLAG: --storage-driver-user="root"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422318    5492 flags.go:33] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422321    5492 flags.go:33] FLAG: --sync-frequency="1m0s"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422324    5492 flags.go:33] FLAG: --system-cgroups=""
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422326    5492 flags.go:33] FLAG: --system-reserved=""
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422329    5492 flags.go:33] FLAG: --system-reserved-cgroup=""
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422333    5492 flags.go:33] FLAG: --tls-cert-file=""
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422336    5492 flags.go:33] FLAG: --tls-cipher-suites="[]"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422340    5492 flags.go:33] FLAG: --tls-min-version=""
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422343    5492 flags.go:33] FLAG: --tls-private-key-file=""
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422345    5492 flags.go:33] FLAG: --topology-manager-policy="none"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422349    5492 flags.go:33] FLAG: --v="3"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422352    5492 flags.go:33] FLAG: --version="false"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422361    5492 flags.go:33] FLAG: --vmodule=""
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422365    5492 flags.go:33] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422369    5492 flags.go:33] FLAG: --volume-stats-agg-period="1m0s"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.422394    5492 feature_gate.go:244] feature gates: &{map[]}
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.423859    5492 feature_gate.go:244] feature gates: &{map[LegacyNodeRoleBehavior:false NodeDisruptionExclusion:true RotateKubeletServerCertificate:true SCTPSupport:true ServiceNodeExclusion:true SupportPodPidsLimit:true]}
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.423940    5492 feature_gate.go:244] feature gates: &{map[LegacyNodeRoleBehavior:false NodeDisruptionExclusion:true RotateKubeletServerCertificate:true SCTPSupport:true ServiceNodeExclusion:true SupportPodPidsLimit:true]}
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.433150    5492 mount_linux.go:168] Detected OS with systemd
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.433271    5492 server.go:424] Version: v1.17.1
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.433323    5492 feature_gate.go:244] feature gates: &{map[LegacyNodeRoleBehavior:false NodeDisruptionExclusion:true RotateKubeletServerCertificate:true SCTPSupport:true ServiceNodeExclusion:true SupportPodPidsLimit:true]}
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.433385    5492 feature_gate.go:244] feature gates: &{map[LegacyNodeRoleBehavior:false NodeDisruptionExclusion:true RotateKubeletServerCertificate:true SCTPSupport:true ServiceNodeExclusion:true SupportPodPidsLimit:true]}
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.433491    5492 plugins.go:100] No cloud provider specified.
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.433510    5492 server.go:540] No cloud provider specified: "" from the config file: ""
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.433519    5492 server.go:830] Client rotation is on, will bootstrap in background
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.445886    5492 bootstrap.go:84] Current kubeconfig file contents are still valid, no bootstrap necessary
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.445957    5492 certificate_store.go:129] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.446355    5492 server.go:857] Starting client certificate rotation.
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.446367    5492 certificate_manager.go:285] Certificate rotation is enabled.
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.446467    5492 certificate_manager.go:556] Certificate expiration is 2020-07-26 09:59:31 +0000 UTC, rotation deadline is 2020-07-20 19:25:44.002114354 +0000 UTC
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.446497    5492 certificate_manager.go:291] Waiting 576h20m33.555623075s for next certificate rotation
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.447060    5492 dynamic_cafile_content.go:129] Loaded a new CA Bundle and Verifier for "client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.447171    5492 dynamic_cafile_content.go:167] Starting client-ca-bundle::/etc/kubernetes/kubelet-ca.crt
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.447357    5492 manager.go:146] cAdvisor running in container: "/sys/fs/cgroup/cpu,cpuacct/system.slice/kubelet.service"
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.448446    5492 fs.go:125] Filesystem UUIDs: map[00000000-0000-4000-a000-000000000002:/dev/sda4 1ccc3ac6-b59d-46bb-9850-229cd8b4e007:/dev/dm-0 676b3306-2850-478c-a817-3f723d49377d:/dev/sda1 D802-CD71:/dev/sda2]
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.448470    5492 fs.go:126] Filesystem partitions: map[/dev/mapper/coreos-luks-root-nocrypt:{mountpoint:/var major:253 minor:0 fsType:xfs blockSize:0} /dev/sda1:{mountpoint:/boot major:8 minor:1 fsType:ext4 blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /run/user/1000:{mountpoint:/run/user/1000 major:0 minor:44 fsType:tmpfs blockSize:0} /sys/fs/cgroup:{mountpoint:/sys/fs/cgroup major:0 minor:25 fsType:tmpfs blockSize:0}]
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.453331    5492 manager.go:193] Machine: {NumCores:40 CpuFrequency:3100000 MemoryCapacity:135090417664 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:21668e85e1264ac78ea115b2fe79408e SystemUUID:4c4c4544-004b-5a10-8050-cac04f484832 BootID:16f2699c-8a76-444a-8f56-e32057152db4 Filesystems:[{Device:/run/user/1000 DeviceMajor:0 DeviceMinor:44 Capacity:13509038080 Type:vfs Inodes:16490529 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:67545206784 Type:vfs Inodes:16490529 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:67545206784 Type:vfs Inodes:16490529 HasInodes:true} {Device:/sys/fs/cgroup DeviceMajor:0 DeviceMinor:25 Capacity:67545206784 Type:vfs Inodes:16490529 HasInodes:true} {Device:/dev/mapper/coreos-luks-root-nocrypt DeviceMajor:253 DeviceMinor:0 Capacity:598985388032 Type:vfs Inodes:292478400 HasInodes:true} {Device:/dev/sda1 DeviceMajor:8 DeviceMinor:1 Capacity:381549568 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[253:0:{Name:dm-0 Major:253 Minor:0 Size:598995877376 Scheduler:none} 8:0:{Name:sda Major:8 Minor:0 Size:599550590976 Scheduler:mq-deadline}] NetworkDevices:[{Name:eno1 MacAddress:18:66:da:9f:b1:0a Speed:1000 Mtu:1500} {Name:eno2 MacAddress:18:66:da:9f:b1:0b Speed:-1 Mtu:1500} {Name:eno3 MacAddress:18:66:da:9f:b1:0c Speed:-1 Mtu:1500} {Name:eno4 MacAddress:18:66:da:9f:b1:0d Speed:1000 Mtu:1500} {Name:enp4s0f0 MacAddress:a0:36:9f:e5:e9:fc Speed:10000 Mtu:1500} {Name:enp4s0f1 MacAddress:a0:36:9f:e5:e9:fe Speed:10000 Mtu:1500} {Name:enp5s0f0 MacAddress:a0:36:9f:e5:e2:a8 Speed:10000 Mtu:1500} {Name:enp5s0f1 MacAddress:a0:36:9f:e5:e2:aa Speed:10000 Mtu:1500}] Topology:[{Id:0 Memory:67476070400 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 20] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]} {Id:1 Threads:[2 22] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]} {Id:2 Threads:[4 24] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]} {Id:3 Threads:[6 26] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]} {Id:4 Threads:[8 28] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]} {Id:8 Threads:[10 30] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]} {Id:9 Threads:[12 32] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]} {Id:10 Threads:[14 34] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]} {Id:11 Threads:[16 36] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]} {Id:12 Threads:[18 38] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]}] Caches:[{Size:26214400 Type:Unified Level:3}]} {Id:1 Memory:67614347264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[1 21] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]} {Id:1 Threads:[3 23] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]} {Id:2 Threads:[5 25] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]} {Id:3 Threads:[7 27] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]} {Id:4 Threads:[9 29] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]} {Id:8 Threads:[11 31] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]} {Id:9 Threads:[13 33] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]} {Id:10 Threads:[15 35] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]} {Id:11 Threads:[17 37] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]} {Id:12 Threads:[19 39] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]}] Caches:[{Size:26214400 Type:Unified Level:3}]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.453643    5492 manager.go:199] Version: {KernelVersion:4.18.0-147.8.1.el8_1.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 44.81.202005250830-0 (Ootpa) DockerVersion:Unknown DockerAPIVersion:Unknown CadvisorVersion: CadvisorRevision:}
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.454069    5492 container_manager_linux.go:265] container manager verified user specified cgroup-root exists: []
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.454102    5492 container_manager_linux.go:270] Creating Container Manager object based on Node Config: {RuntimeCgroupsName:/system.slice/crio.service SystemCgroupsName:/system.slice KubeletCgroupsName: ContainerRuntime:remote CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:systemd KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: ReservedSystemCPUs: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[cpu:{i:{value:0 scale:0} d:{Dec:<nil>} s:0 Format:DecimalSI} ephemeral-storage:{i:{value:10737418240 scale:0} d:{Dec:<nil>} s:10Gi Format:BinarySI} memory:{i:{value:1073741824 scale:0} d:{Dec:<nil>} s:1Gi Format:BinarySI}] HardEvictionThresholds:[{Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.1} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.inodesFree Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>} {Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.15} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:static ExperimentalCPUManagerReconcilePeriod:5s ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms ExperimentalTopologyManagerPolicy:none}
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.454183    5492 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.454189    5492 container_manager_linux.go:305] Creating device plugin manager: true
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.454201    5492 manager.go:126] Creating Device Plugin manager at /var/lib/kubelet/device-plugins/kubelet.sock
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.454215    5492 fake_topology_manager.go:39] [fake topologymanager] AddHintProvider HintProvider:  &{kubelet.sock /var/lib/kubelet/device-plugins/ map[] {0 0} <nil> {{} [0 0 0]} 0x1b64c30 0x72dc9c8 0x1b65500 map[] map[] map[] map[] map[] 0xc000853ce0 [0 1] 0x72dc9c8}
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: I0626 19:05:10.454252    5492 cpu_manager.go:131] [cpumanager] detected CPU topology: &{40 20 2 map[0:{0 0 0} 1:{1 1 1} 2:{0 0 2} 3:{1 1 3} 4:{0 0 4} 5:{1 1 5} 6:{0 0 6} 7:{1 1 7} 8:{0 0 8} 9:{1 1 9} 10:{0 0 10} 11:{1 1 11} 12:{0 0 12} 13:{1 1 13} 14:{0 0 14} 15:{1 1 15} 16:{0 0 16} 17:{1 1 17} 18:{0 0 18} 19:{1 1 19} 20:{0 0 0} 21:{1 1 1} 22:{0 0 2} 23:{1 1 3} 24:{0 0 4} 25:{1 1 5} 26:{0 0 6} 27:{1 1 7} 28:{0 0 8} 29:{1 1 9} 30:{0 0 10} 31:{1 1 11} 32:{0 0 12} 33:{1 1 13} 34:{0 0 14} 35:{1 1 15} 36:{0 0 16} 37:{1 1 17} 38:{0 0 18} 39:{1 1 19}]}
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: E0626 19:05:10.454300    5492 container_manager_linux.go:329] failed to initialize cpu manager: [cpumanager] unable to determine reserved CPU resources for static policy
Jun 26 19:05:10 openshift-worker-1.example.com hyperkube[5492]: F0626 19:05:10.454309    5492 server.go:273] failed to run Kubelet: [cpumanager] unable to determine reserved CPU resources for static policy
Jun 26 19:05:10 openshift-worker-1.example.com systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Jun 26 19:05:10 openshift-worker-1.example.com systemd[1]: kubelet.service: Failed with result 'exit-code'.
Jun 26 19:05:10 openshift-worker-1.example.com systemd[1]: Failed to start Kubernetes Kubelet.
Jun 26 19:05:10 openshift-worker-1.example.com systemd[1]: kubelet.service: Consumed 117ms CPU time

And the node will never become Ready in the node list.

References


Last update: September 13, 2020