Skip to content

Troubleshooting OpenShift on OpenStack worker creation

See https://bugzilla.redhat.com/show_bug.cgi?id=1785705 for the bugzilla.

The following are steps which I executed in my lab to troubleshoot issues with worker creation for OpenShift on OpenStack.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
(overcloud) [stack@undercloud-0 clouds]$ oc describe machine -n openshift-machine-api osc-c5r5c-worker-bgt9b
Name:         osc-c5r5c-worker-bgt9b
Namespace:    openshift-machine-api
Labels:       machine.openshift.io/cluster-api-cluster=osc-c5r5c
              machine.openshift.io/cluster-api-machine-role=worker
              machine.openshift.io/cluster-api-machine-type=worker
              machine.openshift.io/cluster-api-machineset=osc-c5r5c-worker
Annotations:  <none>
API Version:  machine.openshift.io/v1beta1
Kind:         Machine
Metadata:
  Creation Timestamp:  2019-12-20T10:23:14Z
  Finalizers:
    machine.machine.openshift.io
  Generate Name:  osc-c5r5c-worker-
  Generation:     1
  Owner References:
    API Version:           machine.openshift.io/v1beta1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  MachineSet
    Name:                  osc-c5r5c-worker
    UID:                   9077b6df-2312-11ea-9b6c-fa163e431263
  Resource Version:        3455
  Self Link:               /apis/machine.openshift.io/v1beta1/namespaces/openshift-machine-api/machines/osc-c5r5c-worker-bgt9b
  UID:                     bb0af782-2312-11ea-9b6c-fa163e431263
Spec:
  Metadata:
    Creation Timestamp:  <nil>
  Provider Spec:
    Value:
      API Version:  openstackproviderconfig.openshift.io/v1alpha1
      Cloud Name:   openstack
      Clouds Secret:
        Name:       openstack-cloud-credentials
        Namespace:  openshift-machine-api
      Flavor:       m1.openshift
      Image:        rhcos
      Kind:         OpenstackProviderSpec
      Metadata:
        Creation Timestamp:  <nil>
      Networks:
        Filter:
        Subnets:
          Filter:
            Name:  osc-c5r5c-nodes
            Tags:  openshiftClusterID=osc-c5r5c
      Security Groups:
        Filter:
        Name:  osc-c5r5c-worker
      Server Metadata:
        Name:                  osc-c5r5c-worker
        Openshift Cluster ID:  osc-c5r5c
      Tags:
        openshiftClusterID=osc-c5r5c
      Trunk:  true
      User Data Secret:
        Name:  worker-user-data
Events:        <none>
(overcloud) [stack@undercloud-0 clouds]$ oc get machine -n openshift-machine-api 
NAME                     STATE   TYPE   REGION   ZONE   AGE
osc-c5r5c-master-0                                      4h52m
osc-c5r5c-master-1                                      4h52m
osc-c5r5c-master-2                                      4h52m
osc-c5r5c-worker-bgt9b                                  4h51m
osc-c5r5c-worker-qphk7                                  4h51m
osc-c5r5c-worker-vs85h                                  4h51m
(overcloud) [stack@undercloud-0 clouds]$ oc get machineset -n openshift-machine-api 
NAME               DESIRED   CURRENT   READY   AVAILABLE   AGE
osc-c5r5c-worker   3         3                             4h54m
(overcloud) [stack@undercloud-0 clouds]$ 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
(overcloud) [stack@undercloud-0 clouds]$ kubectl get machineset -n openshift-machine-api osc-c5r5c-worker -o yaml
apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  creationTimestamp: "2019-12-20T10:22:02Z"
  generation: 1
  labels:
    machine.openshift.io/cluster-api-cluster: osc-c5r5c
    machine.openshift.io/cluster-api-machine-role: worker
    machine.openshift.io/cluster-api-machine-type: worker
  name: osc-c5r5c-worker
  namespace: openshift-machine-api
  resourceVersion: "3448"
  selfLink: /apis/machine.openshift.io/v1beta1/namespaces/openshift-machine-api/machinesets/osc-c5r5c-worker
  uid: 9077b6df-2312-11ea-9b6c-fa163e431263
spec:
  replicas: 3
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: osc-c5r5c
      machine.openshift.io/cluster-api-machineset: osc-c5r5c-worker
  template:
    metadata:
      creationTimestamp: null
      labels:
        machine.openshift.io/cluster-api-cluster: osc-c5r5c
        machine.openshift.io/cluster-api-machine-role: worker
        machine.openshift.io/cluster-api-machine-type: worker
        machine.openshift.io/cluster-api-machineset: osc-c5r5c-worker
    spec:
      metadata:
        creationTimestamp: null
      providerSpec:
        value:
          apiVersion: openstackproviderconfig.openshift.io/v1alpha1
          cloudName: openstack
          cloudsSecret:
            name: openstack-cloud-credentials
            namespace: openshift-machine-api
          flavor: m1.openshift
          image: rhcos
          kind: OpenstackProviderSpec
          metadata:
            creationTimestamp: null
          networks:
          - filter: {}
            subnets:
            - filter:
                name: osc-c5r5c-nodes
                tags: openshiftClusterID=osc-c5r5c
          securityGroups:
          - filter: {}
            name: osc-c5r5c-worker
          serverMetadata:
            Name: osc-c5r5c-worker
            openshiftClusterID: osc-c5r5c
          tags:
          - openshiftClusterID=osc-c5r5c
          trunk: true
          userDataSecret:
            name: worker-user-data
status:
  fullyLabeledReplicas: 3
  observedGeneration: 1
  replicas: 3
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
/var/log/pods/openshift-machine-api_machine-api-controllers-f64b7f7b8-tm7qc_b1f05af2-2312-11ea-9b6c-fa163e431263/machine-controller/0.log
(...)
2019-12-20T10:23:14.635223358+00:00 stderr F E1220 10:23:14.635191       1 controller.go:239] Failed to check if machine "osc-c5r5c-worker-bgt9b" exists: Error checking if instance exists (machine/actuator.go 346): 
2019-12-20T10:23:14.635223358+00:00 stderr F Error getting a new instance service from the machine (machine/actuator.go 467): Create providerClient err: You must provide exactly one of DomainID or DomainName in a Scope with ProjectName
2019-12-20T10:23:15.636135623+00:00 stderr F I1220 10:23:15.635874       1 controller.go:133] Reconciling Machine "osc-c5r5c-master-0"
2019-12-20T10:23:15.636215058+00:00 stderr F I1220 10:23:15.636192       1 controller.go:304] Machine "osc-c5r5c-master-0" in namespace "openshift-machine-api" doesn't specify "cluster.k8s.io/cluster-name" label, assuming nil cluster
2019-12-20T10:23:15.641348382+00:00 stderr F E1220 10:23:15.641317       1 controller.go:239] Failed to check if machine "osc-c5r5c-master-0" exists: Error checking if instance exists (machine/actuator.go 346): 
2019-12-20T10:23:15.641348382+00:00 stderr F Error getting a new instance service from the machine (machine/actuator.go 467): Create providerClient err: You must provide exactly one of DomainID or DomainName in a Scope with ProjectName
2019-12-20T10:23:16.641614350+00:00 stderr F I1220 10:23:16.641562       1 controller.go:133] Reconciling Machine "osc-c5r5c-master-1"
2019-12-20T10:23:16.641614350+00:00 stderr F I1220 10:23:16.641591       1 controller.go:304] Machine "osc-c5r5c-master-1" in namespace "openshift-machine-api" doesn't specify "cluster.k8s.io/cluster-name" label, assuming nil cluster
2019-12-20T10:23:16.647323379+00:00 stderr F E1220 10:23:16.647295       1 controller.go:239] Failed to check if machine "osc-c5r5c-master-1" exists: Error checking if instance exists (machine/actuator.go 346): 
2019-12-20T10:23:16.647323379+00:00 stderr F Error getting a new instance service from the machine (machine/actuator.go 467): Create providerClient err: You must provide exactly one of DomainID or DomainName in a Scope with ProjectName
2019-12-20T10:23:17.647630531+00:00 stderr F I1220 10:23:17.647587       1 controller.go:133] Reconciling Machine "osc-c5r5c-master-2"
2019-12-20T10:23:17.647695173+00:00 stderr F I1220 10:23:17.647677       1 controller.go:304] Machine "osc-c5r5c-master-2" in namespace "openshift-machine-api" doesn't specify "cluster.k8s.io/cluster-name" label, assuming nil cluster
2019-12-20T10:23:17.652613228+00:00 stderr F E1220 10:23:17.652551       1 controller.go:239] Failed to check if machine "osc-c5r5c-master-2" exists: Error checking if instance exists (machine/actuator.go 346): 
2019-12-20T10:23:17.652613228+00:00 stderr F Error getting a new instance service from the machine (machine/actuator.go 467): Create providerClient err: You must provide exactly one of DomainID or DomainName in a Scope with ProjectName
2019-12-20T10:23:18.652945792+00:00 stderr F I1220 10:23:18.652884       1 controller.go:133] Reconciling Machine "osc-c5r5c-worker-qphk7"

Looking at the secret:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
(overcloud) [stack@undercloud-0 clouds]$ kubectl get secrets -n openshift-machine-api openstack-cloud-credentials -o yaml
apiVersion: v1
data:
  clouds.yaml: Y2xvdWRzOgogIG9wZW5zdGFjazoKICAgIGF1dGg6CiAgICAgIGFwcGxpY2F0aW9uX2NyZWRlbnRpYWxfaWQ6ICIiCiAgICAgIGFwcGxpY2F0aW9uX2NyZWRlbnRpYWxfbmFtZTogIiIKICAgICAgYXBwbGljYXRpb25fY3JlZGVudGlhbF9zZWNyZXQ6ICIiCiAgICAgIGF1dGhfdXJsOiBodHRwOi8vMTcyLjE2LjAuMTMwOjUwMDAvL3YzCiAgICAgIGRlZmF1bHRfZG9tYWluOiAiIgogICAgICBkb21haW5faWQ6ICIiCiAgICAgIGRvbWFpbl9uYW1lOiAiIgogICAgICBwYXNzd29yZDogelA0YmUydWtocENrajR6cVJmVWs4WGpRYgogICAgICBwcm9qZWN0X2RvbWFpbl9pZDogIiIKICAgICAgcHJvamVjdF9kb21haW5fbmFtZTogIiIKICAgICAgcHJvamVjdF9pZDogIiIKICAgICAgcHJvamVjdF9uYW1lOiBhZG1pbgogICAgICB0b2tlbjogIiIKICAgICAgdXNlcl9kb21haW5faWQ6ICIiCiAgICAgIHVzZXJfZG9tYWluX25hbWU6IERlZmF1bHQKICAgICAgdXNlcl9pZDogIiIKICAgICAgdXNlcm5hbWU6IGFkbWluCiAgICBhdXRoX3R5cGU6ICIiCiAgICBjYWNlcnQ6ICIiCiAgICBjZXJ0OiAiIgogICAgY2xvdWQ6ICIiCiAgICBpZGVudGl0eV9hcGlfdmVyc2lvbjogIjMiCiAgICBrZXk6ICIiCiAgICBwcm9maWxlOiAiIgogICAgcmVnaW9uX25hbWU6IHJlZ2lvbk9uZQogICAgcmVnaW9uczogbnVsbAogICAgdmVyaWZ5OiB0cnVlCiAgICB2b2x1bWVfYXBpX3ZlcnNpb246ICIiCg==
kind: Secret
metadata:
  annotations:
    cloudcredential.openshift.io/credentials-request: openshift-cloud-credential-operator/openshift-machine-api-openstack
  creationTimestamp: "2019-12-20T10:23:00Z"
  name: openstack-cloud-credentials
  namespace: openshift-machine-api
  resourceVersion: "2525"
  selfLink: /api/v1/namespaces/openshift-machine-api/secrets/openstack-cloud-credentials
  uid: b2bded52-2312-11ea-9b6c-fa163e431263
type: Opaque

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
(overcloud) [stack@undercloud-0 clouds]$ base64 -d <(echo 'Y2xvdWRzOgogIG9wZW5zdGFjazoKICAgIGF1dGg6CiAgICAgIGFwcGxpY2F0aW9uX2NyZWRlbnRpYWxfaWQ6ICIiCiAgICAgIGFwcGxpY2F0aW9uX2NyZWRlbnRpYWxfbmFtZTogIiIKICAgICAgYXBwbGljYXRpb25fY3JlZGVudGlhbF9zZWNyZXQ6ICIiCiAgICAgIGF1dGhfdXJsOiBodHRwOi8vMTcyLjE2LjAuMTMwOjUwMDAvL3YzCiAgICAgIGRlZmF1bHRfZG9tYWluOiAiIgogICAgICBkb21haW5faWQ6ICIiCiAgICAgIGRvbWFpbl9uYW1lOiAiIgogICAgICBwYXNzd29yZDogelA0YmUydWtocENrajR6cVJmVWs4WGpRYgogICAgICBwcm9qZWN0X2RvbWFpbl9pZDogIiIKICAgICAgcHJvamVjdF9kb21haW5fbmFtZTogIiIKICAgICAgcHJvamVjdF9pZDogIiIKICAgICAgcHJvamVjdF9uYW1lOiBhZG1pbgogICAgICB0b2tlbjogIiIKICAgICAgdXNlcl9kb21haW5faWQ6ICIiCiAgICAgIHVzZXJfZG9tYWluX25hbWU6IERlZmF1bHQKICAgICAgdXNlcl9pZDogIiIKICAgICAgdXNlcm5hbWU6IGFkbWluCiAgICBhdXRoX3R5cGU6ICIiCiAgICBjYWNlcnQ6ICIiCiAgICBjZXJ0OiAiIgogICAgY2xvdWQ6ICIiCiAgICBpZGVudGl0eV9hcGlfdmVyc2lvbjogIjMiCiAgICBrZXk6ICIiCiAgICBwcm9maWxlOiAiIgogICAgcmVnaW9uX25hbWU6IHJlZ2lvbk9uZQogICAgcmVnaW9uczogbnVsbAogICAgdmVyaWZ5OiB0cnVlCiAgICB2b2x1bWVfYXBpX3ZlcnNpb246ICIiCg==')
clouds:
  openstack:
    auth:
      application_credential_id: ""
      application_credential_name: ""
      application_credential_secret: ""
      auth_url: http://172.16.0.130:5000//v3
      default_domain: ""
      domain_id: ""
      domain_name: ""
      password: zP4be2ukhpCkj4zqRfUk8XjQb
      project_domain_id: ""
      project_domain_name: ""
      project_id: ""
      project_name: admin
      token: ""
      user_domain_id: ""
      user_domain_name: Default
      user_id: ""
      username: admin
    auth_type: ""
    cacert: ""
    cert: ""
    cloud: ""
    identity_api_version: "3"
    key: ""
    profile: ""
    region_name: regionOne
    regions: null
    verify: true
    volume_api_version: ""
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
(overcloud) [stack@undercloud-0 clouds]$ cat clouds.yaml 
clouds:
  overcloud:
    auth:
      auth_url: http://172.16.0.130:5000//v3
      username: "admin"
      password: zP4be2ukhpCkj4zqRfUk8XjQb
      project_name: "admin"
      user_domain_name: "Default"
    region_name: "regionOne"
    interface: "public"
    identity_api_version: 3

The problem is in the generated secret:

1
Error getting a new instance service from the machine (machine/actuator.go 467): Create providerClient err: You must provide exactly one of DomainID or DomainName in a Scope with ProjectName

https://github.com/terraform-providers/terraform-provider-openstack/issues/267

Whereas when I check here: https://egallen.com/openshift-42-on-openstack-13-gpu/ This blog article uses project_id, too.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
clouds:
  openstack:
    auth:
      auth_url: http://192.168.168.54:5000/v3
      username: "admin"
      password: XXXXXXXXXXXXXX
      project_id: XXXXXXXXX
      project_name: "admin"
      user_domain_name: "Default"
    region_name: "regionOne"
    interface: "public"
    identity_api_version: 3

However, my clouds.yaml file is actually completely correct:

1
2
3
4
5
6
7
8
9
[stack@undercloud-0 clouds]$ openstack --os-cloud overcloud token issue
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field      | Value                                                                                                                                                                                   |
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| expires    | 2019-12-20T16:24:40+0000                                                                                                                                                                |
| id         | gAAAAABd_Oe4sLSSjwQiCPFhVK9PUFBehqVXbj-r96GdFvRieT51YZQUdm5lc5ic5VKYRFPg4jhPat4ZIdyow1QL-vZnxSK8MUAqUMQnc6xjs80JD-ibCNIg1Gac14Idp1CGIutsaUMS-Ms33LDgEw32S2qomv7LRUCLVcEBwrqwYLHXYE2ohyk |
| project_id | 1bb14f515f0945a4891fe3fa2372a795                                                                                                                                                        |
| user_id    | 0d3f5ab158c64c11b57d58c76d9675f0                                                                                                                                                        |
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

I then tried with:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
clouds:
  openstack:
    auth:
      auth_url: http://172.16.0.130:5000//v3
      username: "admin"
      password: zP4be2ukhpCkj4zqRfUk8XjQb
      project_id: 1bb14f515f0945a4891fe3fa2372a795
      project_name: "admin"
      user_domain_name: "Default"
    region_name: "regionOne"
    interface: "public"
    identity_api_version: 3

Note that I made 2 changes: project_id which I got from openstack project list | grep admin ; and I renamed the cloud credential to name openstack.

This actually worked:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
(overcloud) [stack@undercloud-0 ~]$ openstack server list
+--------------------------------------+------------------------+--------+-------------------------------------------------------------------------+-------+--------------+
| ID                                   | Name                   | Status | Networks                                                                | Image | Flavor       |
+--------------------------------------+------------------------+--------+-------------------------------------------------------------------------+-------+--------------+
| 6726f1d5-977f-48da-9554-acf710d459fe | osc-6gzh2-worker-shw2w | ACTIVE | osc-6gzh2-openshift=172.31.0.28                                         | rhcos | m1.openshift |
| 37af5d51-bc08-4002-bb5b-380061d9efbf | osc-6gzh2-master-0     | ACTIVE | osc-6gzh2-openshift=172.31.0.24                                         | rhcos | m1.openshift |
| 4b52d095-e174-47ea-a699-85286f899f67 | osc-6gzh2-master-1     | ACTIVE | osc-6gzh2-openshift=172.31.0.16                                         | rhcos | m1.openshift |
| 73bb2012-d9ff-46e4-8e62-c6d723ca5711 | osc-6gzh2-master-2     | ACTIVE | osc-6gzh2-openshift=172.31.0.15                                         | rhcos | m1.openshift |
| 53e2da7f-1398-45d3-94e3-0008d8e209d9 | rhel-test1             | ACTIVE | private1=2000:192:168:0:f816:3eff:fe13:275, 192.168.0.113, 172.16.0.110 | rhel  | m1.small     |
+--------------------------------------+------------------------+--------+-------------------------------------------------------------------------+-------+--------------+

The cluster deployed:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
(overcloud) [stack@undercloud-0 clouds]$ ./openshift-install create cluster --dir=install-config/ --log-level=info
INFO Consuming "Install Config" from target directory 
INFO Creating infrastructure resources...         
INFO Waiting up to 30m0s for the Kubernetes API at https://api.osc.redhat.local:6443... 
INFO API v1.14.6+17b1cc6 up                       
INFO Waiting up to 30m0s for bootstrapping to complete... 
INFO Destroying the bootstrap resources...        
INFO Waiting up to 30m0s for the cluster at https://api.osc.redhat.local:6443 to initialize... 
INFO Waiting up to 10m0s for the openshift-console route to be created... 
INFO Install complete!                            
INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/home/stack/clouds/install-config/auth/kubeconfig' 
INFO Access the OpenShift web-console here: https://console-openshift-console.apps.osc.redhat.local 
INFO Login to the console with user: kubeadmin, password: vFKnm-d43eY-aMMBB-52xcU 

Now, there was an issue with 2 of the machines:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
(overcloud) [stack@undercloud-0 clouds]$ oc get machineset -A
NAMESPACE               NAME               DESIRED   CURRENT   READY   AVAILABLE   AGE
openshift-machine-api   osc-6gzh2-worker   3         3         1       1           97m
(overcloud) [stack@undercloud-0 clouds]$ oc get machine -A
NAMESPACE               NAME                     STATE    TYPE           REGION      ZONE   AGE
openshift-machine-api   osc-6gzh2-master-0       ACTIVE   m1.openshift   regionOne   nova   97m
openshift-machine-api   osc-6gzh2-master-1       ACTIVE   m1.openshift   regionOne   nova   97m
openshift-machine-api   osc-6gzh2-master-2       ACTIVE   m1.openshift   regionOne   nova   97m
openshift-machine-api   osc-6gzh2-worker-5d627   ERROR                                      96m
openshift-machine-api   osc-6gzh2-worker-j6j78   ERROR                                      96m
openshift-machine-api   osc-6gzh2-worker-shw2w   ACTIVE   m1.openshift   regionOne   nova   96m
(overcloud) [stack@undercloud-0 clouds]$ oc get machineconfig -A
NAME                                                        GENERATEDBYCONTROLLER                      IGNITIONVERSION   CREATED
00-master                                                   d780d197a9c5848ba786982c0c4aaa7487297046   2.2.0             96m
00-worker                                                   d780d197a9c5848ba786982c0c4aaa7487297046   2.2.0             96m
01-master-container-runtime                                 d780d197a9c5848ba786982c0c4aaa7487297046   2.2.0             96m
01-master-kubelet                                           d780d197a9c5848ba786982c0c4aaa7487297046   2.2.0             96m
01-worker-container-runtime                                 d780d197a9c5848ba786982c0c4aaa7487297046   2.2.0             96m
01-worker-kubelet                                           d780d197a9c5848ba786982c0c4aaa7487297046   2.2.0             96m
99-master-51c90b92-2340-11ea-baab-fa163e272495-registries   d780d197a9c5848ba786982c0c4aaa7487297046   2.2.0             96m
99-master-ssh                                                                                          2.2.0             96m
99-worker-51ca001b-2340-11ea-baab-fa163e272495-registries   d780d197a9c5848ba786982c0c4aaa7487297046   2.2.0             96m
99-worker-ssh                                                                                          2.2.0             96m
rendered-master-8e766b361a127ddfebde5802f36b90ce            d780d197a9c5848ba786982c0c4aaa7487297046   2.2.0             96m
rendered-worker-74267811bebab086ede643b9c9b2ba66            d780d197a9c5848ba786982c0c4aaa7487297046   2.2.0             96m

And it fails with:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
(overcloud) [stack@undercloud-0 clouds]$ oc describe machine -n openshift-machine-api osc-6gzh2-worker-5d627 | tail
        Openshift Cluster ID:  osc-6gzh2
      Tags:
        openshiftClusterID=osc-6gzh2
      Trunk:  true
      User Data Secret:
        Name:  worker-user-data
Events:
  Type     Reason        Age                   From                  Message
  ----     ------        ----                  ----                  -------
  Warning  FailedCreate  2m30s (x23 over 95m)  openstack_controller  CreateError
(overcloud) [stack@undercloud-0 clouds]$ 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
vercloud) [stack@undercloud-0 clouds]$ oc get events -n openshift-machine-api
LAST SEEN   TYPE      REASON              OBJECT                                              MESSAGE
95m         Normal    Scheduled           pod/cluster-autoscaler-operator-65dfcc75bb-gv8mg    Successfully assigned openshift-machine-api/cluster-autoscaler-operator-65dfcc75bb-gv8mg to osc-6gzh2-master-1
95m         Warning   FailedMount         pod/cluster-autoscaler-operator-65dfcc75bb-gv8mg    MountVolume.SetUp failed for volume "cert" : couldn't propagate object cache: timed out waiting for the condition
95m         Warning   FailedMount         pod/cluster-autoscaler-operator-65dfcc75bb-gv8mg    MountVolume.SetUp failed for volume "cluster-autoscaler-operator-token-gtb48" : couldn't propagate object cache: timed out waiting for the condition
95m         Warning   FailedMount         pod/cluster-autoscaler-operator-65dfcc75bb-gv8mg    MountVolume.SetUp failed for volume "ca-cert" : couldn't propagate object cache: timed out waiting for the condition
94m         Normal    Pulling             pod/cluster-autoscaler-operator-65dfcc75bb-gv8mg    Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e6ea0a063874ff9169ef1e0e58c6399a42e163c49321018fa34c838faec99cb4"
94m         Normal    Pulled              pod/cluster-autoscaler-operator-65dfcc75bb-gv8mg    Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e6ea0a063874ff9169ef1e0e58c6399a42e163c49321018fa34c838faec99cb4"
94m         Normal    Created             pod/cluster-autoscaler-operator-65dfcc75bb-gv8mg    Created container cluster-autoscaler-operator
94m         Normal    Started             pod/cluster-autoscaler-operator-65dfcc75bb-gv8mg    Started container cluster-autoscaler-operator
95m         Normal    SuccessfulCreate    replicaset/cluster-autoscaler-operator-65dfcc75bb   Created pod: cluster-autoscaler-operator-65dfcc75bb-gv8mg
94m         Normal    LeaderElection      configmap/cluster-autoscaler-operator-leader        cluster-autoscaler-operator-65dfcc75bb-gv8mg_40da4183-2341-11ea-8630-0a58ac150025 became leader
95m         Normal    ScalingReplicaSet   deployment/cluster-autoscaler-operator              Scaled up replica set cluster-autoscaler-operator-65dfcc75bb to 1
101m        Normal    Scheduled           pod/machine-api-controllers-f64b7f7b8-xb7qn         Successfully assigned openshift-machine-api/machine-api-controllers-f64b7f7b8-xb7qn to osc-6gzh2-master-2
101m        Normal    Pulling             pod/machine-api-controllers-f64b7f7b8-xb7qn         Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fad6250c0e717515d4caf35d6e6e006f7154e2a7bd9d4fec73f0540e155e3119"
101m        Normal    Pulled              pod/machine-api-controllers-f64b7f7b8-xb7qn         Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fad6250c0e717515d4caf35d6e6e006f7154e2a7bd9d4fec73f0540e155e3119"
101m        Normal    Created             pod/machine-api-controllers-f64b7f7b8-xb7qn         Created container controller-manager
101m        Normal    Started             pod/machine-api-controllers-f64b7f7b8-xb7qn         Started container controller-manager
101m        Normal    Pulled              pod/machine-api-controllers-f64b7f7b8-xb7qn         Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fad6250c0e717515d4caf35d6e6e006f7154e2a7bd9d4fec73f0540e155e3119" already present on machine
101m        Normal    Created             pod/machine-api-controllers-f64b7f7b8-xb7qn         Created container machine-controller
101m        Normal    Started             pod/machine-api-controllers-f64b7f7b8-xb7qn         Started container machine-controller
101m        Normal    Pulled              pod/machine-api-controllers-f64b7f7b8-xb7qn         Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f1a9c01710f09ad1d1c105cbc4f4ff88d7c0f5916a628f9414d3a7905fbced8" already present on machine
101m        Normal    Created             pod/machine-api-controllers-f64b7f7b8-xb7qn         Created container nodelink-controller
101m        Normal    Started             pod/machine-api-controllers-f64b7f7b8-xb7qn         Started container nodelink-controller
101m        Normal    SuccessfulCreate    replicaset/machine-api-controllers-f64b7f7b8        Created pod: machine-api-controllers-f64b7f7b8-xb7qn
101m        Normal    ScalingReplicaSet   deployment/machine-api-controllers                  Scaled up replica set machine-api-controllers-f64b7f7b8 to 1
101m        Warning   FailedScheduling    pod/machine-api-operator-655d94c8fd-6vrqg           0/3 nodes are available: 3 node(s) had taints that the pod didn't tolerate.
101m        Normal    Scheduled           pod/machine-api-operator-655d94c8fd-6vrqg           Successfully assigned openshift-machine-api/machine-api-operator-655d94c8fd-6vrqg to osc-6gzh2-master-2
101m        Normal    Pulling             pod/machine-api-operator-655d94c8fd-6vrqg           Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f1a9c01710f09ad1d1c105cbc4f4ff88d7c0f5916a628f9414d3a7905fbced8"
101m        Normal    Pulled              pod/machine-api-operator-655d94c8fd-6vrqg           Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f1a9c01710f09ad1d1c105cbc4f4ff88d7c0f5916a628f9414d3a7905fbced8"
101m        Normal    Created             pod/machine-api-operator-655d94c8fd-6vrqg           Created container machine-api-operator
101m        Normal    Started             pod/machine-api-operator-655d94c8fd-6vrqg           Started container machine-api-operator
102m        Normal    SuccessfulCreate    replicaset/machine-api-operator-655d94c8fd          Created pod: machine-api-operator-655d94c8fd-6vrqg
102m        Normal    ScalingReplicaSet   deployment/machine-api-operator                     Scaled up replica set machine-api-operator-655d94c8fd to 1
6m55s       Warning   FailedCreate        machine/osc-6gzh2-worker-5d627                      CreateError
7m29s       Warning   FailedCreate        machine/osc-6gzh2-worker-j6j78                      CreateError
98m         Warning   FailedCreate        machine/osc-6gzh2-worker-shw2w                      CreateError
97m         Normal    Created             machine/osc-6gzh2-worker-shw2w                      Created Machine osc-6gzh2-worker-shw2w

I connected to master 2 via master 1:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
[stack@undercloud-0 ~]$ eval $(ssh-agent)
[stack@undercloud-0 ~]$ ssh-add  ~/.ssh/id_rsa
[stack@undercloud-0 ~]$ ssh core@api.osc.redhat.local -A
[stack@undercloud-0 ~]$ ssh core@api.osc.redhat.local -A
Warning: Permanently added 'api.osc.redhat.local,172.16.0.108' (ECDSA) to the list of known hosts.
Red Hat Enterprise Linux CoreOS 42.81.20191203.0
WARNING: Direct SSH access to machines is not recommended.

---
Last login: Fri Dec 20 17:27:11 2019 from 172.16.0.65
[core@osc-6gzh2-master-1 ~]$ ssh core@osc-6gzh2-master-0
Red Hat Enterprise Linux CoreOS 42.81.20191203.0
WARNING: Direct SSH access to machines is not recommended.

---
Last login: Fri Dec 20 17:27:13 2019 from 172.31.0.16
sud[core@osc-6gzh2-master-0 ~]$ sudo -i
[root@osc-6gzh2-master-0 ~]# 

And then got:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
[root@osc-6gzh2-master-2 ~]# grep osc-6gzh2-worker-5d627 /var/log -R
/var/log/pods/openshift-machine-api_machine-api-controllers-f64b7f7b8-xb7qn_4d35a879-2340-11ea-baab-fa163e272495/machine-controller/0.log:2019-12-20T15:49:41.304368389+00:00 stderr F I1220 15:49:41.304339       1 controller.go:133] Reconciling Machine "osc-6gzh2-worker-5d627"
/var/log/pods/openshift-machine-api_machine-api-controllers-f64b7f7b8-xb7qn_4d35a879-2340-11ea-baab-fa163e272495/machine-controller/0.log:2019-12-20T15:49:41.304368389+00:00 stderr F I1220 15:49:41.304356       1 controller.go:304] Machine "osc-6gzh2-worker-5d627" in namespace "openshift-machine-api" doesn't specify "cluster.k8s.io/cluster-name" label, assuming nil cluster
/var/log/pods/openshift-machine-api_machine-api-controllers-f64b7f7b8-xb7qn_4d35a879-2340-11ea-baab-fa163e272495/machine-controller/0.log:2019-12-20T15:50:37.383723384+00:00 stderr F I1220 15:50:37.383397       1 controller.go:133] Reconciling Machine "osc-6gzh2-worker-5d627"
/var/log/pods/openshift-machine-api_machine-api-controllers-f64b7f7b8-xb7qn_4d35a879-2340-11ea-baab-fa163e272495/machine-controller/0.log:2019-12-20T15:50:37.383835414+00:00 stderr F I1220 15:50:37.383814       1 controller.go:304] Machine "osc-6gzh2-worker-5d627" in namespace "openshift-machine-api" doesn't specify "cluster.k8s.io/cluster-name" label, assuming nil cluster
/var/log/pods/openshift-machine-api_machine-api-controllers-f64b7f7b8-xb7qn_4d35a879-2340-11ea-baab-fa163e272495/machine-controller/0.log:2019-12-20T15:50:38.552995792+00:00 stderr F I1220 15:50:38.552954       1 controller.go:253] Reconciling machine object osc-6gzh2-worker-5d627 triggers idempotent create.
/var/log/pods/openshift-machine-api_machine-api-controllers-f64b7f7b8-xb7qn_4d35a879-2340-11ea-baab-fa163e272495/machine-controller/0.log:2019-12-20T15:50:52.700244622+00:00 stderr F E1220 15:50:52.698880       1 actuator.go:470] Machine error osc-6gzh2-worker-5d627: error creating Openstack instance: Create new server err: Request forbidden: [POST http://172.16.0.130:8774/v2.1/servers], error message: {"forbidden": {"message": "Quota exceeded for ram: Requested 32768, but already used 132096 of 153600 ram", "code": 403}}
/var/log/pods/openshift-machine-api_machine-api-controllers-f64b7f7b8-xb7qn_4d35a879-2340-11ea-baab-fa163e272495/machine-controller/0.log:2019-12-20T15:50:52.700244622+00:00 stderr F W1220 15:50:52.698926       1 controller.go:255] Failed to create machine "osc-6gzh2-worker-5d627": error creating Openstack instance: Create new server err: Request forbidden: [POST http://172.16.0.130:8774/v2.1/servers], error message: {"forbidden": {"message": "Quota exceeded for ram: Requested 32768, but already used 132096 of 153600 ram", "code": 403}}
/var/log/pods/openshift-machine-api_machine-api-controllers-f64b7f7b8-xb7qn_4d35a879-2340-11ea-baab-fa163e272495/machine-controller/0.log:2019-12-20T15:51:38.126016932+00:00 stderr F I1220 15:51:38.125931       1 controller.go:133] Reconciling Machine "osc-6gzh2-worker-5d627"
/var/log/pods/openshift-machine-api_machine-api-controllers-f64b7f7b8-xb7qn_4d35a879-2340-11ea-baab-fa163e272495/machine-controller/0.log:2019-12-20T15:51:38.126016932+00:00 stderr F I1220 15:51:38.125971       1 controller.go:304] Machine "osc-6gzh2-worker-5d627" in namespace "openshift-machine-api" doesn't specify "cluster.k8s.io/cluster-name" label, assuming nil cluster
/var/log/pods/openshift-machine-api_machine-api-controllers-f64b7f7b8-xb7qn_4d35a879-2340-11ea-baab-fa163e272495/machine-controller/0.log:2019-12-20T15:51:39.076618133+00:00 stderr F I1220 15:51:39.076540       1 controller.go:253] Reconciling machine object osc-6gzh2-worker-5d627 triggers idempotent create.
/var/log/pods/openshift-machine-api_machine-api-controllers-f64b7f7b8-xb7qn_4d35a879-2340-11ea-baab-fa163e272495/machine-controller/0.log:2019-12-20T15:51:48.625264976+00:00 stderr F E1220 15:51:48.625215       1 actuator.go:470] Machine error osc-6gzh2-worker-5d627: error creating Openstack instance: Create new server err: Request forbidden: [POST http://172.16.0.130:8774/v2.1/servers], error message: {"forbidden": {"message": "Quota exceeded for ram: Requested 32768, but already used 132096 of 153600 ram", "code": 403}}
/var/log/pods/openshift-machine-api_machine-api-controllers-f64b7f7b8-xb7qn_4d35a879-2340-11ea-baab-fa163e272495/machine-controller/0.log:2019-12-20T15:51:48.625264976+00:00 stderr F W1220 15:51:48.625239       1 controller.go:255] Failed to create machine "osc-6gzh2-worker-5d627": error creating Openstack instance: Create new server err: Request forbidden: [POST http://172.16.0.130:8774/v2.1/servers], error message: {"forbidden": {"message": "Quota exceeded for ram: Requested 32768, but already used 132096 of 153600 ram", "code": 403}}


Last update: September 13, 2020