Securing kubernetes clusters with Istio and Keycloak

The walk through is inspired from
https://auth0.com/blog/securing-kubernetes-clusters-with-istio-and-auth0/

This walk through use the most recent version of Helm and Istio.

The goal is to install Helm and Istio just like:

[vagrant@master ~]$ istioctl version
client version: 1.5.1
control plane version: 1.5.1
data plane version: 1.5.1 (3 proxies)

[vagrant@master ~]$ helm version
version.BuildInfo{Version:"v3.2.0", GitCommit:"e11b7ce3b12db2941e90399e874513fbd24bcb71", GitTreeState:"clean", GoVersion:"go1.13.10"}

Lets' begin.

Prerequisite

• Vagrant file
• Nexus for docker private registry

Install Helm

• download Helm

[vagrant@master ~]$ wget https://get.helm.sh/helm-v3.2.0-linux-amd64.tar.gz

• Decompress

[vagrant@master ~]$ tar -zxvf helm-v3.2.0-linux-amd64.tar.gz

• move to bin

[vagrant@master ~]$ sudo mv linux-amd64/helm /usr/local/bin/helm

• check helm

[vagrant@master ~]$ helm version
version.BuildInfo{Version:"v3.2.0", GitCommit:"e11b7ce3b12db2941e90399e874513fbd24bcb71", GitTreeState:"clean", GoVersion:"go1.13.10"}

• Initialize a Helm Chart Repository

[vagrant@master ~]$ helm repo add stable https://kubernetes-charts.storage.googleapis.com/

• Update Helm Repo

[vagrant@master ~]$ helm repo update

Istio

• install

[vagrant@master ~]$ curl -L https://raw.githubusercontent.com/istio/istio/1.5.2/release/downloadIstioCandidate.sh | sh -

• go inside istio dis

[vagrant@master ~]$ cd istio-1.5.2/

• export PATh

[vagrant@master istio-1.5.2]$ export PATH="$PATH:/home/vagrant/istio-1.5.2/bin"

• check istio installation.

[vagrant@master ~]$ istioctl verify-install

Checking the cluster to make sure it is ready for Istio installation...

#1. Kubernetes-api
-----------------------
Can initialize the Kubernetes client.
Can query the Kubernetes API Server.

#2. Kubernetes-version
-----------------------
Istio is compatible with Kubernetes: v1.17.4.

#3. Istio-existence
-----------------------
Istio will be installed in the istio-system namespace.

#4. Kubernetes-setup
-----------------------
Can create necessary Kubernetes configurations: Namespace,ClusterRole,ClusterRoleBinding,CustomResourceDefinition,Role,ServiceAccount,Service,Deployments,ConfigMap.

#5. SideCar-Injector
-----------------------
This Kubernetes cluster supports automatic sidecar injection. To enable automatic sidecar injection see https://istio.io/docs/setup/kubernetes/additional-setup/sidecar-injection/#deploying-an-app

-----------------------
Install Pre-Check passed! The cluster is ready for Istio installation.

• apply Istio default profile

[vagrant@master ~]$ istioctl manifest apply

• output

  This will install the default Istio profile into the cluster. Proceed? (y/N) y
  Detected that your cluster does not support third party JWT authentication. Falling back to less secure first party JWT. See https://istio.io/docs/ops/best-practices/security/#configure-third-party-service-account-tokens for details.
  - Applying manifest for component Base...
  ✔ Finished applying manifest for component Base.
  - Applying manifest for component Pilot...
  ✔ Finished applying manifest for component Pilot.
  Waiting for resources to become ready...
  Waiting for resources to become ready...
  Waiting for resources to become ready...
  - Applying manifest for component IngressGateways...
  - Applying manifest for component AddonComponents...
  ✔ Finished applying manifest for component IngressGateways.
  ✔ Finished applying manifest for component AddonComponents.
  
  
  ✔ Installation complete
  

• Add a namespace label to instruct Istio to automatically inject Envoy sidecar proxies when you deploy your application later:

[vagrant@master ~]$ kubectl label namespace default istio-injection=enabled

• check pod

[vagrant@master ~]$ kubectl get pods -n istio-system
NAME                                    READY   STATUS    RESTARTS   AGE
istio-ingressgateway-64f6f9d5c6-74m5t   1/1     Running   0          2m27s
istiod-5bb879d86c-rzx6d                 1/1     Running   0          3m47s
prometheus-77b9c64b9c-rzdxl             2/2     Running   0          2m27s

Running the Book info Application

• clone the project (the original version here https://github.com/auth0-blog/istio-auth0.git) but the original version hasn't been upgrade k8s's resources.

[vagrant@master ~]$ git clone https://github.com/Charnnarong/istio-auth0.git 

• cd it

[vagrant@master ~]$ cd istio-auth0

• Apply bookinfo.yaml

[vagrant@master istio-auth0]$ k apply -f platform/kube/bookinfo.yaml

• Get Pod. ( it takes times. approximately 3 minutes)

[vagrant@master istio-auth0]$ kubectl get pods
NAME                              READY   STATUS    RESTARTS   AGE
details-v1-b64c6587c-mlcm2        2/2     Running   0          3m1s
productpage-v1-6949c86b7f-dmbkf   2/2     Running   0          3m1s
ratings-v1-568cf588f7-299mx       2/2     Running   0          3m1s
reviews-v1-84dcf85db4-hzrp5       2/2     Running   0          3m1s
reviews-v2-8586bf7b84-w57gk       2/2     Running   0          3m1s
reviews-v3-5b7c57bfcb-m5mzz       2/2     Running   0          3m1s

• deploy Istio's gateway.

[vagrant@master istio-auth0]$ k apply -f networking/bookinfo-gateway.yaml

• deploy Istio's virtual services.

[vagrant@master istio-auth0]$ k apply -f networking/bookinfo-virtualservice.yaml

• check istio-ingressgateway

[vagrant@master istio-auth0]$ k get svc -n istio-system istio-ingressgateway
NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)
                  AGE
istio-ingressgateway   LoadBalancer   10.110.85.170   <pending>     15020:31979/TCP,80:31234/TCP,443:30161/TCP,15029:31666/TCP,15030:31244/TCP,15031:30831/TCP,15032:30713/TCP,31400:32065/TCP,15443:32706/TCP   13h

• access via port 80 which is mapped to 31234

Supporting SignIn and SignUp.

• create dir for keycloak volume

[vagrant@node3 ~]$ mkdir /tmp/keycloak

• run key keycloak

[vagrant@node3 ~]$ docker run -d -p 8180:8080 -p 8280:8443 -e KEYCLOAK_USER=admin -e KEYCLOAK_PASSWORD=admin -v /tmp/keycloak/:/tmp --name my-keycloak jboss/keycloak:9.0.3

• access keycloak admin https://172.42.42.30:8280/
• add realm bookshop

• add book-ui client

• update issuer and jwk uri, in file security\bookinfo-policy.yaml

apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
  name: bookinfo
spec:
  targets:
  - name: reviews
  - name: ratings
  - name: details
  origins:
  - jwt:
      # issuer: "https://{YOUR_DOMAIN}/"
      issuer: "https://172.42.42.30:8280/auth/realms/bookshop"
      # jwksUri: "https://{YOUR_DOMAIN}/.well-known/jwks.json"
      jwksUri: "https://172.42.42.30:8280/auth/realms/bookshop/protocol/openid-connect/certs"
  principalBinding: USE_ORIGIN

• apply book security info

[vagrant@master istio-auth0]$ k apply -f security/bookinfo-policy.yaml

• try to access product page again you will see an error

• Enabling Service to Service Authentication

[vagrant@master istio-auth0]$ k apply -f networking/destination-rule-mtls.yaml
#--- output---
destinationrule.networking.istio.io/reviews created
destinationrule.networking.istio.io/ratings created
destinationrule.networking.istio.io/details created

Enable user to authenticate Through Keycloak.

• add user to keycloak

• set bob's password to bob

• client secret here.

• add requirement to project. src\productpage\requirements.txt

authlib==0.10
six==1.11.0

• modified file src\productpage\productpage.py

from authlib.flask.client import OAuth
from six.moves.urllib.parse import urlencode

# Bootstrap(app)

# AUTH0_CALLBACK_URL = "http://{YOUR-CLUSTER-PUBLIC-IP}/callback"
AUTH0_CALLBACK_URL = "http://172.42.42.99:31234/callback"

# AUTH0_CLIENT_ID = "{YOUR-APPLICATION-CLIENT-ID}"
AUTH0_CLIENT_ID = "book-ui"

# AUTH0_CLIENT_SECRET = "{YOUR-APPLICATION-CLIENT-SECRET}"
AUTH0_CLIENT_SECRET = "b35e1fab-0675-4f83-9e96-61c1a31ac3f9"

# AUTH0_DOMAIN = "{YOUR-AUTH0-DOMAIN}"
AUTH0_DOMAIN = "172.42.42.30:8280"
AUTH0_BASE_URL = 'https://' + AUTH0_DOMAIN

# AUTH0_AUDIENCE = "{YOUR-AUDIENCE}"
AUTH0_AUDIENCE = "account"

oauth = OAuth(app)
auth0 = oauth.register(
    'auth0',
    client_id=AUTH0_CLIENT_ID,
    client_secret=AUTH0_CLIENT_SECRET,
    api_base_url=AUTH0_BASE_URL,
    # https://172.42.42.30:8280/auth/realms/bookshop/protocol/openid-connect/token
    access_token_url=AUTH0_BASE_URL + '/auth/realms/bookshop/protocol/openid-connect/token',
    # https://172.42.42.30:8280/auth/realms/bookshop/protocol/openid-connect/auth
    authorize_url=AUTH0_BASE_URL + '/auth/realms/bookshop/protocol/openid-connect/auth',
    client_kwargs={
        'scope': 'openid profile',
    },
)

• The file should looks like:

• still on the src/productpage/productpage.py
• add some routes (diff below.)

@app.route('/login')
def login():
    return auth0.authorize_redirect(redirect_uri=AUTH0_CALLBACK_URL, 
                                    audience=AUTH0_AUDIENCE)
@app.route('/callback')
def callback():
    response = auth0.authorize_access_token()          # 1
    session['access_token'] = response['access_token'] # 2
    userinfoResponse = auth0.get('userinfo')           # 3
    userinfo = userinfoResponse.json()
    session['user'] = userinfo['nickname']             # 4
    return redirect('/productpage')

@app.route('/logout')
def logout():
    session.clear()
    # params = {'returnTo': url_for('front', _external=True),
    #           'client_id': AUTH0_CLIENT_ID}
    params = {'redirect_uri': url_for('front', _external=True)}
    # return redirect(auth0.api_base_url + '/v2/logout?' + urlencode(params))
    #   https://172.42.42.30:8280/auth/realms/bookshop/protocol/openid-connect/logout
    return redirect(auth0.api_base_url + '/auth/realms/bookshop/protocol/openid-connect/logout?' + urlencode(params))

• add access token in header (image below)

def getForwardHeaders(request):
    headers = {}

    if 'access_token' in session:
        headers['Authorization'] = 'Bearer ' + session['access_token']

Deploying the New Version of the Product Page

• build and push into private docker repo

[vagrant@master istio-auth0]$ docker build -t private.repo.com:5086/productpage:istio-auth0 ./src/productpage

• push the image.

[vagrant@master istio-auth0]$ docker push private.repo.com:5086/productpage:istio-auth0

• Check if it in the repo.

• Check if productpage-v1 is still in k8s :D

[vagrant@master istio-auth0]$ k get deployments productpage-v1
NAME             READY   UP-TO-DATE   AVAILABLE   AGE
productpage-v1   1/1     1            1          xxh

• set the new image to it.

kubectl set image deployment/productpage-v1 \
  productpage=private.repo.com:5086/productpage:istio-auth0

#--- output ----
deployment.apps/productpage-v1 image updated

• login (user: bob , pass: bob)

• Get error: This is happening because, by default, Istio blocks unexpected outgoing requests (in this case, Istio is blocking your web application from communicating with Keycloak to get details about who logged in)

Unexpected problem Istio Side-car Unhealthy

When istio sidecar is not running

 [vagrant@master istio-auth0]$ k get po
  NAME                              READY   STATUS    RESTARTS   AGE
  details-v1-b64c6587c-576bj        1/2     Running   0          38m
#---
[vagrant@master istio-auth0]$ k describe pod details-v1-b64c6587c-576bj
#...
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  #...
  Warning  Unhealthy  25s (x1200 over 40m)  kubelet, node2     Readiness probe failed: HTTP probe failed with statuscode: 503

To fix it.

[vagrant@master bin]$ pwd
/home/vagrant/istio-1.5.1/bin
[vagrant@master bin]$ ./istioctl manifest apply --set values.sidecarInjectorWebhook.rewriteAppHTTPProbe=true
[vagrant@master istio-auth0]$ k delete -f platform/kube/bookinfo.yaml
[vagrant@master istio-auth0]$ k create -f platform/kube/bookinfo.yaml
#..... keep deleting and reapply for security\bookinfo-policy.yaml, networking\*.yaml and re-setimage for productpage.

Defining an Istio Egress

• apply Egress

[vagrant@master istio-auth0]$ k apply -f networking/auth0-egress.yaml
Error from server: error when creating "networking/auth0-egress.yaml": admission webhook "validation.istio.io" denied the request: configuration is invalid: 2 errors occurred:
        * domain name "42.42.30" invalid (top level domain "30" cannot be all-numeric)
        * hosts must be FQDN if no endpoints are provided for resolution mode DNS

• add name to keycloak.

[root@master istio-auth0]# echo "172.42.42.30 keycloakhost" >> /etc/hosts
[root@master istio-auth0]# cat /etc/hosts
172.42.42.99 master
172.42.42.20 private.repo.com
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
172.42.42.30 keycloakhost

[root@node2 vagrant]# echo "172.42.42.30 keycloakhost" >> /etc/hosts
[root@node3 vagrant]# echo "172.42.42.30 keycloakhost" >> /etc/hosts

• modified networking\auth0-egress.yaml; replace 172.42.42.30 to keycloakhost
• reapply auth0-egress.yaml

[vagrant@master istio-auth0]$ k apply -f networking/auth0-egress.yaml
serviceentry.networking.istio.io/auth0-ext created
virtualservice.networking.istio.io/keycloakhost created

• update security\bookinfo-policy.yaml domain :

- jwt:
    # issuer: "https://{YOUR_DOMAIN}/"
    # issuer: "https://172.42.42.30:8280/auth/realms/bookshop"
    issuer: "https://keycloakhost:8280/auth/realms/bookshop"
    # jwksUri: "https://{YOUR_DOMAIN}/.well-known/jwks.json"
    # jwksUri: "https://172.42.42.30:8280/auth/realms/bookshop/protocol/openid-connect/certs"
    jwksUri: "https://keycloakhost:8280/auth/realms/bookshop/protocol/openid-connect/certs"

• reapply it.

[vagrant@master istio-auth0]$ k apply -f security/bookinfo-policy.yaml

• login

Caveat

After sign out, Book details and Book Reviews are suppose to be broken, but it is still work. Looks like there is something about Jwt and MutualTLS. Some Istio's tickets mentioned that Istion version 1.5 has some improvement, while the yaml resources were written with an older version. Anyway this conclude how Istio can work with OAuth and OpenID server.

EOF