Programming Kubernetes: Developing Cloud-Native Applications

spec: toppings: - mozzarella - tomato We register the preceding CRD and then create the margherita object: $ kubectl create -f pizza-crd.yaml $ kubectl create -f margherita-pizza.yaml As expected, we get back the same object for both versions: $ kubectl get pizza margherita -o yaml apiVersion: restaurant.programming-kubernetes.info/v1beta1 kind: Pizza metadata: creationTimestamp: \"2019-04-14T11:39:20Z\" generation: 1 name: margherita namespace: pizza-apiserver resourceVersion: \"47959\" selfLink: /apis/restaurant.programming- kubernetes.info/v1beta1/namespaces/pizza-apiserver/ pizzas/margherita uid: f18427f0-5ea9-11e9-8219-124e4d2dc074 spec: toppings: - mozzarella - tomato Kubernetes uses the canonical version order; that is: v1alpha1 Unstable: might go away or change any time and often disabled by default. v1beta1 Towards stable: exists at least in one release in parallel to v1; contract: no incompatible API changes.

v1 Stable or generally available (GA): will stay for good, and will be compatible. The GA versions come first in that order, then the betas, and then the alphas, with the major version ordered from high to low and the same for the minor version. Every CRD version not fitting into this pattern comes last, ordered alphabetically. In our case, the preceding kubectl get pizza therefore returns v1beta1, although the created object was in version v1alpha1. Conversion Webhook Architecture Now let’s add the conversion from v1alpha1 to v1beta1 and back. CRD conversions are implemented via webhooks in Kubernetes. Figure 9-2 shows the flow: 1. The client (e.g., our kubectl get pizza margherita) requests a version. 2. etcd has stored the object in some version. 3. If the versions do not match, the storage object is sent to the webhook server for conversion. The webhook returns a response with the converted object. 4. The converted object is sent back to the client. Figure 9-2. Conversion webhook

We have to implement this webhook server. Before doing so, let’s look at the webhook API. The Kubernetes API server sends a ConversionReview object in the API group apiextensions.k8s.io/v1beta1: type ConversionReview struct { metav1.TypeMeta `json:\",inline\"` Request *ConversionRequest Response *ConversionResponse } The request field is set in the payload sent to the webhook. The response field is set in the response. The request looks like this: type ConversionRequest struct { ... // `desiredAPIVersion` is the version to convert given objects to. // For example, \"myapi.example.com/v1.\" DesiredAPIVersion string // `objects` is the list of CR objects to be converted. Objects []runtime.RawExtension } The DesiredAPIVersion string has the usual apiVersion format we know from TypeMeta: group/version. The objects field has a number of objects. It is a slice because for one list request for pizzas, the webhook will receive one conversion request, with this slice being all objects for the list request. The webhook converts and sets the response: type ConversionResponse struct { ... // `convertedObjects` is the list of converted versions of

`request.objects` // if the `result` is successful otherwise empty. The webhook is expected to // set apiVersion of these objects to the ConversionRequest.desiredAPIVersion. // The list must also have the same size as input list with the same objects // in the same order (i.e. equal UIDs and object meta). ConvertedObjects []runtime.RawExtension // `result` contains the result of conversion with extra details if the // conversion failed. `result.status` determines if the conversion failed // or succeeded. The `result.status` field is required and represents the // success or failure of the conversion. A successful conversion must set // `result.status` to `Success`. A failed conversion must set `result.status` // to `Failure` and provide more details in `result.message` and return http // status 200. The `result.message` will be used to construct an error // message for the end user. Result metav1.Status } The result status tells the Kubernetes API server whether the conversion was successful. But when in the request pipeline is our conversion webhook actually called? What kind of input object can we expect? To understand this better, take a look at the general request pipeline in Figure 9-3: all those solid and striped circles are where conversion takes place in the k8s.io/apiserver code.

Figure 9-3. Conversion webhook calls for CRs In contrast to aggregated custom API servers (see “Internal Types and Conversion”), CRs do not use internal types but convert directly between the external API versions. Hence, only those yellow circles are actually doing conversions in Figure 9-4; the solid circles are NOOPs for CRDs. In other words: CRD conversion takes place only from and to etcd.

Figure 9-4. Where conversion takes place for CRs Therefore, we can assume our webhook will be called from those two places in the request pipeline (refer to Figure 9-3). Also note that patch requests do automatic retries on conflict (updates cannot retry, and they respond with errors directly to the caller). Each retry consists of a read and write to etcd (the yellow circles in Figure 9-3) and therefore leads to two calls to the webhook per iteration. WARNING All the warnings about the criticality of conversion in “Conversions” apply here as well: conversions must be correct. Bugs quickly lead to data loss and inconsistent behavior of the API. Before we start implementing the webhook, some final words about what the webhook can do and must avoid:

The order of the objects in request and response must not change. ObjectMeta with the exception of labels and annotation must not be mutated. Conversion is all or nothing: either all objects are successfully converted or all fail. Conversion Webhook Implementation With the theory behind us, we are ready to start the implementation of the webhook project. You can find the source at the repository, which includes: A webhook implementation as an HTTPS web server A number of endpoints: /convert/v1beta1/pizza converts a pizza object between v1alpha1 and v1beta1. /admit/v1beta1/pizza defaults the spec.toppings field to mozzarella, tomato, salami. /validate/v1beta1/pizza verifies that each specified topping has a corresponding toppings object. The last two endpoints are admission webhooks, which will be discussed in detail in “Admission Webhooks”. The same webhook binary will serve both admission and conversion. The v1beta1 in these paths should not be confused with v1beta1 of our restaurant API group, but it is meant as the apiextensions.k8s.io API group version we support as a webhook. Someday v1 of that webhook API will be supported,1 at which time we’ll add the corresponding v1 as another endpoint, in order to support old (as of today) and new Kubernetes clusters. It is possible

to specify inside the CRD manifest which versions a webhook supports. Let’s look into how this conversion webhook actually works. Afterwards we will take a deeper dive into how to deploy the webhook into a real cluster. Note again that webhook conversion is still alpha in 1.14 and must be enabled manually using the CustomResourceWebhookConversion feature gate, but it is available as beta in 1.15. Setting Up the HTTPS Server The first step is to start a web server with support for transport layer security, or TLS (i.e., HTTPS). Webhooks in Kubernetes require HTTPS. The conversion webhook even requires certificates that are successfully checked by the Kubernetes API server against the CA bundle provided in the CRD object. In the example project, we make use of the secure serving library that is part of the k8s.io/apiserver. It provides all TLS flags and behavior you might be used to from deploying a kube-apiserver or an aggregated API server binary. The k8s.io/apiserver secure serving code follows the options-config pattern (see “Options and Config Pattern and Startup Plumbing”). It is very easy to embed that code into your own binary: func NewDefaultOptions() *Options { o := &Options{ *options.NewSecureServingOptions(), } o.SecureServing.ServerCert.PairName = \"pizza-crd-webhook\" return o } type Options struct { SecureServing options.SecureServingOptions }

type Config struct { SecureServing *server.SecureServingInfo } func (o *Options) AddFlags(fs *pflag.FlagSet) { o.SecureServing.AddFlags(fs) } func (o *Options) Config() (*Config, error) { err := o.SecureServing.MaybeDefaultWithSelfSignedCerts(\"0.0.0.0\", nil, nil) if err != nil { return nil, err } c := &Config{} if err := o.SecureServing.ApplyTo(&c.SecureServing); err != nil { return nil, err } return c, nil } In the main function of the binary, this Options struct is instantiated and wired to a flag set: opt := NewDefaultOptions() fs := pflag.NewFlagSet(\"pizza-crd-webhook\", pflag.ExitOnError) globalflag.AddGlobalFlags(fs, \"pizza-crd-webhook\") opt.AddFlags(fs) if err := fs.Parse(os.Args); err != nil { panic(err) } // create runtime config cfg, err := opt.Config() if err != nil { panic(err) } stopCh := server.SetupSignalHandler() ...

// run server restaurantInformers.Start(stopCh) if doneCh, err := cfg.SecureServing.Serve( handlers.LoggingHandler(os.Stdout, mux), time.Second * 30, stopCh, ); err != nil { panic(err) } else { <-doneCh } In place of the three dots, we set up the HTTP multiplexer with our three paths as follows: // register handlers restaurantInformers := restaurantinformers.NewSharedInformerFactory( clientset, time.Minute * 5, ) mux := http.NewServeMux() mux.Handle(\"/convert/v1beta1/pizza\", http.HandlerFunc(conversion.Serve)) mux.Handle(\"/admit/v1beta1/pizza\", http.HandlerFunc(admission.ServePizzaAdmit)) mux.Handle(\"/validate/v1beta1/pizza\", http.HandlerFunc(admission.ServePizzaValidation(restaurantInformers))) restaurantInformers.Start(stopCh) As the pizza validation webhook at the path /validate/v1beta1/pizza has to know the existing topping objects in the cluster, we instantiate a shared informer factory for the restaurant.programming- kubernetes.info API group. Now we’ll look at the actual conversion webhook implementation behind conversion.Serve. It is a normal Golang HTTP handler function, meaning it gets a request and a response writer as arguments. The request body contains a ConversionReview object from the API group apiextensions.k8s.io/v1beta1. Hence, we have to first read

the body from the request, and then decode the byte slice. We do this by using a deserializer from API Machinery: func Serve(w http.ResponseWriter, req *http.Request) { // read body body, err := ioutil.ReadAll(req.Body) if err != nil { responsewriters.InternalError(w, req, fmt.Errorf(\"failed to read body: %v\", err)) return } // decode body as conversion review gv := apiextensionsv1beta1.SchemeGroupVersion reviewGVK := gv.WithKind(\"ConversionReview\") obj, gvk, err := codecs.UniversalDeserializer().Decode(body, &reviewGVK, &apiextensionsv1beta1.ConversionReview{}) if err != nil { responsewriters.InternalError(w, req, fmt.Errorf(\"failed to decode body: %v\", err)) return } review, ok := obj.(*apiextensionsv1beta1.ConversionReview) if !ok { responsewriters.InternalError(w, req, fmt.Errorf(\"unexpected GroupVersionKind: %s\", gvk)) return } if review.Request == nil { responsewriters.InternalError(w, req, fmt.Errorf(\"unexpected nil request\")) return } ... } This code makes use of the codec factory codecs, which is derived from a scheme. This scheme has to include the types of apiextensions.k8s.io/v1beta1. We also add the types of our restaurant API group. The passed ConversionReview object will have

our pizza type embedded in a runtime.RawExtension type—more about that in a second. First let’s create our scheme and the codec factory: import ( apiextensionsv1beta1 \"k8s.io/apiextensions- apiserver/pkg/apis/apiextensions/v1beta1\" \"github.com/programming-kubernetes/pizza- crd/pkg/apis/restaurant/install\" ... ) var ( scheme = runtime.NewScheme() codecs = serializer.NewCodecFactory(scheme) ) func init() { utilruntime.Must(apiextensionsv1beta1.AddToScheme(scheme)) install.Install(scheme) } A runtime.RawExtension is a wrapper for Kubernetes-like objects embedded in a field of another object. Its structure is actually very simple: type RawExtension struct { // Raw is the underlying serialization of this object. Raw []byte `protobuf:\"bytes,1,opt,name=raw\"` // Object can hold a representation of this extension - useful for working // with versioned structs. Object Object `json:\"-\"` } In addition, runtime.RawExtension has special JSON and protobuf marshaling two methods. Moreover, there is special logic around the conversion to runtime.Object on the fly, when converting to internal types—that is, automatic encoding and decoding.

In this case of CRDs, we don’t have internal types, and therefore that conversion magic does not play a role. Only RawExtension.Raw is filled with a JSON byte slice of the pizza object sent to the webhook for conversion. Thus, we will have to decode this byte slice. Note again that one ConversionReview potentially carries a number of objects, such that we have to loop over all of them: // convert objects review.Response = &apiextensionsv1beta1.ConversionResponse{ UID: review.Request.UID, Result: metav1.Status{ Status: metav1.StatusSuccess, }, } var objs []runtime.Object for _, in := range review.Request.Objects { if in.Object == nil { var err error in.Object, _, err = codecs.UniversalDeserializer().Decode( in.Raw, nil, nil, ) if err != nil { review.Response.Result = metav1.Status{ Message: err.Error(), Status: metav1.StatusFailure, } break } } obj, err := convert(in.Object, review.Request.DesiredAPIVersion) if err != nil { review.Response.Result = metav1.Status{ Message: err.Error(), Status: metav1.StatusFailure, } break } objs = append(objs, obj) }

The convert call does the actual conversion of in.Object, with the desired API version as the target version. Note here that we break the loop immediately when the first error occurs. Finally, we set the Response field in the ConversionReview object and write it back as the response body of the request using API Machinery’s response writer, which again uses our codec factory to create a serializer: if review.Response.Result.Status == metav1.StatusSuccess { for _, obj = range objs { review.Response.ConvertedObjects = append(review.Response.ConvertedObjects, runtime.RawExtension{Object: obj}, ) } } // write negotiated response responsewriters.WriteObject( http.StatusOK, gvk.GroupVersion(), codecs, review, w, req, ) Now, we have to implement the actual pizza conversion. After all this plumbing above, the conversion algorithm is the easiest part. It just checks that we actually got a pizza object of the known versions and then does the conversion from v1beta1 to v1alpha1 and vice versa: func convert(in runtime.Object, apiVersion string) (runtime.Object, error) { switch in := in.(type) { case *v1alpha1.Pizza: if apiVersion != v1beta1.SchemeGroupVersion.String() { return nil, fmt.Errorf(\"cannot convert %s to %s\", v1alpha1.SchemeGroupVersion, apiVersion) } klog.V(2).Infof(\"Converting %s/%s from %s to %s\", in.Namespace, in.Name, v1alpha1.SchemeGroupVersion, apiVersion) out := &v1beta1.Pizza{

TypeMeta: in.TypeMeta, ObjectMeta: in.ObjectMeta, Status: v1beta1.PizzaStatus{ Cost: in.Status.Cost, }, } out.TypeMeta.APIVersion = apiVersion idx := map[string]int{} for _, top := range in.Spec.Toppings { if i, duplicate := idx[top]; duplicate { out.Spec.Toppings[i].Quantity++ continue } idx[top] = len(out.Spec.Toppings) out.Spec.Toppings = append(out.Spec.Toppings, v1beta1.PizzaTopping{ Name: top, Quantity: 1, }) } return out, nil case *v1beta1.Pizza: if apiVersion != v1alpha1.SchemeGroupVersion.String() { return nil, fmt.Errorf(\"cannot convert %s to %s\", v1beta1.SchemeGroupVersion, apiVersion) } klog.V(2).Infof(\"Converting %s/%s from %s to %s\", in.Namespace, in.Name, v1alpha1.SchemeGroupVersion, apiVersion) out := &v1alpha1.Pizza{ TypeMeta: in.TypeMeta, ObjectMeta: in.ObjectMeta, Status: v1alpha1.PizzaStatus{ Cost: in.Status.Cost, }, } out.TypeMeta.APIVersion = apiVersion for i := range in.Spec.Toppings { for j := 0; j < in.Spec.Toppings[i].Quantity; j++ { out.Spec.Toppings = append(

out.Spec.Toppings, in.Spec.Toppings[i].Name) } } return out, nil default: } klog.V(2).Infof(\"Unknown type %T\", in) return nil, fmt.Errorf(\"unknown type %T\", in) } Note that in both directions of the conversion, we just copy TypeMeta and ObjectMeta, change the API version to the desired one, and then convert the toppings slice, which is actually the only part of the objects which structurally differs. If there are more versions, another two-way conversion is necessary between all of them. Alternatively, of course, we could use a hub version as in aggregated API servers (see “Internal Types and Conversion”), instead of implementing conversions from and to all supported external versions. Deploying the Conversion Webhook We now want to deploy the conversion webhook. You can find all the manifests on GitHub. Conversion webhooks for CRDs are launched in the cluster and put behind a service object, and that service object is referenced by the conversion webhook specification in the CRD manifest: apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: pizzas.restaurant.programming-kubernetes.info spec: ... conversion: strategy: Webhook

webhookClientConfig: caBundle: BASE64-CA-BUNDLE service: namespace: pizza-crd name: webhook path: /convert/v1beta1/pizza The CA bundle must match the serving certificate used by the webhook. In our example project, we use a Makefile to generate certificates using OpenSSL and plug them into the manifests using text replacement. Note here that the Kubernetes API server assumes that the webhook supports all specified versions of the CRD. There is also only one such webhook possible per CRD. But as CRDs and conversion webhooks are usually owned by the same team, this should be enough. Also note that the service port must be 443 in the current apiextensions.k8s.io/v1beta1 API. The service can map this, however, to any port used by the webhook pods. In our example, we map 443 to 8443, served by the webhook binary. Seeing Conversion in Action Now that we understand how the conversion webhook works and how it is wired into the cluster, let’s see it in action. We assume you’ve checked out the example project. In addition, we assume that you have a cluster with webhook conversion enabled (either via feature gate in a 1.14 cluster or through a 1.15+ cluster, which has webhook conversion enabled by default). One way to get such a cluster is via the kind project, which provides support for Kubernetes 1.14.1 and a local kind-config.yaml file to enable the alpha feature gate for webhook conversion (“What Does Programming Kubernetes Mean?” linked a number of other options for development clusters):

kind: Cluster apiVersion: kind.sigs.k8s.io/v1alpha3 kubeadmConfigPatchesJson6902: - group: kubeadm.k8s.io version: v1beta1 kind: ClusterConfiguration patch: | - op: add path: /apiServer/extraArgs value: {} - op: add path: /apiServer/extraArgs/feature-gates value: CustomResourceWebhookConversion=true Then we can create a cluster: $ kind create cluster --image kindest/node-images:v1.14.1 --config kind-config.yaml $ export KUBECONFIG=\"$(kind get kubeconfig-path --name=\"kind\")\" Now we can deploy our manifests: $ cd pizza-crd $ cd manifest/deployment $ make $ kubectl create -f ns.yaml $ kubectl create -f pizza-crd.yaml $ kubectl create -f topping-crd.yaml $ kubectl create -f sa.yaml $ kubectl create -f rbac.yaml $ kubectl create -f rbac-bind.yaml $ kubectl create -f service.yaml $ kubectl create -f serving-cert-secret.yaml $ kubectl create -f deployment.yaml These manifests contain the following files: ns.yaml Creates the pizza-crd namespace. pizza-crd.yaml

Specifies the pizza resource in the restaurant.programming- kubernetes.info API group, with the v1alpha1 and v1beta1 versions, and the webhook conversion configuration as shown previously. topping-crd.yaml Specifies the toppings CR in the same API group, but only in the v1alpha1 version. sa.yaml Introduces the webhook service account. rbac.yaml Defines a role to read, list, and watch toppings. rbac-bind.yaml Binds the earlier RBAC role to the webhook service account. service.yaml Defines the webhook services, mapping port 443 to 8443 of the webhook pods. serving-cert-secret.yaml Contains the serving certificate and private key to be used by the webhook pods. The certificate is also used directly as the CA bundle in the preceding pizza CRD manifest. deployment.yaml Launches webhook pods, passing --tls-cert-file and --tls- private-key the serving certificate secret. After this we can create a margherita pizza finally: $ cat ../examples/margherita-pizza.yaml apiVersion: restaurant.programming-kubernetes.info/v1alpha1

kind: Pizza metadata: name: margherita spec: toppings: - mozzarella - tomato $ kubectl create ../examples/margherita-pizza.yaml pizza.restaurant.programming-kubernetes.info/margherita created Now, with the conversion webhook in place, we can retrieve the same object in both versions. First explicitly in the v1alpha1 version: $ kubectl get pizzas.v1alpha1.restaurant.programming-kubernetes.info \\ margherita -o yaml apiVersion: restaurant.programming-kubernetes.info/v1alpha1 kind: Pizza metadata: creationTimestamp: \"2019-04-14T21:41:39Z\" generation: 1 name: margherita namespace: pizza-crd resourceVersion: \"18296\" pizzas/margherita uid: 15c1c06a-5efe-11e9-9230-0242f24ba99c spec: toppings: - mozzarella - tomato status: {} Then the same object as v1beta1 shows the different toppings structure: $ kubectl get pizzas.v1beta1.restaurant.programming-kubernetes.info \\ margherita -o yaml apiVersion: restaurant.programming-kubernetes.info/v1beta1 kind: Pizza metadata: creationTimestamp: \"2019-04-14T21:41:39Z\" generation: 1 name: margherita

namespace: pizza-crd resourceVersion: \"18296\" pizzas/margherita uid: 15c1c06a-5efe-11e9-9230-0242f24ba99c spec: toppings: - name: mozzarella quantity: 1 - name: tomato quantity: 1 status: {} Meanwhile, in the log of the webhook pod we see this conversion call: I0414 21:46:28.639707 1 convert.go:35] Converting pizza- crd/margherita from restaurant.programming-kubernetes.info/v1alpha1 to restaurant.programming-kubernetes.info/v1beta1 10.32.0.1 - - [14/Apr/2019:21:46:28 +0000] \"POST /convert/v1beta1/pizza?timeout=30s HTTP/2.0\" 200 968 Hence, the webhook is doing its job as expected. Admission Webhooks In “Use Cases for Custom API Servers” we discussed the use cases in which an aggregated API server is a better choice than using CRs. A lot of the reasons given are about having the freedom to implement certain behavior using Golang instead of being restricted to declarative features in CRD manifests. We have seen in the previous section how Golang is used to build CRD conversion webhooks. A similar mechanism is used to add custom admission to CRDs, again in Golang. Basically we have the same freedom as with custom admission plug- ins in aggregated API servers (see “Admission”): there are mutating

and validating admission webhooks, and they are called at the same position as for native resources, as shown in Figure 9-5. Figure 9-5. Admission in the CR request pipeline We saw CRD validation based on OpenAPI in “Validating Custom Resources”. In Figure 9-5, validation is done in the box labeled “Validation.” The validating admission webhooks are called after that, the mutating admission webhooks before. The admission webhooks are put nearly at the end of the admission plug-in order, before quota. Admission webhooks are beta in Kubernetes 1.14 and therefore available in most clusters. TIP For v1 of the admission webhooks API, it is planned to allow up to two passes through the admission chain. This means that an earlier admission plug-in or webhook can depend on the output of later plug-ins or webhooks, to a certain degree. So, in the future this mechanism will get even more powerful.

Admission Requirements in the Restaurant Example The restaurant example uses admission for multiple things: spec.toppings defaults if it is nil or empty to mozzarella, tomato, and salami. Unknown fields should be dropped from the CR JSON and not persisted in etcd. spec.toppings must contain only toppings that have a corresponding topping object. The first two use cases are mutating; the third use case is purely validating. Therefore, we will use one mutating webhook and one validating webhook to implement those steps. NOTE Work is in progress on native defaulting via OpenAPI v3 validation schemas. OpenAPI has a default field, and the API server will apply that in the future. Moreover, dropping unknown fields will become the standard behavior for every resource, done by the Kubernetes API server through a mechanism called pruning. Pruning is available as beta in Kubernetes 1.15. Defaulting is planned to be available as beta in 1.16. When both features are available in the target cluster, the two use cases from the preceding list can be implemented without any webhook at all. Admission Webhook Architecture Admission webhooks are structurally very similar to the conversion webhooks we saw earlier in the chapter. They are deployed in the cluster, put behind a service mapping port 443 to some port of the pods, and called using a review object,

AdmissionReview in the API group admission.k8s.io/v1beta1: --- // AdmissionReview describes an admission review request/response. type AdmissionReview struct { metav1.TypeMeta `json:\",inline\"` // Request describes the attributes for the admission request. // +optional Request *AdmissionRequest `json:\"request,omitempty\"` // Response describes the attributes for the admission response. // +optional Response *AdmissionResponse `json:\"response,omitempty\"` } --- The AdmissionRequest contains all the information we are used to from the admission attributes (see “Implementation”): // AdmissionRequest describes the admission.Attributes for the admission request. type AdmissionRequest struct { // UID is an identifier for the individual request/response. It allows us to // distinguish instances of requests which are otherwise identical (parallel // requests, requests when earlier requests did not modify etc). The UID is // meant to track the round trip (request/response) between the KAS and the // WebHook, not the user request. It is suitable for correlating log entries // between the webhook and apiserver, for either auditing or debugging. UID types.UID `json:\"uid\"` // Kind is the type of object being manipulated. For example: Pod Kind metav1.GroupVersionKind `json:\"kind\"` // Resource is the name of the resource being requested. This is not the // kind. For example: pods Resource metav1.GroupVersionResource `json:\"resource\"` // SubResource is the name of the subresource being requested. This is a // different resource, scoped to the parent resource, but it may

have a // different kind. For instance, /pods has the resource \"pods\" and the kind // \"Pod\", while /pods/foo/status has the resource \"pods\", the sub resource // \"status\", and the kind \"Pod\" (because status operates on pods). The // binding resource for a pod though may be /pods/foo/binding, which has // resource \"pods\", subresource \"binding\", and kind \"Binding\". // +optional SubResource string `json:\"subResource,omitempty\"` // Name is the name of the object as presented in the request. On a CREATE // operation, the client may omit name and rely on the server to generate // the name. If that is the case, this method will return the empty string. // +optional Name string `json:\"name,omitempty\"` // Namespace is the namespace associated with the request (if any). // +optional Namespace string `json:\"namespace,omitempty\"` // Operation is the operation being performed Operation Operation `json:\"operation\"` // UserInfo is information about the requesting user UserInfo authenticationv1.UserInfo `json:\"userInfo\"` // Object is the object from the incoming request prior to default values // being applied // +optional Object runtime.RawExtension `json:\"object,omitempty\"` // OldObject is the existing object. Only populated for UPDATE requests. // +optional OldObject runtime.RawExtension `json:\"oldObject,omitempty\"` // DryRun indicates that modifications will definitely not be persisted // for this request. // Defaults to false. // +optional DryRun *bool `json:\"dryRun,omitempty\"` }

The same AdmissionReview object is used for both mutating and validating admission webhooks. The only difference is that in the mutating case, the AdmissionResponse can have a field patch and patchType, to be applied inside the Kubernetes API server after the webhook response has been received there. In the validating case, these two fields are kept empty on response. The most important field for our purposes here is the Object field, which—as in the preceding conversion webhook—uses the runtime.RawExtension type to store a pizza object. We also get the old object for update requests and could, say, check for fields that are meant to be read-only but are changed in a request. We don’t do this here in our example. But you will encounter many cases in Kubernetes where such logic is implemented—for example, for most fields of a pod, as you can’t change the command of a pod after it is created. The patch returned by the mutating webhook must be of type JSON Patch (see RFC 6902) in Kubernetes 1.14. This patch describes how the object should be modified to fulfill the required invariant. Note that it is best practice to validate every mutating webhook change in a validating webhook at the very end, at least if those enforced properties are significant for the behavior. Imagine some other mutating webhook touches the same fields in an object. Then you cannot be sure that the mutating changes will survive until the end of the mutating admission chain. There is no order currently in mutating webhooks other than alphabetic order. There are ongoing discussions to change this in one way or another in the future. For validating webhooks the order does not matter, obviously, and the Kubernetes API server even calls validating webhooks in parallel to reduce latency. In contrast, mutating webhooks add latency to

every request that passes through them, as they are called sequentially. Common latencies—of course heavily depending on the environment—are around 100ms. So running many webhooks in sequence leads to considerable latencies that the user will experience when creating or updating objects. Registering Admission Webhooks Admission webhooks are not registered in the CRD manifest. The reason is that they apply not only to CRDs, but to any kind of resource. You can even add custom admission webhooks to standard Kubernetes resources. Instead there are registration objects: MutatingWebhookRegistration and ValidatingWebhookRegistration. They differ only in the kind name: apiVersion: admissionregistration.k8s.io/v1beta1 kind: MutatingWebhookConfiguration metadata: name: restaurant.programming-kubernetes.info webhooks: - name: restaurant.programming-kubernetes.info failurePolicy: Fail sideEffects: None admissionReviewVersions: - v1beta1 rules: - apiGroups: - \"restaurant.programming-kubernetes.info\" apiVersions: - v1alpha1 - v1beta1 operations: - CREATE - UPDATE resources: - pizzas clientConfig:

service: namespace: pizza-crd name: webhook path: /admit/v1beta1/pizza caBundle: CA-BUNDLE This registers our pizza-crd webhook from the beginning of the chapter for mutating admission for our two versions of the resource pizza, the API group restaurant.programming-kubernetes.info, and the HTTP verbs CREATE and UPDATE (which includes patches as well). There are further ways in webhook configurations to restrict the matching resources—for example, a namespace selector (to exclude, e.g., a control plane namespace to avoid bootstrapping issues) and more advanced resource patterns with wildcards and subresources. Last but not least is a failure mode, which can be either Fail or Ignore. It specifies what to do if the webhook cannot be reached or fails for other reasons. WARNING Admission webhooks can break clusters if they are deployed in the wrong way. Admission webhook matching core types can make the whole cluster inoperable. Special care must be taken to call admission webhooks for non- CRD resources. Specifically, it is good practice to exclude the control plane and the webhook resources themselves from the webhook. Implementing an Admission Webhook With the work we’ve done on the conversion webhook in the beginning of the chapter, it is not hard to add admission capabilities. We also saw that the paths /admit/v1beta1/pizza and

/validate/v1beta1/pizza are registered in the main function of the pizza-crd-webhook binary: mux.Handle(\"/admit/v1beta1/pizza\", http.HandlerFunc(admission.ServePizzaAdmit)) mux.Handle(\"/validate/v1beta1/pizza\", http.HandlerFunc( admission.ServePizzaValidation(restaurantInformers))) The first part of the two HTTP handler implementations looks nearly the same as for the conversion webhook: func ServePizzaAdmit(w http.ResponseWriter, req *http.Request) { // read body body, err := ioutil.ReadAll(req.Body) if err != nil { responsewriters.InternalError(w, req, fmt.Errorf(\"failed to read body: %v\", err)) return } // decode body as admission review reviewGVK := admissionv1beta1.SchemeGroupVersion.WithKind(\"AdmissionReview\") decoder := codecs.UniversalDeserializer() into := &admissionv1beta1.AdmissionReview{} obj, gvk, err := decoder.Decode(body, &reviewGVK, into) if err != nil { responsewriters.InternalError(w, req, fmt.Errorf(\"failed to decode body: %v\", err)) return } review, ok := obj.(*admissionv1beta1.AdmissionReview) if !ok { responsewriters.InternalError(w, req, fmt.Errorf(\"unexpected GroupVersionKind: %s\", gvk)) return } if review.Request == nil { responsewriters.InternalError(w, req, fmt.Errorf(\"unexpected nil request\")) return }

... } In the case of the validating webhook, we have to wire the informer (used to check that toppings exist in the cluster). We return an internal error as long as the informer is not synced. An informer that is not synced has incomplete data, so the toppings might not be known and the pizza would be rejected although they are valid: func ServePizzaValidation(informers restaurantinformers.SharedInformerFactory) func (http.ResponseWriter, *http.Request) { toppingInformer := informers.Restaurant().V1alpha1().Toppings().Informer() toppingLister := informers.Restaurant().V1alpha1().Toppings().Lister() return func(w http.ResponseWriter, req *http.Request) { if !toppingInformer.HasSynced() { responsewriters.InternalError(w, req, fmt.Errorf(\"informers not ready\")) return } // read body body, err := ioutil.ReadAll(req.Body) if err != nil { responsewriters.InternalError(w, req, fmt.Errorf(\"failed to read body: %v\", err)) return } // decode body as admission review gv := admissionv1beta1.SchemeGroupVersion reviewGVK := gv.WithKind(\"AdmissionReview\") obj, gvk, err := codecs.UniversalDeserializer().Decode(body, &reviewGVK, &admissionv1beta1.AdmissionReview{}) if err != nil { responsewriters.InternalError(w, req, fmt.Errorf(\"failed to decode body: %v\", err)) return

} review, ok := obj.(*admissionv1beta1.AdmissionReview) if !ok { responsewriters.InternalError(w, req, fmt.Errorf(\"unexpected GroupVersionKind: %s\", gvk)) return } if review.Request == nil { responsewriters.InternalError(w, req, fmt.Errorf(\"unexpected nil request\")) return } ... } } As in the webhook conversion case, we have set up the scheme and the codec factory with the admission API group and our restaurant API group: var ( scheme = runtime.NewScheme() codecs = serializer.NewCodecFactory(scheme) ) func init() { utilruntime.Must(admissionv1beta1.AddToScheme(scheme)) install.Install(scheme) } With these two, we decode the embedded pizza object (this time only one, no slice) from the AdmissionReview: // decode object if review.Request.Object.Object == nil { var err error review.Request.Object.Object, _, err = codecs.UniversalDeserializer().Decode(review.Request.Object.Raw, nil, nil) if err != nil { review.Response.Result = &metav1.Status{

Message: err.Error(), Status: metav1.StatusFailure, } responsewriters.WriteObject(http.StatusOK, gvk.GroupVersion(), codecs, review, w, req) return } } Then we can do the actual mutating admission (the defaulting of spec.toppings for both API versions): orig := review.Request.Object.Raw var bs []byte switch pizza := review.Request.Object.Object.(type) { case *v1alpha1.Pizza: // default toppings if len(pizza.Spec.Toppings) == 0 { pizza.Spec.Toppings = []string{\"tomato\", \"mozzarella\", \"salami\"} } bs, err = json.Marshal(pizza) if err != nil { responsewriters.InternalError(w, req, fmt.Errorf\"unexpected encoding error: %v\", err)) return } case *v1beta1.Pizza: // default toppings if len(pizza.Spec.Toppings) == 0 { pizza.Spec.Toppings = []v1beta1.PizzaTopping{ {\"tomato\", 1}, {\"mozzarella\", 1}, {\"salami\", 1}, } } bs, err = json.Marshal(pizza) if err != nil { responsewriters.InternalError(w, req, fmt.Errorf(\"unexpected encoding error: %v\", err)) return }

default: review.Response.Result = &metav1.Status{ Message: fmt.Sprintf(\"unexpected type %T\", review.Request.Object.Object), Status: metav1.StatusFailure, } responsewriters.WriteObject(http.StatusOK, gvk.GroupVersion(), codecs, review, w, req) return } Alternatively, we could use the conversion algorithms from the conversion webhook and then implement defaulting only for one of the versions. Both approaches are possible, and which one makes more sense depends on the context. Here, the defaulting is simple enough to implement it twice. The final step is to compute the patch—the difference between the original object (stored in orig as JSON) and the new defaulted one: // compare original and defaulted version ops, err := jsonpatch.CreatePatch(orig, bs) if err != nil { responsewriters.InternalError(w, req, fmt.Errorf(\"unexpected diff error: %v\", err)) return } review.Response.Patch, err = json.Marshal(ops) if err != nil { responsewriters.InternalError(w, req, fmt.Errorf(\"unexpected patch encoding error: %v\", err)) return } typ := admissionv1beta1.PatchTypeJSONPatch review.Response.PatchType = &typ review.Response.Allowed = true We use the JSON-Patch library (a fork of Matt Baird’s with critical fixes) to derive the patch from the original object orig and the modified object bs, both passed as JSON byte slices. Alternatively, we could operate directly on untyped JSON data and create the

JSON-Patch manually. Again, it depends on the context. Using a diff library is convenient. Then, as in the webhook conversion, we conclude by writing the response to the response writer, using the codec factory created previously: responsewriters.WriteObject( http.StatusOK, gvk.GroupVersion(), codecs, review, w, req, ) The validating webhook is very similar, but it uses the toppings lister from the shared informer to check for the existence of the topping objects: switch pizza := review.Request.Object.Object.(type) { case *v1alpha1.Pizza: for _, topping := range pizza.Spec.Toppings { _, err := toppingLister.Get(topping) if err != nil && !errors.IsNotFound(err) { responsewriters.InternalError(w, req, fmt.Errorf(\"failed to lookup topping %q: %v\", topping, err)) return } else if errors.IsNotFound(err) { review.Response.Result = &metav1.Status{ Message: fmt.Sprintf(\"topping %q not known\", topping), Status: metav1.StatusFailure, } responsewriters.WriteObject(http.StatusOK, gvk.GroupVersion(), codecs, review, w, req) return } } review.Response.Allowed = true case *v1beta1.Pizza: for _, topping := range pizza.Spec.Toppings { _, err := toppingLister.Get(topping.Name) if err != nil && !errors.IsNotFound(err) { responsewriters.InternalError(w, req, fmt.Errorf(\"failed to lookup topping %q: %v\", topping,

err)) return } else if errors.IsNotFound(err) { review.Response.Result = &metav1.Status{ Message: fmt.Sprintf(\"topping %q not known\", topping), Status: metav1.StatusFailure, } responsewriters.WriteObject(http.StatusOK, gvk.GroupVersion(), codecs, review, w, req) return } } review.Response.Allowed = true default: review.Response.Result = &metav1.Status{ Message: fmt.Sprintf(\"unexpected type %T\", review.Request.Object.Object), Status: metav1.StatusFailure, } } responsewriters.WriteObject(http.StatusOK, gvk.GroupVersion(), codecs, review, w, req) Admission Webhook in Action We deploy the two admission webhooks by creating the two registration objects in the cluster: $ kubectl create -f validatingadmissionregistration.yaml $ kubectl create -f mutatingadmissionregistration.yaml After this, we can’t create pizzas with unknown toppings anymore: $ kubectl create -f ../examples/margherita-pizza.yaml Error from server: error when creating \"../examples/margherita- pizza.yaml\": admission webhook \"restaurant.programming-kubernetes.info\" denied the request: topping \"tomato\" not known Meanwhile, in the webhook log we see:

I0414 22:45:46.873541 1 pizzamutation.go:115] Defaulting pizza- crd/ in version admission.k8s.io/v1beta1, Kind=AdmissionReview 10.32.0.1 - - [14/Apr/2019:22:45:46 +0000] \"POST /admit/v1beta1/pizza?timeout=30s HTTP/2.0\" 200 871 10.32.0.1 - - [14/Apr/2019:22:45:46 +0000] \"POST /validate/v1beta1/pizza?timeout=30s HTTP/2.0\" 200 956 After creating the toppings in the example folder, we can create the margherita pizza again: $ kubectl create -f ../examples/topping-tomato.yaml $ kubectl create -f ../examples/topping-salami.yaml $ kubectl create -f ../examples/topping-mozzarella.yaml $ kubectl create -f ../examples/margherita-pizza.yaml pizza.restaurant.programming-kubernetes.info/margherita created Last but not least, let’s check that defaulting works as expected. We want to create an empty pizza: apiVersion: restaurant.programming-kubernetes.info/v1alpha1 kind: Pizza metadata: name: salami spec: This is supposed to be defaulted to a salami pizza, and it is: $ kubectl create -f ../examples/empty-pizza.yaml pizza.restaurant.programming-kubernetes.info/salami created $ kubectl get pizza salami -o yaml apiVersion: restaurant.programming-kubernetes.info/v1beta1 kind: Pizza metadata: creationTimestamp: \"2019-04-14T22:49:40Z\" generation: 1 name: salami namespace: pizza-crd resourceVersion: \"23227\" uid: 962e2dda-5f07-11e9-9230-0242f24ba99c spec: toppings:

- name: tomato quantity: 1 - name: mozzarella quantity: 1 - name: salami quantity: 1 status: {} Voilà, a salami pizza with all the toppings that we expect. Enjoy! Before concluding the chapter, we want to look toward an apiextensions.k8s.io/v1 API group version (i.e., nonbeta, general availability) of CRDs—namely, the introduction of structural schemas. Structural Schemas and the Future of CustomResourceDefinitions From Kubernetes 1.15 on, the OpenAPI v3 validation schema (see “Validating Custom Resources”) is getting a more central role for CRDs in the sense that it will be mandatory to specify a schema if any of these new features is used: CRD conversion (see Figure 9-2) Pruning (see “Pruning Versus Preserving Unknown Fields”) Defaulting (see “Default Values”) OpenAPI Schema Publishing Strictly speaking, the definition of a schema is still optional and every existing CRD will keep working, but without a schema your CRD is excluded from any new feature. In addition, the specified schema must follow certain rules to enforce that the specified types are actually sane in the sense of adhering to the Kubernetes API conventions. We call these structural schema.

Structural Schemas A structural schema is an OpenAPI v3 validation schema (see “Validating Custom Resources”) that obeys the following rules: 1. The schema specifies a nonempty type (via type in OpenAPI) for the root, for each specified field of an object node (via properties or additionalProperties in OpenAPI), and for each item in an array node (via items in OpenAPI), with the exception of: A node with x-kubernetes-int-or-string: true A node with x-kubernetes-preserve-unknown- fields: true 2. For each field in an object and each item in an array, which is set within an allOf, anyOf, oneOf, or not, the schema also specifies the field/item outside of those logical junctors. 3. The schema does not set description, type, default, additionProperties, or nullable within an allOf, anyOf, oneOf, or not, with the exception of the two patterns for x- kubernetes-int-or-string: true (see “IntOrString and RawExtensions”). 4. If metadata is specified, then only restrictions on metadata.name and metadata.generateName are allowed. Here is an example that is not structural: properties: foo: pattern: \"abc\" metadata: type: object properties: name:

type: string pattern: \"^a\" finalizers: type: array items: type: string pattern: \"my-finalizer\" anyOf: - properties: bar: type: integer minimum: 42 required: [\"bar\"] description: \"foo bar object\" It is not a structural schema because of the following violations: The type at the root is missing (rule 1). The type of foo is missing (rule 1). bar inside of anyOf is not specified outside (rule 2). bar’s type is within anyOf (rule 3). The description is set within anyOf (rule 3). metadata.finalizer might not be restricted (rule 4). In contrast, the following, corresponding schema is structural: type: object description: \"foo bar object\" properties: foo: type: string pattern: \"abc\" bar: type: integer metadata: type: object properties: name:

type: string pattern: \"^a\" anyOf: - properties: bar: minimum: 42 required: [\"bar\"] Violations of the structural schema rules are reported in the NonStructural condition in the CRD. Verify for yourself that the schema of the cnat example in “Validating Custom Resources” and the schemas in the pizza CRD example are indeed structural. Pruning Versus Preserving Unknown Fields CRDs traditionally store any (possibly validated) JSON as is in etcd. This means that unspecified fields (if there is an OpenAPI v3 validation schema at all) will be persisted. This is in contrast to native Kubernetes resources like a pod. If the user specifies a field spec.randomField, this will be accepted by the API server HTTPS endpoint but dropped (we call this pruning) before writing that pod to etcd. If a structural OpenAPI v3 validation schema is defined (either in the global spec.validation.openAPIV3Schema or for each version), we can enable pruning (which drops unspecified fields on creation and on update) by setting spec.preserveUnknownFields to false. Let’s look at the cnat example.2 With a Kubernetes 1.15 cluster at hand, we enable pruning: apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: ats.cnat.programming-kubernetes.info spec:

... preserveUnknownFields: false Then we try to create an instance with an unknown field: apiVersion: cnat.programming-kubernetes.info/v1alpha1 kind: At metadata: name: example-at spec: schedule: \"2019-07-03T02:00:00Z\" command: echo \"Hello, world!\" someGarbage: 42 If we retrieve this object with kubectl get at example-at, we see that the someGarbage value is dropped: apiVersion: cnat.programming-kubernetes.info/v1alpha1 kind: At metadata: name: example-at spec: schedule: \"2019-07-03T02:00:00Z\" command: echo \"Hello, world!\" We say that someGarbage has been pruned. As of Kubernetes 1.15, pruning is available in apiextensions/v1beta1, but it defaults to off; that is, spec.preserveUnknownFields defaults to true. In apiextensions/v1, no new CRD with spec.preserveUnknownFields: true will be allowed to be created. Controlling Pruning With spec.preserveUnknownField: false in the CRD, pruning is enabled for all CRs of that type and in all versions. It is possible, though, to opt out of pruning for a JSON subtree via x-kubernetes-

preserve-unknown-fields: true in the OpenAPI v3 validation schema: type: object properties: json: x-kubernetes-preserve-unknown-fields: true The field json can store any JSON value, without anything being pruned. It is possible to partially specify the permitted JSON: type: object properties: json: x-kubernetes-preserve-unknown-fields: true type: object description: this is arbitrary JSON With this approach, only object type values are allowed. Pruning is enabled again for each specified property (or additionalProperties): type: object properties: json: x-kubernetes-preserve-unknown-fields: true type: object properties: spec: type: object properties: foo: type: string bar: type: string With this, the value:

json: spec: foo: abc bar: def something: x status: something: x will be pruned to: json: spec: foo: abc bar: def status: something: x This means that the something field in the specified spec object is pruned (because “spec” is specified), but everything outside is not. status is not specified such that status.something is not pruned. IntOrString and RawExtensions There are situations where structural schemas are not expressive enough. One of those is a polymorphic field—one that can be of different types. We know IntOrString from native Kubernetes API types. It is possible to have IntOrString in CRDs using the x-kubernetes- int-or-string: true directive inside the schema. Similarly, runtime.RawExtensions can be declared using the x-kubernetes- embedded-object: true. For example: type: object properties: intorstr: type: object

x-kubernetes-int-or-string: true embedded: x-kubernetes-embedded-object: true x-kubernetes-preserve-unknown-fields: true This declares: A field called intorstr that holds either an integer or a string A field called embedded that holds a Kubernetes-like object such as a complete pod specification Refer to the official CRD documentation for all the details about these directives. The last topic we want to talk about that depends on structural schemas is defaulting. Default Values In native Kubernetes types, it is common to default certain values. Defaulting used to be possible for CRDs only by way of mutating admission webhooks (see “Admission Webhooks”). As of Kubernetes 1.15, however, defaulting support is added (see the design document) to CRDs directly via the OpenAPI v3 schema described in the previous section. NOTE As of 1.15 this is still an alpha feature, meaning it’s disabled by default behind the feature gate CustomResourceDefaulting. But with promotion to beta, probably in 1.16, it will become ubiquitous in CRDs. In order to default certain fields, just specify the default value via the default keyword in the OpenAPI v3 schema. This is very useful when you are adding new fields to a type.

Starting with the schema of the cnat example from “Validating Custom Resources”, let’s assume we want to make the container image customizable, but default to a busybox image. For that we add the image field of string type to the OpenAPI v3 schema and set the default to busybox: type: object properties: apiVersion: type: string kind: type: string metadata: type: object spec: type: object properties: schedule: type: string pattern: \"^\\d{4}-([0]\\d|1[0-2])-([0-2]\\d|3[01])...\" command: type: string image: type: string default: \"busybox\" required: - schedule - command status: type: object properties: phase: type: string required: - metadata - apiVersion - kind - spec If the user creates an instance without specifying the image, the value is automatically set:

apiVersion: cnat.programming-kubernetes.info/v1alpha1 kind: At metadata: name: example-at spec: schedule: \"2019-07-03T02:00:00Z\" command: echo \"hello world!\" On creation, this turns automatically into: apiVersion: cnat.programming-kubernetes.info/v1alpha1 kind: At metadata: name: example-at spec: schedule: \"2019-07-03T02:00:00Z\" command: echo \"hello world!\" image: busybox This looks super convenient and significantly improves the user experience of CRDs. What’s more, all old objects persisted in etcd will automatically inherit the new field when read from the API server.3 Note that persisted objects in etcd will not be rewritten (i.e., migrated automatically). In other words, on read the default values are only added on the fly and are only persisted when the object is updated for another reason. Summary Admission and conversion webhooks take CRDs to a completely different level. Before these features, CRs were mostly used for small, not-so-serious use cases, often for configuration and for in- house applications where API compatibility was not that important. With webhooks CRs look much more like native resources, with a long lifecycle and powerful semantics. We have seen how to

implement dependencies between different resources and how to set defaulting of fields. At this point you probably have a lot of ideas about where these features can be used in existing CRDs. We are curious to see the innovations of the community based on these features in the future. 1 apiextensions.k8s.io and admissionregistration.k8s.io are both scheduled to be promoted to v1 in Kubernetes 1.16. 2 We use the cnat example instead of the pizza example due to the simple structure of the former—for example, there’s only one version. Of course, all of this scales to multiple versions (i.e., one schema version). 3 For example, via kubectl get ats -o yaml.

Appendix A. Resources General The official Kubernetes Documentation The Kubernetes community on GitHub The client-go docs channel on the Kubernetes Slack instance Kubernetes deep dive: API Server – part 1 Kubernetes deep dive: API Server – part 2 Kubernetes deep dive: API Server – part 3 Kubernetes API Server, Part I The Mechanics of Kubernetes GoDoc for k8s.io/api Books Kubernetes: Up and Running, 2nd Edition by Kelsey Hightower et al. (O’Reilly) Cloud Native DevOps with Kubernetes by John Arundel and Justin Domingus (O’Reilly) Managing Kubernetes by Brendan Burns and Craig Tracey (O’Reilly) Kubernetes Cookbook by Sébastien Goasguen and Michael Hausenblas (O’Reilly)

The Kubebuilder Book Tutorials and Examples Kubernetes by Example The Katacoda Kubernetes Playground Banzai Cloud Operator SDK Operator Developer Guide Articles Writing a Kubernetes Operator in Golang Stay Informed with Kubernetes Informers Events, the DNA of Kubernetes Kubernetes Events Explained Level Triggering and Reconciliation in Kubernetes Comparing Kubernetes Operator Pattern with Alternatives Kubernetes Operators Kubernetes Custom Resource, Controller and Operator Development Tools Demystifying Kubernetes Operators with the Operator SDK: Part 1 Under the Hood of Kubebuilder Framework Best Practices for Building Kubernetes Operators and Stateful Apps Kubernetes Operator Development Guidelines

Mutating Webhooks with slok/kubewebhook Repositories kubernetes-client organization kubernetes/kubernetes kubernetes/perf-tests cncf/apisnoop open-policy-agent/gatekeeper stakater/Konfigurator ynqa/kubernetes-rust hossainemruz/k8s-initializer-finalizer-practice munnerz/k8s-api-pager-demo m3db/m3db-operator