Init the go-learning project

.github/workflows/main.yml

@@ -0,0 +1,47 @@
+# This is a basic workflow to help you get started with Actions
+
+name: CI
+
+# Controls when the workflow will run
+on:
+  # Triggers the workflow on push or pull request events but only for the master branch
+  push:
+    branches: [ master ]
+  pull_request:
+    branches: [ master ]
+
+  # Allows you to run this workflow manually from the Actions tab
+  workflow_dispatch:
+
+# A workflow run is made up of one or more jobs that can run sequentially or in parallel
+jobs:
+  misspell-check:
+    runs-on: ubuntu-latest
+    container: pouchcontainer/pouchlinter:v0.1.2
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+
+#      - name: Lint markdown files
+#        run: find  ./ -name  "*.md" | xargs mdl -r ~MD010,~MD013,~MD022,~MD024,~MD029,~MD031,~MD032,~MD033,~MD034,~MD036
+
+  golang-lint:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+        with:
+          submodules: true
+
+      - name: Set up Golang
+        uses: actions/setup-go@v2
+        with:
+          go-version: 1.16
+
+      - name: Run go fmt test
+        run: hack/verify-gofmt.sh
+        env:
+          GO111MODULE: auto
+
+#      - name: Run go-learning unit test
+#        run: go test -v ./...

.gitignore

@@ -0,0 +1,4 @@
+# Files generated by JetBrains
+.idea/
+.DS_Store
+.vscode/

README.md

@@ -0,0 +1,38 @@
+# Go-Learning Overview
+
+Go-Learning's mission statement is:
+
+    To provide a learning and practise map for OpenStack, kubernetes, golang and the others.
+
+go-learning 适用于有一定 `kubernetes` 经验，且想更进一步的同学。
+
+- kubernetes 功能增强 [Pixiu(貔貅)](https://github.com/caoyingjunz/pixiu)
+- 快速部署 [kubez-ansible](https://github.com/caoyingjunz/kubez-ansible)
+- workload 自动扩缩容 [piuxiu-autoscaler](https://github.com/caoyingjunz/pixiu-autoscaler)
+
+## Kubernetes
+- [kubectl plugin 源码分析](./doc/kubernetes/kubectl-plugin.md)
+- [scheduler 源码分析一 - 启动](./doc/kubernetes/scheduler-start.md)
+- [scheduler 源码分析二 - 调度](./doc/kubernetes/scheduler-schedule.md)
+- [kubernetes 网络分析](./doc/kubernetes/network.md)
+- [kube-proxy 源码分析](./doc/kubernetes/kube-proxy.md)
+- [operator 用法展示](./doc/kubernetes/operator.md)
+- [CSI 注册机制源码分析](./doc/kubernetes/csi.md)
+- [cloud-provider-openstack](https://github.com/kubernetes/cloud-provider-openstack)
+
+## Examples
+- [Examples](./examples/README.md) 提供丰富的 `kubernetes` 用法举例.
+- [pixiuctl](https://github.com/caoyingjunz/go-learning/tree/master/practise/cobra-practise/pixiuctl) 基于 [cobra](https://github.com/spf13/cobra) 实现命令行
+  - subcommand
+  - plugin
+- [gRPC Usage](./practise/grpc-practise/README.md)
+- [gin&informer](./practise/k8s-practise/gin-informer.go) 提供 `gin` 调用 `informer` 的用法
+
+## TODO
+- scheduler 代码分析(WIP)
+- kubelet 代码分析
+- 微服务学习（istio）
+- gc 机制分析
+- pod 驱逐代码分析
+
+Copyright 2019 caoyingjun ([email protected]) Apache License 2.0

algorithm/README.rst

@@ -0,0 +1 @@
+# Algorithm

doc/kubernetes/controller.md

@@ -0,0 +1,51 @@
+### Kubernetes Defined Controllers
+
+1. CRD 的定义 参考 `sample-controller`, 创建 `CRD` 和 其对应的 `Foo`.
+
+[crd.yaml](https://github.com/kubernetes/sample-controller/blob/master/artifacts/examples/crd.yaml)
+```bash
+apiVersion: apiextensions.k8s.io/v1beta1
+kind: CustomResourceDefinition
+metadata:
+  name: foos.samplecontroller.k8s.io
+spec:
+  group: samplecontroller.k8s.io
+  version: v1alpha1
+  names:
+    kind: Foo
+    # 复数名
+    plural: foos
+    # 单数名
+    singular: foo
+    # 简称，类似 deployment 的简称是 deploy
+    shortNames：
+    - fo
+  scope: Namespaced
+```
+
+[example-foo.yaml](https://github.com/kubernetes/sample-controller/blob/master/artifacts/examples/example-foo.yaml)
+
+```bash
+apiVersion: samplecontroller.k8s.io/v1alpha1
+kind: Foo
+metadata:
+  name: example-foo
+spec:
+  deploymentName: example-foo
+  replicas: 1
+```
+
+`controller 流程`
+![流程图](./../pictures/crd.jpeg)
+
+`ectd 命令 example`
+```bash
+ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt --key=/etc/kubernetes/pki/etcd/healthcheck-client.key get /registry/apiextensions.k8s.io/customresourcedefinitions/ --prefix
+```
+
+```bash
+ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt --key=/etc/kubernetes/pki/etcd/healthcheck-client.key get /registry/bolingcavalry.k8s.io/students/default/object-student --print-value-only
+ ```
+
+
+

doc/kubernetes/csi.md

@@ -0,0 +1,185 @@
+# CSI Plugin 注册机制源码分析
+
+### node-driver-registrar 流程解析
+
+1. rpc 调用 nfs-csi-plugin 中的 identity server 中的 GetPluginInfo 方法，返回我们自研plugin相关的基本信息， 获取 csi driver name
+
+2. 启动 grpc server，并监听在宿主机的 /var/lib/kubelet/plugins_registry/${csiDriverName}-reg.sock （对应容器的/registration/${driver-name}-reg.sock）
+
+3. 该 grpc server 提供 GetInfo 和 NotifyRegistrationStatus 方法供 kubelet plugin manager 调用，
+
+### kubelet plugin manager 流程解析
+
+1. 当 ds 部署的 node-driver-registrar sidecar 启动时，/var/lib/kubelet/plugins_registry 会新增一个 socket file nfs.csi.k8s.io-reg.sock
+
+2. 这个 nfs.csi.k8s.io-reg.sock 会被 plugin watcher 监听并写入desiredStateOfWorld缓存中（通过github.com/fsnotify/fsnotify实现）
+
+3. reconciler 会对比缓存，找出需要新增或者删除的 csi plugin。当新增时，reconciler 会通过 grpc client（/var/lib/kubelet/plugins_registry）调用 node-driver-registrar sidecar container中rpc server提供的的GetInfo，然后根据返回字段的type，找到对应的 plugin hander，
+
+```go
+// socketPath 为 /registration/${driver-name}-reg.sock
+client, conn, err := dial(socketPath, dialTimeoutDuration)
+        // 调用node-driver-registrar sidecar container中rpc server提供的的GetInfo
+        infoResp, err := client.GetInfo(ctx, &registerapi.InfoRequest{})
+        // 这里handler就是上文说的CSIPlugin type的csi.RegistrationHandler{}对象
+        handler, ok := pluginHandlers[infoResp.Type]
+        // 调用handler.ValidatePlugin
+        if err := handler.ValidatePlugin(infoResp.Name, infoResp.Endpoint, infoResp.SupportedVersions); err != nil {
+        }
+        ...
+        // 加入actualStateOfWorldUpdater缓存
+        err = actualStateOfWorldUpdater.AddPlugin(cache.PluginInfo{
+            SocketPath: socketPath,
+            Timestamp:  timestamp,
+            Handler:    handler,
+            Name:       infoResp.Name,
+        })
+// infoResp.Endpoint 就是 nfs-csi 的 socket path
+if err := handler.RegisterPlugin(infoResp.Name, infoResp.Endpoint, infoResp.SupportedVersions); err != nil {
+            return og.notifyPlugin(client, false, fmt.Sprintf("RegisterPlugin error -- plugin registration failed with err: %v", err))
+        }
+    ...
+    return registerPluginFunc
+```
+
+4. plugin handler 会根据传入的 csi-plugin 监听的 socket path，直接和我们 nfs-csi-plugin 通信，并调用该对象的 ValidatePlugin 和 RegisterPlugin 来注册插件，这里的注册插件其实就是设置 node annotation 和 创建/更新CSINode对象。
+
+```bash
+metadata:
+ annotations:
+   csi.volume.kubernetes.io/nodeid: '{"nfs.csi.k8s.io":"kube-master"}'
+
+# kubectl get csinode
+NAME           DRIVERS   AGE
+kubez-master   1         101d
+kubez-node1    1         99d
+kubez-node2    1         101d
+```
+
+5. kubelet plugin-magner 通过 rpc 调用 NotifyRegistrationStatus 告知 node-driver-registrar 注册结果。
+
+### external-provisioner 流程解析
+
+1. external-provisioner 是运行在 k8s 集群内部的 controller，它作为 sidecar container 和 csi-nfs 运行在同一个 pod 中.
+
+```bash
+containers:
+- args:
+ - -v=2
+ - --csi-address=$(ADDRESS)
+ - --leader-election
+ env:
+ - name: ADDRESS
+   value: /csi/csi.sock
+ image: jacky06/csi-provisioner:v2.1.0
+ name: csi-provisioner
+ volumeMounts:
+ - mountPath: /csi
+   name: socket-dir
+- args:
+ - -v=5
+ - --nodeid=$(NODE_ID)
+ - --endpoint=$(CSI_ENDPOINT)
+ env:
+ - name: NODE_ID
+   valueFrom:
+     fieldRef:
+       apiVersion: v1
+       fieldPath: spec.nodeName
+ - name: CSI_ENDPOINT
+   value: unix:///csi/csi.sock
+ image: mcr.microsoft.com/k8s/csi/nfs-csi:latest
+```
+
+2. external-provisioner 通过 informer 机制去监听 pvc/pv，当新建 pv 或者 pvc 的时候，external-provisioner 会通过 grpc (/csi/csi.sock) 调用 nfs 的 CreateVolume(DeleteVolume)方法来实际创建一个外部存储 volume
+
+```go
+// NewCSIProvisioner creates new CSI provisioner.
+//
+// vaLister is optional and only needed when VolumeAttachments are
+// meant to be checked before deleting a volume.
+func NewCSIProvisioner(client kubernetes.Interface,
+	connectionTimeout time.Duration,
+	...
+) controller.Provisioner {
+    ...
+	csiClient := csi.NewControllerClient(grpcClient)
+
+	provisioner := &csiProvisioner{
+		client:                                client,
+		grpcClient:                            grpcClient,
+		csiClient:                             csiClient,
+		snapshotClient:                        snapshotClient,
+		...
+		driverName:                            driverName,
+		pluginCapabilities:                    pluginCapabilities,
+		controllerCapabilities:                controllerCapabilities,
+        ...
+		scLister:                              scLister,
+		csiNodeLister:                         csiNodeLister,
+		nodeLister:                            nodeLister,
+		claimLister:                           claimLister,
+		vaLister:                              vaLister,
+		eventRecorder:                         eventRecorder,
+	}
+    ...
+	return provisioner
+}
+
+// 创建
+func (p *csiProvisioner) Provision(ctx context.Context, options controller.ProvisionOptions) (*v1.PersistentVolume, controller.ProvisioningState, error) {
+    ...
+	createCtx := markAsMigrated(ctx, result.migratedVolume)
+	createCtx, cancel := context.WithTimeout(createCtx, p.timeout)
+	defer cancel()
+	rep, err := p.csiClient.CreateVolume(createCtx, req)
+    ...
+}
+
+// 删除
+func (p *csiProvisioner) Delete(ctx context.Context, volume *v1.PersistentVolume) error {
+    ...
+	_, err = p.csiClient.DeleteVolume(deleteCtx, &req)
+    ...
+```
+
+### kubelet 通过调用 csiClient 去完成 pvc 的挂载
+
+```go
+type csiClient interface {
+        NodeGetInfo(ctx context.Context) (
+                nodeID string,
+                maxVolumePerNode int64,
+                accessibleTopology map[string]string,
+                err error)
+        NodePublishVolume(
+                ctx context.Context,
+                volumeid string,
+                readOnly bool,
+                stagingTargetPath string,
+                targetPath string,
+                accessMode api.PersistentVolumeAccessMode,
+                publishContext map[string]string,
+                volumeContext map[string]string,
+                secrets map[string]string,
+                fsType string,
+                mountOptions []string,
+        ) error
+        NodeExpandVolume(ctx context.Context, volumeid, volumePath string, newSize resource.Quantity) (resource.Quantity, error)
+        NodeUnpublishVolume(
+                ctx context.Context,
+                volID string,
+                targetPath string,
+        ) error
+        NodeStageVolume(ctx context.Context,
+                volID string,
+                publishVolumeInfo map[string]string,
+                stagingTargetPath string,
+                fsType string,
+                accessMode api.PersistentVolumeAccessMode,
+                secrets map[string]string,
+                volumeContext map[string]string,
+                mountOptions []string,
+        ) error
+...
+```