k8s(十三)、企业级docker仓库Harbor在kubernetes上搭建使用

前言

Harbor是Docker镜像仓库管理系统，用于企业级管理容器镜像，支持对接ldap进行权限管理,由VMVare中国团队开发。

项目地址:
https://github.com/goharbor/harbor/

安装方式:
1.Docker Compose方式部署:
https://github.com/goharbor/harbor/blob/master/docs/installation_guide.md

2.通过kubernetes直接部署:
新版本已废弃

3.通过helm/kubernetes方式部署:
https://github.com/goharbor/harbor-helm/blob/master/README.md
推荐使用,本文将采用此方式

Helm安装

helm简介

Helm是一个用于Kubernetes的包管理工具,每一个安装包构成一个chart基本单位.想象一下,一个完整的部署,可能包含前端\后端\中间件\DB的部署实例,以及k8s管理对应的secret/configmap/svc/ing/pv/pvc等众多资源类型,在部署管理或者卸载之前,如果逐一编辑管理这些资源的yaml文件,工作量比较庞大且容易疏漏.helm的出现很好的解决了这个问题,使用模板以及变量替换的方式,按需生成部署yaml文件包,并且向kubernetes api发送调度请求,虽然额外地引入了一些学习成本,但对于部署包的管理无疑是带来了很大的便利性的.

helm组件

Helm

Helm是一个cli客户端，可以完成如下内容：

1.创建/打包/调试Chart
2.创建本地Chart仓库,管理本地和远程Chart仓库
3.与Tiller交互并完成Chart的安装，升级，删除，回滚，查看等操作

Tiller

1.监听helm客户端的请求,根据chart生成相应的release
2.将release的资源信息发送给kubernetes api,实现资源增删改查.
3.追踪release的状态,实现资源的更新/回滚操作

安装:
安装较为简单,参考官方文档,不做复述:
https://docs.helm.sh/using_helm/#installing-helm

Harbor安装

这里采用次新版本的v1.5.3版本

root@h1:# wget https://github.com/goharbor/harbor/archive/v1.5.3.tar.gz
root@h1:# tar -xf https://github.com/goharbor/harbor/archive/v1.5.3.tar.gz 
root@h1:# cd harbor-1.5.3/contrib/helm/harbor/

helm方式安装的harbor默认是https的,因此需要k8s集群ingress网关开启https协议,traefik开启https参考此前文章:
https://blog.csdn.net/ywq935/article/details/79886793

更新helm dependency

harbor的helm部署依赖了postgresql的helm，在官方的安装文档没有明确说明，直接按照官方文档说明安装，就会缺失postgresql的部署，导致helm install failed

helm dependency update

修改value.yaml文件,添加pvc持久存储并将相应组件上下文中的volume部分注释打开:

mysql:
  volumes:
    data:
    storageClass: "cephrbd"
    accessMode: ReadWriteOnce

adminserver:
  volumes:
    config:
    storageClass: "cephrbd"
    accessMode: ReadWriteOnce
    size: 1G

registry:
  volumes:
    data:
    storageClass: "cephrbd"
    accessMode: ReadWriteOnce
    size: 5G

完整value.yaml文件如下:

# Configure persisten Volumes per application
## Applications that require storage have a `volumes` definition which will be used
## when `persistence.enabled` is set to true.
## example
# mysql:
#   volumes:
#     data:
## Persistent Volume Storage Class
## If defined, storageClassName: <storageClass>
## If set to "-", storageClassName: "", which disables dynamic provisioning
## If undefined (the default) or set to null, no storageClassName spec is
##   set, choosing the default provisioner.  (gp2 on AWS, standard on
##   GKE, AWS & OpenStack)
##
#     storageClass: "-"
#     accessMode: ReadWriteOnce
#     size: 1Gi

mysql:
  volumes:
    data:
    storageClass: "cephrbd"
    accessMode: ReadWriteOnce
adminserver:
  volumes:
    config:
    storageClass: "cephrbd"
    accessMode: ReadWriteOnce
    size: 1G

registry:
  volumes:
    data:
    storageClass: "cephrbd"
    accessMode: ReadWriteOnce
    size: 5G

## Configure resource requests and limits per application
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
# mysql:
#   resources:
#     requests:
#       memory: 256Mi
#       cpu: 100m

persistence:
  enabled: true

# The tag for Harbor docker images.
harborImageTag: &harbor_image_tag v1.4.0

# The FQDN for Harbor service.
externalDomain: harbor.my.domain
# If set to true, you don't need to set tlsCrt/tlsKey/caCrt, but must add
# Harbor FQDN as insecure-registries for your docker client.
insecureRegistry: false
#insecureRegistry: true
# The TLS certificate for Harbor. The common name of tlsCrt must match the externalDomain above.
tlsCrt: 
#tlsCrt: root/traefik/ssl/kokoerp.crt
tlsKey:
#tlsKey: root/traefik/ssl/kokoerp.key
caCrt:

# The secret key used for encryption. Must be a string of 16 chars.
secretKey: not-a-secure-key

# These annotations allow the registry to work behind the nginx
# ingress controller.
ingress:
  annotations:
    ingress.kubernetes.io/ssl-redirect: "true"
    ingress.kubernetes.io/body-size: "0"
    ingress.kubernetes.io/proxy-body-size: "0"

adminserver:
  image:
    repository: vmware/harbor-adminserver
    tag: *harbor_image_tag
    pullPolicy: IfNotPresent
  emailHost: "smtp.mydomain.com"
  emailPort: "25"
  emailUser: "sample_admin@mydomain.com"
  emailSsl: "false"
  emailFrom: "admin <sample_admin@mydomain.com>"
  emailIdentity: ""
  emailInsecure: "False"
  emailPwd: not-a-secure-password
  adminPassword: Harbor12345
  authenticationMode: "db_auth"
  selfRegistration: "on"
  ldap:
    url: "ldaps://ldapserver"
    searchDN: ""
    baseDN: ""
    filter: "(objectClass=person)"
    uid: "uid"
    scope: "2"
    timeout: "5"
    verifyCert: "True"
  ## Persist data to a persistent volume
  volumes:
    config:
      storageClass: "cephrbd"
      accessMode: ReadWriteOnce
      size: 1Gi
  # resources:
  #  requests:
  #    memory: 256Mi
  #    cpu: 100m

## jobservice
#
jobservice:
  image:
    repository: vmware/harbor-jobservice
    tag: *harbor_image_tag
    pullPolicy: IfNotPresent
  secret: not-a-secure-secret
# resources:
#   requests:
#     memory: 256Mi
#     cpu: 100m

## UI
#
ui:
  image:
    repository: vmware/harbor-ui
    tag: *harbor_image_tag
    pullPolicy: IfNotPresent
  secret: not-a-secure-secret
  privateKeyPem: |
    -----BEGIN RSA PRIVATE KEY-----
    MIIJKAIBAAKCAgEA4WYbxdrFGG6RnfyYKlHYML3lEqtA9cYWWOynE9BeaEr/cMnM
    bBr1dd91/Nm6RiYhQvTDU2Kc6NejqjdliW5B9xUoVKayri8OU81a8ViXeNgKwCPR
    AiTTla1zoX5DnvoxpO9G3lxyNvTKXc0cw8NjQDAXpaDbzJYLkshCeuyD9bco8R96
    /zrpBEX8tADN3+3yA3fMcZzVXsBm4BTpHJRk/qBpHYEPSHzxyH3iGMNKk3vMUBZz
    e0EYkK8NCA2CuEKMnC3acx9IdRwkx10abGvHQCLRCVY7rGoak+b0oZ99RJIRQ9Iq
    YXsn8fsMBQly6xxvSeY5XuSP7Xb6JKDt3y8Spi4gR1M/5aEzhuOyu201rMna7Rs/
    GPfaKjBlbX0jiLDa7v4zjsBPsPaf/c4uooz3ICLsdukaom+E538R0EiOkXt/wyw2
    2YmaWNCsYlEpke7cVC33e/0dPBq4IHsVflawSF9OWS23ikVAs/n+76KjuucEDmbT
    aKUYAJjvAmZL14j+EKc/CoplhCe6pKhavjmNIOfCSdlreIPBhOVbf1f817wKoSIZ
    qVyCA1AYNkI9RYS00axtJGBGMlKbdQqCNpLL58c6To2awmckIZCEcATKOp++NoGm
    Ib0bhdSasdGB5VCtwZVluN8bLl13zBKoxTGjNlEatUGDRnDAnLdZbXXffjsCAwEA
    AQKCAgBEUigO8/4UJse6xKr3APHv7E94NjKtjMqPT8RhDCLhqAH/lRuClTVb8k0Y
    RILi6oHggsKGDvkS1vJEESCU5LfYBjDAX/r/M0I7gp6TU1AukAXKMdETvkfoMbg/
    9j7W/G152hF4KztvjwmcHyUd7aay+SDh0n1taPm/FzaXfgONwmQFmo40uQ2SfwhX
    I3tD6iMWjASLV4eRfe5w88WpJQ3r5IGYMNuKFF1RcV7MNL3xMHBAwl1kudmRWY4w
    p6+83Gc0m+2AQbY70TkQuRbeUFkIBsWn99yEqXC+7h2us+JLm57iGN1ByQvVnEwL
    Zs7Pl0Hge4leSxeZWhv+aE1R/jm/VdG4dglInuhED0ug8WAJg58IkDYfMKOOALHx
    +0CNHE02XqqUIFwboZJSYTjMYvFL1i14L30FWnqH/0kDs4whXHbnGWhVustsMSK9
    iyIGepuGhMnvtUF1wa/SrBd12qfDj68QHDXsKKbs6eTNYHfn3QL9uisrfMIa5HAt
    nX2YOsAVxg+yvxkWD6n1DU+a/+pAu6iAgiwyxSZiyn6vJUE2zO6pJNbk1kJW6jU3
    A69srtbO4jQn4EM859XYSqdqwXgJL+XJEYNbBcHalmiIOvRg9CCvDSKS7M5rJ0M1
    L7oCzl6EW+zUb4JHkSO7V5uxIZu2sEduw5gofQ3OT9L/qDhDIQKCAQEA8T/8okF2
    Q7SOj3su6KKX6H/ab31SvHECf/oeJtH8ZfLBYL55Yof0pZwq8iXQ26d8cH7FPKBo
    hz0RZ9i2S3bYkzEVCPv9ISFg1NACxL3dU0PMBnmbmg2vPhMzEuQI2JOUu6ILOXEN
    mImvfjZXps/b8OjQgzicH0skBBcbUlXT3a4fF52ktC8FiXgBG9JYg5CsXmfPRxci
    ITa4w4ZLEuECmtJieS0MdKXPLwUVv3e2BlNS6c1JzXyp6EyX/euJ8cCe3n/GHbTY
    2j1OO+xTQfQJVf6S9f2mSzjdHe9KZwWKgyxQ9dZ9Qtho2z/gUN9/UkL52fdljjlw
    ++b/z9Ppcl9K0QKCAQEA7y4Fv8dPFLLnr0R/S7eoAKa0S95xVe97EJHVUhWyOI09
    K9VdZHp6be8W0Yd9h/Ks8Zi4EPRiTTaF3yA3iADwdKFeZt49jGzeM+Gl7Q2Ll98W
    I5gOdJkHSVAP2uK7qSjZ8lPCu4iUYRsae+Psam7Yd6X17RP0M966PlUFj1nnrJjQ
    EN4zeh/m01q9vqebB9C1W/ZiJ6rpt6VVHAcOQQ69F/lKdTif4XCvbMIhIXTYNifk
    1oIv2qTDnfzzv+bgrlvpBJPpPYR0Oc7WoEpyd1Y9IzienLZi8RnujV//FXEmJ45E
    F9GE1HOmoERdEWA1bMYhOO5OfRY1HSMuFMA4+5ojSwKCAQEAmwubio/1uMemw3HQ
    kPRGGsdolDR/4tniWGtfy2UzCDY+r7Vaf8eOpIy8UQmatEBsykO+8RrKcvf9Yrc1
    WUSVJevqb+67HPq9p6fTz6uSPXwZ+KNZLGXVFVjzfxWM1dvrP7eB7TXKHhmG7t9v
    76Yw3SBTObI9LCN3jyVmisDcO+E23E+VVbPOpC260K2b81ocXUPsQ+0LIztu/UIm
    p4hyyxug6+3WznTttXNYKch+9IvCgr5Ly0NuUvw+xpMFAZjgwXBu3BKpN4Ek8YAN
    dhqnkVveCTguErQF78IlGBbIkUr+8TAbKsW4hggEWxV4V17yAnJsEz65bTtldqTj
    qHyzsQKCAQBGhv6g/2d9Rgf1cbBLpns+vel6Wbx3x6c1SptpmgY0kMlR7JeeclM5
    qX/EBzzn4pJGp27XaQi3lfVBxyE41HYTHiZVFQF3L/8Rs18XGKBqBxljI4pXrWwt
    nRMfyy3lAqvJvhM082A1hiV4FMx40fi4x1JON00SIoIusSlzjOI4zdLEtpDdWRza
    g+5hktCvLEbeODfXVJmYUoNXQWldm7f8osDm8eyLMIw5+MCGOgsrZPYgnsD3qxAX
    vSgvFSh5oZaDiA4F2tHe3fQBzhIUyHQ8t4xlz447ZBcozv7L1tKWZWgE0f5mGzgu
    GBqNbh4y1fWj8Plp/ytoTSBgdBIZdukjAoIBAELJPSVFnlf/gv6OWRCHyKxquGjv
    fEn/E8bw5WSqMcj/7wiSJozr0Y8oyWjtWXObliLRQXcEhC8w3lLMjNqnFzQOAI7s
    Oa6BQPigqyXZPXG5GK+V0TlUYvZQn9sfCq4YCxUBNtQ4GHbKKl3FGQL3rJiuFr6G
    fVcetuDFNCiIGYbUF+giJ2cEN3a/Q+7fR6V4xC7VDdL+BqM09wZ6R98G48XzCKKp
    ekNpEfmvJiuk9tFFQwDPWcQ6uyHqesK/Wiweo5nh5y2ZPipwcb0uBoYOQH60NqEL
    6MXRVNdtKujjl1XZkG053Nvcz/YfF6lFjDekwgfd9m49b/s0EGTrl7z9z8Y=
    -----END RSA PRIVATE KEY-----
# resources:
#  requests:
#    memory: 256Mi
#    cpu: 100m

## MySQL Settings. Currently Harbor does not support an external
## MySQL server, only their own image. Until this is fixed, do not
## Change the settings below.
#
mysql:
  image:
    repository: vmware/harbor-db
    tag: *harbor_image_tag
    pullPolicy: IfNotPresent
  # If left blank will use the included mysql service name.
  host: ~
  port: 3306
  user: "root"
  pass: "registry"
  database: "registry"
  volumes:
    data:
      storageClass: "cephrbd"
      accessMode: ReadWriteOnce
      size: 1Gi
  # resources:
  #  requests:
  #    memory: 256Mi
  #    cpu: 100m

registry:
  image:
    repository: vmware/registry-photon
    tag: v2.6.2-v1.4.0
    pullPolicy: IfNotPresent
  httpSecret: not-a-secure-secret
  logLevel:
# comment out one of the below to use your cloud's object storage.
#  objectStorage:
#    gcs:
#      keyfile: ""
#      bucket: ""
#      chunksize: "5242880"
#    s3:
#      region: ""
#      accesskey: ""
#      secretkey: ""
#      bucket: ""
#      encrypt: "true"
#    azure:
#      accountname: ""
#      accountkey: ""
#      container: ""
  rootCrt: |
    -----BEGIN CERTIFICATE-----
    MIIE0zCCArugAwIBAgIJAIgs3S+hsjhmMA0GCSqGSIb3DQEBCwUAMAAwHhcNMTcx
    MTA5MTcyNzQ5WhcNMjcxMTA3MTcyNzQ5WjAAMIICIjANBgkqhkiG9w0BAQEFAAOC
    Ag8AMIICCgKCAgEA4WYbxdrFGG6RnfyYKlHYML3lEqtA9cYWWOynE9BeaEr/cMnM
    bBr1dd91/Nm6RiYhQvTDU2Kc6NejqjdliW5B9xUoVKayri8OU81a8ViXeNgKwCPR
    AiTTla1zoX5DnvoxpO9G3lxyNvTKXc0cw8NjQDAXpaDbzJYLkshCeuyD9bco8R96
    /zrpBEX8tADN3+3yA3fMcZzVXsBm4BTpHJRk/qBpHYEPSHzxyH3iGMNKk3vMUBZz
    e0EYkK8NCA2CuEKMnC3acx9IdRwkx10abGvHQCLRCVY7rGoak+b0oZ99RJIRQ9Iq
    YXsn8fsMBQly6xxvSeY5XuSP7Xb6JKDt3y8Spi4gR1M/5aEzhuOyu201rMna7Rs/
    GPfaKjBlbX0jiLDa7v4zjsBPsPaf/c4uooz3ICLsdukaom+E538R0EiOkXt/wyw2
    2YmaWNCsYlEpke7cVC33e/0dPBq4IHsVflawSF9OWS23ikVAs/n+76KjuucEDmbT
    aKUYAJjvAmZL14j+EKc/CoplhCe6pKhavjmNIOfCSdlreIPBhOVbf1f817wKoSIZ
    qVyCA1AYNkI9RYS00axtJGBGMlKbdQqCNpLL58c6To2awmckIZCEcATKOp++NoGm
    Ib0bhdSasdGB5VCtwZVluN8bLl13zBKoxTGjNlEatUGDRnDAnLdZbXXffjsCAwEA
    AaNQME4wHQYDVR0OBBYEFCMYYMOL0E/Uyj5wseDfIl7o4ELsMB8GA1UdIwQYMBaA
    FCMYYMOL0E/Uyj5wseDfIl7o4ELsMAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQEL
    BQADggIBABG8fPvrrR+erpwQFuB/56j2i6sO+qoOJPpAMYwkzICrT0eerWAavwoy
    f0UAKN7cUeEJXjIR7s7CogGFijWdaWaQsXUD0zJq5aotLYZLimEc1O0uAmJEsfYC
    v7mG07eU6ge22sSo5hxhVplGt52hnXnT0DdgSRZpq2mvgd9lcopAidM+KHlaasXk
    IecHKM99KX9D8smr0AcQ6M/Ygbf2qjO9YRmpBIjyQWEake4y/4LWm+3+v08ecg4B
    g+iMC0Rw1QcPqgwaGaWu71RtYhyTg7SnAknb5nBcHIbLb0hdLgQTa3ZdtXgqchIi
    GuFlEBmHFZP6bLJORRUQ0ari5wpXIsYfrB4T8PybTzva3OCMlEsMjuysFr9ewhzM
    9UGLiSQNDyKA10J8WwlzbeD0AAW944hW4Dbg6SWv4gAo51T+6AukRdup5y6lfQ5a
    h4Lbo6pzaA369IsJBntvKvia6hUf/SghnbG7pCHX/AEilcgTb13HndF/G+7aZgKR
    mi9qvNRSDsE/BrgZawovp81+j6aL4y6UtXYspHr+SuWsKYsaH7pl5HspNCyJ5vV6
    dpJAwosFBqSEnI333wAunpMYmi/jKHH/j4WqjLnCInp0/wouzYu42l8Pmz591BSp
    Jag500bEBxqI2RLELgMt/bUdjp4N2M7mrxdrN+2579HTzb6Hviu9
    -----END CERTIFICATE-----
  ## Persist data to a persistent volume
  volumes:
    data:
      storageClass: "cephrbd"
      accessMode: ReadWriteOnce
      size: 5Gi
  # resources:
  #  requests:
  #    memory: 256Mi
  #    cpu: 100m

clair:
  enabled: true
  image:
    repository: vmware/clair-photon
    tag: v2.0.1-v1.4.0
    pullPolicy: IfNotPresent
## The following needs to match the credentials
## in the `postgresql` configuration under the
## `postgresql` namespace below.
  postgresPassword: not-a-secure-password
  postgresUser: clair
  postgresDatabase: clair
# resources:
#  requests:
#    memory: 256Mi
#    cpu: 100m
# pgResources:
#  requests:
#    memory: 256Mi
#    cpu: 100m
#  volumes:
#    pgData:
#      storageClass: "cephrbd"
#      accessMode: ReadWriteOnce
#      size: 1Gi
  # resources:
  #  requests:
  #    memory: 256Mi
  #    cpu: 100m

## Notary support is not yet fully implemented in the Helm Charts
## Enabling it will just break things.
#
notary:
  enabled: false

## Settings for postgresql dependency.
## see https://github.com/kubernetes/charts/tree/master/stable/postgresql
## for further configurables.
postgresql:
  postgresUser: clair
  postgresPassword: not-a-secure-password
  postgresDatabase: clair
  persistence:
    # enabled: false
    enabled: true
    storageClass: "cephrbd"
    accessMode: ReadWriteOnce
    size: 8Gi

部署

helm install . --debug --name hub --set externalDomain=hub.test.com

等待一段时间的镜像拉取后,查看pod:

hub-harbor-adminserver-0                 1/1       Running            1          3h
hub-harbor-clair-6c7d9dcdb7-q4lv4        1/1       Running            2          3h
hub-harbor-jobservice-75f7fbcc9c-ggwp4   1/1       Running            2          3h
hub-harbor-mysql-0                       1/1       Running            0          3h
hub-harbor-registry-0                    1/1       Running            1          3h
hub-harbor-ui-57b4674ff9-kcfq6           1/1       Running            0          3h
hub-postgresql-ccf8d56d5-jg4wq           1/1       Running            1          3h

添加dns,指向ingress gateway(即traefik的node ip):

echo "192.168.1.238  hub.test.com"  >> /etc/hosts

浏览器打开测试,默认登录口令为admin/Harbor12345

可以看到默认有一个公开项目library,尝试往这里推送镜像

Docker推送测试

找一台安装了docker的机器,修改docker服务脚本:

~# cat /lib/systemd/system/docker.service

#ExecStart=/usr/bin/dockerd -H fd://
ExecStart=/usr/bin/dockerd  --insecure-registry hub.kokoerp.com

重启docker并登录测试:

~# systemctl daemon-reload
~# systemctl restart docker

root@h2:~# docker login hub.test.com
Username (admin): admin
Password: 
Login Succeeded

推送镜像测试:

root@h2:~# docker tag busybox hub.test.com/library/busybox:test1
root@h2:~# docker push hub.test.com/library/busybox:test1
The push refers to a repository [hub.test.com/library/busybox]
8a788232037e: Pushed 
test1: digest: sha256:915f390a8912e16d4beb8689720a17348f3f6d1a7b659697df850ab625ea29d5 size: 527

可以看到推送成功