Develop (#168)

* Explain login/paas settings different between development/production (#79) * login settings SITE_URL diff in develoment/production * update doc: LOGIN_DOMAIN in paas development * fix develop login url 404 (#81) * bugfix: 我的应用-应用部署-下架 title错误 (#94) (#117) * Devops support (#167) * add python3 framework for resource download * add tag as extra field for api/v2/app_info * add bk_devops_nav logo * add is_display to app, for desktop * update py3 framework to 1.0.2 * add api for bk-sops light app * refactor : api for bk-sops light app * refactor 1: api for bk-sops light app * refactor 2: api for bk-sops light app * delete syncdb command for django version >= 1.9 * set requests default timeout 5 second and add some logs when exception * set HTTP_TIMEOUT variable in settings * add newline in settings_default.py * support py3 for esb sdk, update translation for some apis, support DELETE/PUT method for request some apis, support get bk apidoc from md files * up cc some apidocs * update VERSION to 3.2.3 * upgrade gunicorn to 19.9.0 * add guide for python3 framework * add app env var admin * update py3 framework to 1.0.3 * fix js pagination error * fix js pagination error * up cc confapis * Devops support fixbug20190306 (#14) * add error message where user info (phone, chname) check * fixbug: light app logo image name ramdom * refactor * up cc/sops apis * up cc/sops confapis * add try-except for bk_backend.authenticate (#15) * Bugfix 0311 (#16) * add try-except for bk_backend.authenticate * update bk_devops_nav logo * app.yml support app env init while setup (#17) * app.yml support app env init while setup * fix app env key/value validation for app.yml * env key/value type cast to str before validate * unify the app_env_var value length to 1000 * support old login account manage page url (#18) * remove on_migrate * add doc to release.md: update python framework to support https * update release.md * third server password should not be chinese * fix link too long text-overflow * fix some issues * remove header for some files * bugfix: bktoken decrypt wrong * batch issues fix * change version to 3.2.5 * modify default user logo (#51) * fix some icons display error (#52) * fix fta imap (#56) * Update release.md * fix the too complex of account _is_bk_token_valid (#59) * Explain login/paas settings different between development/production (#79) (#80) * login settings SITE_URL diff in develoment/production * update doc: LOGIN_DOMAIN in paas development * 修复本地开发登录URL 404问题 (#82) * fix develop login url 404 (#81) * 更新 sops 组件，更新cmsi 发送语音通知组件 (#84) * up sops confapis, cmsi send_voice_msg * add component conf for cmsi send_voice_msg * 更新release说明 (#85) * up sops confapis, cmsi send_voice_msg * add component conf for cmsi send_voice_msg * up release doc * fix login decrypt token code bug (#88) * paas_agent support python3 (#89) * fix sync blocking bug when build script times out (#64) * Upgrade python framework to 2.0.0 (#90) * upgrade python framework to 2.0.0 * change new framework icon * fix fta imap (#103) * Update weixin doc and fix ligth app logo (#105) * fix light app lose bug * update weixin notify doc * update weixin notify doc * fixbug: import user do not response bug * fix cc delete_object_attribute apidoc (#106) * update guide doc of python framework (#108) * bugfix: 滚动按钮提示内容错误。issue #97 (#113) * fixbug:Development Language Change #96 (#116) * add app-env title (#124) * add resource migration for python doc url and delete united dev-env (#129) * Add docker (#131) * add docker docs for quick experience * update docker support docs * update docker support docs * upgrade framework to 2.1.4.11 (#134) * update link or replace ce with open source version (#135) * 93:修改BUG：当版本历史为空时, 前端出现两条横线 (#123) * add static version for css/js (#136) * Fix user avatar (#142) * fix user avatar * add import_common_template for sops * Bugfix 0717 (#143) * fix #128; remove django specific version in document * fix #140; resource link to official website * use HTTP_SCHEMA for job/cc link int (#145) * update cc apis, add cmsi send_msg/get_msg_type (#146) * saas info show settings style (#150) * fix #147; add/delete user refresh list without query term (#149) * disable bk_language while the i18n is not support in ce (#155) * fix resource download icon (#156) * fix resource download icon * fix pep8 fail * fix login api ApiErrorCodeEnumV2 missing USER_NOT_EXISTS2 (#158) * support logo file in pkg root dir (#164) * Update docs (#166) * update readme.md/readme_en.md Co-authored-by: nannan00 <[email protected]> Co-authored-by: jamesge <[email protected]> Co-authored-by: alex-smile <[email protected]> Co-authored-by: wenjaron <[email protected]> Co-authored-by: kindmaketheworldpeace <[email protected]> Co-authored-by: canway-obama <[email protected]> Co-authored-by: molly0755 <[email protected]> Co-authored-by: 谢先斌 <[email protected]>
TencentBlueKing · Jan 6, 2020 · 7a6fe1e · 7a6fe1e
1 parent 709511b
commit 7a6fe1e
Show file tree

Hide file tree

Showing 261 changed files with 6,810 additions and 2,560 deletions.
diff --git a/VERSION b/VERSION
@@ -1,2 +1,2 @@
-PaaS      3.2.2
+PaaS      3.2.5
 PaaSAgent 3.1.0
diff --git a/docs/install/ce_paas_agent_install.md b/docs/install/ce_paas_agent_install.md
@@ -2,7 +2,7 @@
 
 ## 系统要求
 
-- Python版本: python2.7
+- Python版本: python2.7/python3.6.7
 - Go
 - Nginx
 - Linux环境
@@ -12,21 +12,33 @@ PaaSAgent需要部署在满足系统要求的app服务器上。建议最少准
 
 #### 1. 基础环境初始化
 
-python 2.7环境初始化
+python 环境初始化
 
 ```
 # 确认基础python环境是python2.7
 $ python --version
 Python 2.7.15
 
+# 执行以下命令确定python的安装路径
+$ which python
+
 # 安装 virtualenv 及 预装包
 $ pip install virtualenv virtualenvwrapper
+$ pip install supervisor==3.3.3
 $ pip install -r paas-ce/paasagent/etc/build/packages/requirements.txt
 
-# 查看python路径
-$ which python
+# 如果使用python3，建议将python3安装到/opt/py36目录下
+# 从源码包编译安装python3.6.7时，可能需要手动建立python和pip的软链
+$ cd /opt/py36/bin
+$ ln -s python3.6 python
+$ ln -s pip3 pip
+
+# 确认python3的版本
+$ /opt/py36/bin/python --version
+Python 3.6.7
 ```
 
+
 构建目录及新建用户
 
 ```
@@ -62,7 +74,7 @@ $ sed -i "s#TPL_AGENT_ROOT#${AGENT_ROOT}#g" paas_agent_config.yaml nginx/paasage
 $ chmod +x build/virtualenv/build  build/virtualenv/saas/buildsaas
 ```
 
-注意: `build` 和 `buildsaas` 中的 `VIRTUALENVWRAPPER_PYTHON` 变量需要设置正确的 python 路径，默认`/usr/bin/python`, 如果上一步`which python`的返回结果不是`/usr/bin/python`, 需要修改为对应正确的路径
+注意: `build` 和 `buildsaas` 中的 `VIRTUALENVWRAPPER_PYTHON` 变量需要设置正确的 python 路径，默认`/usr/bin/python`, 如果上一步`which python`的返回结果不是`/usr/bin/python`, 需要修改为对应正确的路径；如果用到了python3，`PYTHON3_PATH`也需要设置成正确的路径，默认`/opt/py36`
 
 #### 3. 蓝鲸智云开发者中心注册服务器
 

diff --git a/paas-ce/paas/VERSION b/paas-ce/paas/VERSION
@@ -1 +1 @@
-3.2.2
+3.2.5
diff --git a/paas-ce/paas/appengine/requirements.txt b/paas-ce/paas/appengine/requirements.txt
@@ -1,6 +1,6 @@
 django==1.8.11
 pymysql==0.6.7
 requests==2.21.0
-gunicorn==19.6.0
+gunicorn==19.9.0
 uWSGI==2.0.13.1
 gevent==1.1.2
diff --git a/paas-ce/paas/esb/apps/sdk_management/sdk_tmpl/blueking/component/base.py b/paas-ce/paas/esb/apps/sdk_management/sdk_tmpl/blueking/component/base.py
@@ -40,7 +40,7 @@ def __call__(self, *args, **kwargs):
         self.url = self.get_url_with_api_ver()
         try:
             return self._call(*args, **kwargs)
-        except ComponentAPIException, e:
+        except ComponentAPIException as e:
             # Combine log message
             log_message = [e.error_message, ]
             log_message.append('url=%(url)s' % {'url': e.api_obj.url})
@@ -75,7 +75,7 @@ def _call(self, *args, **kwargs):
         # Request remote server
         try:
             resp = self.client.request(self.method, self.url, params=params, data=data)
-        except Exception, e:
+        except Exception as e:
             logger.exception('Error occurred when requesting method=%s url=%s',
                              self.method, self.url)
             raise ComponentAPIException(self, u'Request component error, Exception: %s' % str(e))

diff --git a/paas-ce/paas/esb/apps/sdk_management/sdk_tmpl/blueking/component/client.py b/paas-ce/paas/esb/apps/sdk_management/sdk_tmpl/blueking/component/client.py
@@ -13,8 +13,8 @@
 import time
 import random
 import logging
-import urlparse
 
+from .compat import urlparse
 from . import conf
 from . import collections
 from .utils import get_signature
@@ -133,7 +133,7 @@ def request(self, method, url, params=None, data=None, **kwargs):
         if method == 'POST':
             params = {}
 
-        url_path = urlparse.urlparse(url).path
+        url_path = urlparse(url).path
         # signature always in GET params
         params.update({
             'bk_timestamp': int(time.time()),

diff --git a/paas-ce/paas/esb/apps/sdk_management/sdk_tmpl/blueking/component/compat.py b/paas-ce/paas/esb/apps/sdk_management/sdk_tmpl/blueking/component/compat.py
@@ -0,0 +1,26 @@
+# -*- coding: utf-8 -*-
+"""
+Tencent is pleased to support the open source community by making 蓝鲸智云PaaS平台社区版 (BlueKing PaaS Community Edition) available.
+Copyright (C) 2017-2018 THL A29 Limited, a Tencent company. All rights reserved.
+Licensed under the MIT License (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
+http://opensource.org/licenses/MIT
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
+""" # noqa
+import sys
+
+_ver = sys.version_info
+
+is_py2 = (_ver[0] == 2)
+
+is_py3 = (_ver[0] == 3)
+
+
+if is_py2:
+    from urlparse import urlparse
+
+    str = unicode
+
+elif is_py3:
+    from urllib.parse import urlparse  # noqa
+
+    str = str
diff --git a/paas-ce/paas/esb/apps/sdk_management/sdk_tmpl/blueking/component/utils.py b/paas-ce/paas/esb/apps/sdk_management/sdk_tmpl/blueking/component/utils.py
@@ -11,6 +11,8 @@
 import hmac
 import hashlib
 
+from .compat import str
+
 
 def get_signature(method, path, app_secret, params=None, data=None):
     """generate signature
@@ -23,8 +25,10 @@ def get_signature(method, path, app_secret, params=None, data=None):
         kwargs['data'] = data
     kwargs = '&'.join([
         '%s=%s' % (k, v)
-        for k, v in sorted(kwargs.iteritems(), key=lambda x: x[0])
+        for k, v in sorted(kwargs.items(), key=lambda x: x[0])
     ])
     orignal = '%s%s?%s' % (method, path, kwargs)
-    signature = base64.b64encode(hmac.new(str(app_secret), orignal, hashlib.sha1).digest())
-    return signature
+    app_secret = app_secret.encode('utf-8') if isinstance(app_secret, str) else app_secret
+    orignal = orignal.encode('utf-8') if isinstance(orignal, str) else orignal
+    signature = base64.b64encode(hmac.new(app_secret, orignal, hashlib.sha1).digest())
+    return signature if isinstance(signature, str) else signature.decode('utf-8')
diff --git a/paas-ce/paas/esb/common/base_loggers.py b/paas-ce/paas/esb/common/base_loggers.py
@@ -7,6 +7,7 @@
 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
 """ # noqa
 import json
+import copy
 
 from common.log import logger_api, logger
 from common.base_utils import datetime_format
@@ -23,6 +24,11 @@ def write(self, request, response):
             kwargs = request.g.kwargs_copy
         else:
             kwargs = request.g.kwargs
+
+        if request.g.system_name == 'CMSI' and request.g.component_alias_name == 'send_mail':
+            kwargs = copy.copy(kwargs)
+            kwargs.pop('attachments', None)
+
         msecs_cost = (request.g.ts_request_end - request.g.ts_request_start) * 1000
         if isinstance(response, dict):
             message = response and response.get('message', '')

diff --git a/paas-ce/paas/esb/common/base_utils.py b/paas-ce/paas/esb/common/base_utils.py
@@ -215,9 +215,9 @@ def get_client_real_ip(request):
 
 
 def get_request_params(request):
-    if request.method not in ('GET', 'POST'):
-        raise error_codes.COMMON_ERROR.format_prompt(
-            'Request method error, please apply GET or POST request.', replace=True)
+    # if request.method not in ('GET', 'POST'):
+    #     raise error_codes.COMMON_ERROR.format_prompt(
+    #         'Request method error, please apply GET or POST request.', replace=True)
     # "GET"方法
     if request.method == 'GET':
         request_params = dict(request.GET.items())
@@ -262,3 +262,26 @@ def load_yaml(path):
 def read_file(path):
     with open(path, 'r') as fp:
         return fp.read()
+
+
+def html_escape(html, is_json=False):
+    """
+    Replace special characters "&", "<" and ">" to HTML-safe sequences.
+    If the optional flag quote is true, the quotation mark character (")
+    is also translated.
+    rewrite the cgi method
+    @param html: html代码
+    @param is_json: 是否为json串（True/False） ，默认为False
+    """
+    # &转换
+    if not is_json:
+        html = html.replace('&', '&amp;')  # Must be done first!
+    # <>转换
+    html = html.replace("<", "&lt;")
+    html = html.replace(">", "&gt;")
+    # 单双引号转换
+    if not is_json:
+        # html = html.replace(' ', "&nbsp;")
+        html = html.replace('"', "&quot;")
+        html = html.replace("'", "&#39;")
+    return html
diff --git a/paas-ce/paas/esb/common/constants.py b/paas-ce/paas/esb/common/constants.py
@@ -26,12 +26,17 @@
 API_TYPE_OP = 'operate'
 API_TYPE_Q = 'query'
 
+HTTP_METHOD = FancyDict((
+    ('GET', 'GET'),
+    ('POST', 'POST'),
+))
+
 
 BK_SYSTEMS = {
     'BK_LOGIN': {
         'name': 'BK_LOGIN',
-        'label': _(u'蓝鲸登录平台'),
-        'remark': u'蓝鲸登录平台，管理用户登录验证，及用户信息',
+        'label': _(u'蓝鲸统一登录'),
+        'remark': u'蓝鲸统一登录，管理用户登录验证，及用户信息',
     },
     'BK_PAAS': {
         'name': 'BK_PAAS',

diff --git a/paas-ce/paas/esb/common/errors.py b/paas-ce/paas/esb/common/errors.py
@@ -144,7 +144,7 @@ class ErrorCodes(object):
     """
 
     error_codes = (
-        # 13063xx, user error
+        # 13064xx, user error
         ErrorCode('OPERATOR_REQUIRED', 1306401, 'You must specify the current operator'),
         ErrorCode('USER_PERMISSION_DENIED', 1306402, 'User permission is insufficient'),
         ErrorCode('APP_PERMISSION_DENIED', 1306403, 'APP permission is insufficient'),

diff --git a/paas-ce/paas/esb/components/bk/apis/bk_login/get_user.py b/paas-ce/paas/esb/components/bk/apis/bk_login/get_user.py
@@ -71,6 +71,14 @@ class GetUser(Component):
     class Form(BaseComponentForm):
         bk_token = forms.CharField(label='login token', required=False)
         username = forms.CharField(label='username', required=False)
+        bk_username = forms.CharField(label='bk_username', required=False)
+
+        def clean(self):
+            data = self.cleaned_data
+            return {
+                'bk_token': data['bk_token'],
+                'username': data['username'] or data['bk_username'],
+            }
 
     def handle(self):
         self.response.payload = self.outgoing.http_client.get(

diff --git a/paas-ce/paas/esb/components/bk/apis/cc/toolkit/tools.py b/paas-ce/paas/esb/components/bk/apis/cc/toolkit/tools.py
@@ -45,8 +45,8 @@ def post_request(self, host, path, data=None, headers_json=True, **kwargs):
         except Exception:
             logger.exception('response: %s', response)
             raise CommonAPIError(
-                'An exception occurred while requesting CC interface, '
-                'please contact the component developer to handle it.')
+                'An error occurred while requesting CC interface, '
+                'the response content does not contain code field.')
 
         if code != '0':
             return {

diff --git a/paas-ce/paas/esb/components/bk/apis/fta/toolkit/imap.py b/paas-ce/paas/esb/components/bk/apis/fta/toolkit/imap.py
@@ -174,20 +174,22 @@ def search(self, charset=None, unseen=None, before=None, since=None, size_limit=
     def filter(self, mails, sent_from=None, sent_to=None, subject=None, before=None, since=None,
                fetch_header_already=False):
         result_list = []
-        mails = mails if fetch_header_already else self.iter_fetch_chunks(mails=mails, criteria=self.CRITERIA_HEADER,)
+        mails = mails
+        if not fetch_header_already:
+            for index in range(0, len(mails)):
+                self.fetch_header(mails[index * 100:(index + 1) * 100])
         sent_from = sent_from.split(",") if sent_from else ()
         sent_to = sent_to.split(",") if sent_to else ()
         subject = [subject] if subject else []
         for mail in mails:
-            if ((before and before <= mail.time) or (since and since > mail.time)):
+            if mail.time is None or (before and before <= mail.time) or (since and since > mail.time):
                 continue
             if sent_from and not contain_one(mail.sender, sent_from):
                 continue
             if sent_to and not contain_one(mail.receiver, sent_to):
                 continue
             if subject and not contain_one(mail.subject, subject):
                 continue
-
             result_list.append(mail)
         return result_list
 
@@ -197,7 +199,7 @@ def fetch_header(self, mails):
 
     def fetch(self, mails, criteria=None):
         mail_mappings = {str(mail.uid): mail for mail in mails}
-        status, result = self.imap_client.fetch(",".join(mail_mappings.keys()), criteria or self.CRITERIA,)
+        status, result = self.imap_client.fetch(",".join(mail_mappings.keys()), criteria or self.CRITERIA)
         if status != "OK":
             raise Exception("fetch mail[%s] failed", mail_mappings.keys())
 

diff --git a/paas-ce/paas/esb/components/bk/apisv2/cc/bind_role_privilege.py b/paas-ce/paas/esb/components/bk/apisv2/cc/bind_role_privilege.py
@@ -33,7 +33,7 @@ class BindRolePrivilege(Component):
 
     | {{ _("字段") }}      |  {{ _("类型") }}      | {{ _("必选") }}   |  {{ _("描述") }}      |
     |-----------|------------|--------|------------|
-    | bk_supplier_account |  string    | {{ _("是") }}     | {{ _("开发商帐号") }} |
+    | bk_supplier_account |  string    | {{ _("是") }}     | {{ _("开发商账号") }} |
     | bk_obj_id           |  string    | {{ _("是") }}     | {{ _("模型ID") }} |
     | bk_property_id      |  string    | {{ _("是") }}     | {{ _("模型对应用户角色属性ID") }}   |
     | data                |  list      | {{ _("否") }}     | {{ _("角色权限数据，输入为空数组则不绑定任何权限") }}   |

diff --git a/paas-ce/paas/esb/components/bk/apisv2/cc/create_business.py b/paas-ce/paas/esb/components/bk/apisv2/cc/create_business.py
@@ -47,7 +47,7 @@ class CreateBusiness(Component):
     | bk_biz_tester     |  string  | {{ _("是") }}     | {{ _("测试人员") }} |
     | time_zone         |  string  | {{ _("是") }}     | {{ _("时区") }} |
 
-    **注意：此处的输入参数仅对必填以及系统内置的参数做了说明，其余需要填写的参数取决于用户自己定义的属性字段。**
+    **{{ _("注意：此处的输入参数仅对必填以及系统内置的参数做了说明，其余需要填写的参数取决于用户自己定义的属性字段") }}**
 
     ### {{ _("请求参数示例") }}
 

diff --git a/paas-ce/paas/esb/components/bk/apisv2/cc/create_set.py b/paas-ce/paas/esb/components/bk/apisv2/cc/create_set.py
@@ -41,11 +41,11 @@ class CreateSet(Component):
 
     | {{ _("字段") }}      |  {{ _("类型") }}      | {{ _("必选") }}   |  {{ _("描述") }}      |
     |-----------|------------|--------|------------|
-    | bk_parent_id        |  int     | {{ _("是") }}     | {{ _("父实例的ID") }} |
+    | bk_parent_id        |  int     | {{ _("是") }}     | {{ _("父节点的ID") }} |
     | bk_set_name         |  string  | {{ _("是") }}     | {{ _("集群名字") }} |
     | default             |  int     | {{ _("否") }}     | {{ _("0-普通集群，1-内置模块集合，默认为0") }} |
 
-    **注意：其它需要的字段由模型定义**
+    **{{ _("注意：其它需要的字段由模型定义") }}**
 
     ### {{ _("请求参数示例") }}
 

diff --git a/paas-ce/paas/esb/components/bk/apisv2/cc/get_host_base_info.py b/paas-ce/paas/esb/components/bk/apisv2/cc/get_host_base_info.py
@@ -57,12 +57,12 @@ class GetHostBaseInfo(Component):
         "data": [
             {
                 "bk_property_id": "bk_host_name",
-                "bk_property_name": "主机名",
+                "bk_property_name": "host name",
                 "bk_property_value": "centos7"
             },
             {
                 "bk_property_id": "bk_host_id",
-                "bk_property_name": "主机ID",
+                "bk_property_name": "host ID",
                 "bk_property_value": "10000"
             }
         ]

diff --git a/paas-ce/paas/esb/components/bk/apisv2/cc/search_biz_inst_topo.py b/paas-ce/paas/esb/components/bk/apisv2/cc/search_biz_inst_topo.py
@@ -31,7 +31,7 @@ class SearchBizInstTopo(Component):
 
     | {{ _("字段") }}      |  {{ _("类型") }}      | {{ _("必选") }}   |  {{ _("描述") }}      |
     |-----------|------------|--------|------------|
-    | bk_supplier_account |  string  | {{ _("否") }}     | {{ _("开发商帐号") }} |
+    | bk_supplier_account |  string  | {{ _("否") }}     | {{ _("开发商账号") }} |
     | bk_biz_id           |  int     | {{ _("是") }}     | {{ _("业务id") }} |
     | level               |  int     | {{ _("否") }}     | {{ _("拓扑的层级索引，索引取值从0开始，默认值为2，当设置为 -1 的时候会读取完整的业务实例拓扑") }} |
 
@@ -57,21 +57,21 @@ class SearchBizInstTopo(Component):
         "data": [
             {
                 "bk_inst_id": 2,
-                "bk_inst_name": "蓝鲸",
+                "bk_inst_name": "blueking",
                 "bk_obj_id": "biz",
-                "bk_obj_name": "业务",
+                "bk_obj_name": "business",
                 "child": [
                     {
                         "bk_inst_id": 3,
-                        "bk_inst_name": "作业平台",
+                        "bk_inst_name": "job",
                         "bk_obj_id": "set",
-                        "bk_obj_name": "集群",
+                        "bk_obj_name": "set",
                         "child": [
                             {
                                 "bk_inst_id": 5,
                                 "bk_inst_name": "job",
                                 "bk_obj_id": "module",
-                                "bk_obj_name": "模块",
+                                "bk_obj_name": "module",
                                 "child": []
                             }
                         ]