-
Notifications
You must be signed in to change notification settings - Fork 313
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix(bulkload): bulkload cause many node coredump #2036
Commits on Jul 17, 2024
-
feat(build): Support to specify THIRDPARTY_ROOT by setting the enviro…
…nment variable (apache#2035) Building Pegasus thirdparty libraries costs long time, it would be meaningful to reuse a built thirdparty directory when build Pegasus source code in different directories. This patch introduces an environment variable `PEGASUS_THIRDPARTY_ROOT` to indicate the thirdparty directory, if it has been built, it can be skipt to save time and disk space.
Configuration menu - View commit details
-
Copy full SHA for 70b5e5f - Browse repository at this point
Copy the full SHA 70b5e5fView commit details -
feat(shell): add JSON format data output to some backup_policy comman…
…ds (apache#2030) Add JSON output to some backup policy commands to facilitate the writing of automation scripts. Backup policy commands including: - ls_backup_policy - query_backup_policy ls_backup_policy Output example by Tabler format ``` [p1] backup_provider_type : hdfs_service backup_interval : 86400s app_ids : {3} start_time : 03:36 status : enabled backup_history_count : 1 [p2] backup_provider_type : hdfs_service backup_interval : 86400s app_ids : {3} start_time : 20:25 status : enabled backup_history_count : 1 ``` ls_backup_policy Output example by JSON format ``` { "p1": { "backup_provider_type": "hdfs_service", "backup_interval": "86400s", "app_ids": "{3}", "start_time": "03:36", "status": "enabled", "backup_history_count": "1" }, "p2": { "backup_provider_type": "hdfs_service", "backup_interval": "86400s", "app_ids": "{3}", "start_time": "20:25", "status": "enabled", "backup_history_count": "1" } } ``` query_backup_policy Output example by Tabler format ``` [p1] backup_provider_type : hdfs_service backup_interval : 86400s app_ids : {3} start_time : 03:36 status : enabled backup_history_count : 1 [backup_info] id start_time end_time app_ids 1716781003199 2024-05-27 03:36:43 - {3} [p2] backup_provider_type : hdfs_service backup_interval : 86400s app_ids : {3} start_time : 20:25 status : enabled backup_history_count : 1 [backup_info] id start_time end_time app_ids 1716840160297 2024-05-27 20:02:40 - {3} ``` query_backup_policy Output example by JSON format ``` { "p1": { "backup_provider_type": "hdfs_service", "backup_interval": "86400s", "app_ids": "{3}", "start_time": "03:36", "status": "enabled", "backup_history_count": "1" }, "p1_backup_info": { "1716781003199": { "id": "1716781003199", "start_time": "2024-05-27 03:36:43", "end_time": "-", "app_ids": "{3}" } }, "p2": { "backup_provider_type": "hdfs_service", "backup_interval": "86400s", "app_ids": "{3}", "start_time": "20:25", "status": "enabled", "backup_history_count": "1" }, "p2_backup_info": { "1716840160297": { "id": "1716840160297", "start_time": "2024-05-27 20:02:40", "end_time": "-", "app_ids": "{3}" } } } ```
Configuration menu - View commit details
-
Copy full SHA for b7e6a8e - Browse repository at this point
Copy the full SHA b7e6a8eView commit details -
chore(pack): support to pack server binaries separately (apache#2034)
This patch adds a new flag `--separate_servers` to indicate whether to pack `pegasus_collector`,`pegasus_meta_server` and `pegasus_replica_server` binaries, otherwise a combined `pegasus_server` binary will be packed in the pegasus_server_xxx.tar. When build server in option `--separate_servers`,the corresponding option to use pack command is: ``` ./run.sh pack_server -s or ./run.sh pack_server --separate_servers ./run.sh pack_tools -s or ./run.sh pack_tools --separate_servers ```
Configuration menu - View commit details
-
Copy full SHA for d9ef442 - Browse repository at this point
Copy the full SHA d9ef442View commit details -
fix(duplication): add warning message while trying to add a duplicati…
…on that has been existing for the same table with the same remote cluster (apache#2038) apache#2039
Configuration menu - View commit details
-
Copy full SHA for 956a9bc - Browse repository at this point
Copy the full SHA 956a9bcView commit details -
fix(shell): Add PARSE_OPT_STRS marco for shell command execute flag i…
…nput mode (apache#2040) The marco `PARSE_STRS` execute strs with `param_index`,and it only execute the number of params_index of input strs. The marco `PARSE_OPT_STRS` can execute input strs with flag. The historical flag input mode should be continued.
Configuration menu - View commit details
-
Copy full SHA for 37b59d9 - Browse repository at this point
Copy the full SHA 37b59d9View commit details -
chore(go-client): add generation thrift files of go-client (apache#1917)
apache#1881 By uploading generation thrift files, the go client can be used directly by users through "go get" without the need to compile it locally.
Configuration menu - View commit details
-
Copy full SHA for 1ea072e - Browse repository at this point
Copy the full SHA 1ea072eView commit details -
fix(go-client): update config once replica server failed and forward …
…to primary meta server if it was changed (apache#1916) apache#1880 apache#1856 As for apache#1856: when go client is writing to one partition and the replica node core dump, go client will finish after timeout without updating the configuration. In this case, the go client only restart to solve the problem. In this pr, the client would update configuration of table automatically when someone replica core dump. After testing, we found that the the replica error is "context.DeadlineExceeded" (incubator-pegasus/go-client/pegasus/table_connector.go) when the replica core dump. Therefore, when client meets the error, the go client will update configuration automatically. Besides, this request will not retry. Because only in the case of timeout, the configuration will be automatically updated. If you try again before then, it will still fail. There is also the risk of infinite retries. Therefore, it is better to directly return the request error to the user and let the user try again. As for apache#1880: When the client sends an RPC message "RPC_CM_QUERY_PARTITION_CONFIG_BY_INDEX" to the meta server, if the meta server isn't primary, the response that forward to the primary meta server will return. According to the above description, assuming that the client does not have a primary meta server configured, we can connect to the primary meta server in this way. About tests: 1. Start onebox, and the primary meta server is not added to the go client configuration. 2. The go client writes data to a certain partition and then kills the replica process.
Configuration menu - View commit details
-
Copy full SHA for eb4425c - Browse repository at this point
Copy the full SHA eb4425cView commit details -
fix(FQDN): Fix the bug of crash caused by un-resolved IP address (apa…
…che#2044) apache#2007 In servers, we assume that the remote IPs may can't be reverse resolved, in this case, warning or error messages are logged instead of crashing. But in tests, we assume that all the IPs can be reverse resolved.
Configuration menu - View commit details
-
Copy full SHA for 347a827 - Browse repository at this point
Copy the full SHA 347a827View commit details -
chore: fix dead link http://open-falcon.org/ (apache#2046)
apache#2047 Use "github.com/open-falcon" instead of dead link "open-falcon.org".
Configuration menu - View commit details
-
Copy full SHA for e1c28f0 - Browse repository at this point
Copy the full SHA e1c28f0View commit details -
feat(remote_command): provide the query for the progress of decrees i…
…ncluding both local writes and duplications (apache#2045) There are many kinds of decrees while writing locally and duplicating to remote clusters, for example, the max decree in prepare list, the last decree that has ever been committed, the last decree that has been applied into rocksdb memtable, the last decree that has been flushed into rocksdb sst files, the max decree that has been confirmed by remote cluster for duplication, etc.. These decrees are very useful while we want to watch the progress of all the local writes and duplications. These decrees might also help us diagnose the problems. Therefore, we provide a tool in the way of `remote_command` to show the decrees for each replica.
Configuration menu - View commit details
-
Copy full SHA for 5dd98c3 - Browse repository at this point
Copy the full SHA 5dd98c3View commit details -
fix(duplication): reduce the delay before last mutation is duplicated…
… to the remote cluster (apache#2048) apache#2050 As is described by the issue, the problem is that we have to waits 2 ~ 3 minutes (until some empty write gets in) before the last mutation is duplicated to the remote cluster. The reason is that the last committed decree of the last mutation (i.e. `mutation.data.header.last_committed_decree`), rather than the decree of the last mutation (i.e. `mutation.data.header.decree`), is chosen as the max decree that is duplicated to the remote cluster. Instead, the max committed decree should be chosen as the max decree that is duplicated to the remote cluster. After the optimization, the delay has been reduced from 2 ~ 3 minutes to about 0.1 seconds.
Configuration menu - View commit details
-
Copy full SHA for 79add77 - Browse repository at this point
Copy the full SHA 79add77View commit details -
feat(java-client): use thrift-maven-plugin to generate thrift sources…
… instead of script (apache#1997) apache#1664 Add thrift-maven-plugin by referencing following links: - https://issues.apache.org/jira/browse/THRIFT-1536 - apache/parquet-java#600
Configuration menu - View commit details
-
Copy full SHA for 7c1a40d - Browse repository at this point
Copy the full SHA 7c1a40dView commit details -
chore(run.sh): link pegasus_shell only doesnt exist (apache#2053)
Use run.sh start pegasus shell will create a symbolic link every time. But in docker production environment. It cant create a symbolic link in the only-read filesystem of container. So when symbolic link is exist we should not create.
Configuration menu - View commit details
-
Copy full SHA for 25e3d75 - Browse repository at this point
Copy the full SHA 25e3d75View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2f3457c - Browse repository at this point
Copy the full SHA 2f3457cView commit details -
fix(scripts): downgrade_node does not work with set -e (apache#2056)
The downgrade_node scripts usually used in the scale down replica server. The function of downgrade_node scripts implementation with wild char of shell output. - The character matching is not success for every single line in shell output. So add "set -e" will exit with 1 and report failed. - Fix a shell grammar problem.
Configuration menu - View commit details
-
Copy full SHA for e267c86 - Browse repository at this point
Copy the full SHA e267c86View commit details -
feat(backup): Add --force option for 'disable_backup_policy' shell co…
…mmand (apache#2057) Before this patch, once a backup policy is added and enabled, it's impossible to disable it when a new job of the policy is starting, even if there are some reasons block the job to complete. This patch add a new flag '-f|--force' to disable the policy by force, then it's possible to stop the job after restarting the servers.
Configuration menu - View commit details
-
Copy full SHA for 6cf8879 - Browse repository at this point
Copy the full SHA 6cf8879View commit details -
fix(backup): Fix the stack overflow when read large sst file (apache#…
…2059) After refactoring to use RocksDB APIs to read files from local filesystem, it may cause stack overflow when the file to read is larger than the stack size (say 8MB). This patch changes to use heap instead of stack to store the file content.
Configuration menu - View commit details
-
Copy full SHA for b9853e4 - Browse repository at this point
Copy the full SHA b9853e4View commit details -
feat(remote_command): change some remote_command shell output to JSON…
… format (apache#2058) Some remote commands shell output are format by json. And some remote command are not. Change the output of register_int_command, register_bool_command to JSON format to improve readability by programs (e.g., Python scripts).
Configuration menu - View commit details
-
Copy full SHA for 0bc7ea0 - Browse repository at this point
Copy the full SHA 0bc7ea0View commit details -
fix(rocksdb): fix the problem that the usage of block cache is not se…
…t for the metric (apache#2060) apache#2061 The monitor of the block cache usage is server-level and created by std::call_once in a replica-level object, running periodically to update the block cache usage. However, once the replica-level object is stopped, the server-level monitor would be cancelled; as a result, the block cache usage would never be updated.
Configuration menu - View commit details
-
Copy full SHA for e05d5d7 - Browse repository at this point
Copy the full SHA e05d5d7View commit details -
fix(duplication): create checkpoint for the replica with 0 or 1 record (
apache#2054) apache#2069 To create the checkpoint of the replica with 0 or 1 record immediately: - set the min decree for checkpoint to at least 1, which means the checkpoint would inevitably be created even if the replica is empty. - for the empty replica, an empty write would be committed to increase the decree to at least 1 to ensure that the checkpoint would be created. - the max decree in rocksdb memtable (the last applied decree) is considered as the min decree that should be covered by the checkpoint, which means currently all of the data in current rocksdb should be included into the created checkpoint. The following configuration is added to control the retry interval for triggering checkpoint: ```diff [replication] + trigger_checkpoint_retry_interval_ms = 100 ```
Configuration menu - View commit details
-
Copy full SHA for 18a612a - Browse repository at this point
Copy the full SHA 18a612aView commit details -
refactor(format): bump clang-format to 14 (apache#2063)
After GitHub actions forced to run on node20 [1], and node20 depends on GLIBC_2.28, we have to run actions on newer operation systems which has higher builtin glibc. Before this patch, we are using clang-format-3.9 to format C++ code, but if we using newer OS (say Ubuntu 22.04), the clang-format-3.9 is too old and it's difficult to install such an old version. This patch bumps the clang-format to 14 as the code format tool, and update relative chores, such as updating `cpp_clang_format_linter` action job in .github/workflows/lint_and_test_cpp.yaml, removing clang-format-3.9 docker images, adding more options in `.clang-format` (almost all of the options are kept as old version and default values). The main part of this patch is the C++ code updating according to the newer clang-format, they are formated automatically. 1. https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/
Configuration menu - View commit details
-
Copy full SHA for 86ca8dd - Browse repository at this point
Copy the full SHA 86ca8ddView commit details -
chore(CI): Add envs in build_debug_on_centos7 to ensure job can be ra…
…n on centOS 7 (apache#2065) actions/runner#2906
Configuration menu - View commit details
-
Copy full SHA for 06312ca - Browse repository at this point
Copy the full SHA 06312caView commit details -
feat(FQDN): Add host_port structures for clients and bump actions/che…
…ckout from v3 to v4 (apache#2062) After All GitHub Actions run on Node20 instead of Node16 by default [1], this patch bumps the actions/checkout version from v3 to v4. When the related yaml files changed, the clients CIs are triggered, then they expose that the new introduced structure `host_port` in IDL as unknow because it is not generated by thrift automatically, we have to implement it manually. So this patch also simplily implements the `host_port` structure in python-client and nodejs-client. It should be mentioned that only the `build_debug_on_centos7` is still using `actions/checkout@v3`, see [2] for details. 1. https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/ 2. apache#2065
Configuration menu - View commit details
-
Copy full SHA for 3ed77d9 - Browse repository at this point
Copy the full SHA 3ed77d9View commit details -
chore(CI): Force CI to run on node16 (apache#2070)
The glibc version on ubuntu1804 and centos7 is lower than the node20 required, so we need to force the node version to 16 when running CI. See more details: actions/checkout#1809
Configuration menu - View commit details
-
Copy full SHA for 1d0fdf5 - Browse repository at this point
Copy the full SHA 1d0fdf5View commit details -
chore(CI): Force to use actions/checkout@v3 when OS maybe CentOS 7 or…
… Ubuntu 18.04 (apache#2072) To solve problem in GitHub actions: ``` Run actions/checkout@v4 /usr/bin/docker exec e63787d641b0351b6c65ad895ccd98db84d6796141ad087c4952bc7f68b03753 sh -c "cat /etc/*release | grep ^ID" /__e/node[20](https://github.com/apache/incubator-pegasus/actions/runs/9908766114/job/27375256228#step:3:21)/bin/node: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by /__e/node20/bin/node) ```
Configuration menu - View commit details
-
Copy full SHA for 602b80b - Browse repository at this point
Copy the full SHA 602b80bView commit details -
Configuration menu - View commit details
-
Copy full SHA for c10baa9 - Browse repository at this point
Copy the full SHA c10baa9View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8e09a45 - Browse repository at this point
Copy the full SHA 8e09a45View commit details -
fix(Dockerfile): Fix the yum install failure caused by its EOL (apach…
…e#2075) Resolve apache#2071.
Configuration menu - View commit details
-
Copy full SHA for 5b70f5b - Browse repository at this point
Copy the full SHA 5b70f5bView commit details -
Configuration menu - View commit details
-
Copy full SHA for a6f777f - Browse repository at this point
Copy the full SHA a6f777fView commit details