Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add to support single table with flexible support for multiple partition keys. #181

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

ezrealpan
Copy link

@ezrealpan ezrealpan commented Oct 30, 2024

…e partition keys.

  • Do only one thing
  • Non breaking API changes
  • Tested

What did this pull request do?

通常情况下,一张业务逻辑表有多个分表的维度。例如,订单表可以按订单ID分表;同样,按用户ID分表,在根据用户ID查询订单时也能提速,就像我们常查看自己的订单那样。这个组件原来也是支持这种功能,只不过需要初始化不同的数据库实例才能实现。为了更好的支持这个功能:赋能单表,灵活支持多分表策略,我基于go的上下文的特点实现了这个功能。

User Case Description

// 配置 Gorm Sharding 中间件,注册分表策略配置
db.Use(sharding.RegisterWithKeys(map[string]sharding.Config{
	"orders_order_year": {
		ShardingKey:           "order_year",
		ShardingAlgorithm:     customShardingAlgorithmWithOrderYear, // 使用自定义的分片算法
		PrimaryKeyGenerator:   sharding.PKCustom,                    // 使用自定义的主键生成函数
		PrimaryKeyGeneratorFn: customePrimaryKeyGeneratorFn,         // 自定义主键生成函数
	},
	"orders_user_id": {
		ShardingKey:         "user_id",
		NumberOfShards:      4,
		ShardingAlgorithm:   customShardingAlgorithmWithUserId, // 使用自定义的分片算法
		PrimaryKeyGenerator: sharding.PKSnowflake,              // 使用 Snowflake 算法生成主键
	},
	"orders_order_id": {
		ShardingKey:           "order_id",
		ShardingAlgorithm:     customShardingAlgorithmWithOrderId, // 使用自定义的分片算法
		PrimaryKeyGenerator:   sharding.PKCustom,
		PrimaryKeyGeneratorFn: customePrimaryKeyGeneratorFn,
	},
}))

if shardingKey, ok := ctx.Value("sharding_key").(string); ok {
key = fmt.Sprintf("%s_%v", table, shardingKey)
}
if r, ok := pool.sharding.configs[key]; ok {
Copy link
Collaborator

@huacnlee huacnlee Oct 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

看起来这个思路是可行,如果能完善,我觉得我们可以引入这个改进。

sharding_key%s_%v 这种在实际应用的时候需要掌握这套规则,不是一个完整的 API 设计。

例如 sharding_key 应该是一个 const,而 key 的我没想到好的解决方法,目前这样,以后其他人使用会存在各种各样的问题,他们很容易遇到无法命中配置。

如果是 sharding_key 已经传了,但 sharding.configs 又找不到对应配置的情况下,应该直接报错。

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

有道理,这两点我完善一下~

test/sharding_test.go Outdated Show resolved Hide resolved
@huacnlee huacnlee changed the title enables sharding for a single table with flexible support for multipl… Add to support single table with flexible support for multiple partition keys. Oct 30, 2024
sharding.go Outdated
)

var (
ShardingIgnoreStoreKey = "sharding_ignore"
)

// ContextKeyForShardingKey is the context key for sharding key.
const ContextKeyForShardingKey = "sharding_key"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

改成 ShardingContextKey 与上面的对应,另外把上面那个也改成 const 合并在一起

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

好的~

sharding.go Outdated
@@ -46,6 +52,10 @@ type Config struct {
// For example, for a product order table, you may want to split the rows by `user_id`.
ShardingKey string

// logical table name.Suport multiple table names with same sharding key.
// For example, for user and order table, you may want to shard by `user_id`.
TableNames []string
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个是必须要的吗?我看上一次都没有这个,这样会增加 config 理解的复杂度,容易产生使用上的歧义。

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

我准备要合并这个功能,但你得确保尽量少的改动 API,非必要的时候不要引入新的 API 设计。

TableNames 这个我感觉无法接受,这会让整个 Sharding 对于 table 的组织配置变得复杂。最初的设计进需要考虑 sharding_key,而 table 本身是由 Gorm 的 table_name 映射机制来的(这个实际上最初来自 ActiveRecord 的 Model 和 Table 的约定推导方式)。

因此这里不应该存在特别配置 TableNames 的场景,TableNames 始终应该从 Gorm 里面来,如果要自定义,也应当用 Gorm 的 TableName 来实现,而不是在 Sharding 里面处理。

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image
image
那我把方法暴露出去,注册的逻辑还是保持传map的形式您看这样是否ok

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants