[Umbrella] unified parameter class and refactor connector creation logic

[liunaijie]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I had searched in the issues and found no similar issues.

Describe the proposal

This issue has two proposal:

1. move all parameter that user can adjust to a single class, connector use XXXSourceConfig/ XXXSourceOptions.
   Update EnvCommandOptions to describe the parameters we can adjust in config env part.
   Update ServerConfigOptions to describe the parameters we can adjust about Zeta Engine config (config in SeaTunnel.yaml).

The benefits:

• We can easily know what parameter can be adjusted, that can help us to tune the performance.
• When version release, we can easily know which parameter has been changed, which can break the user version upgrade, which is added in new version, So we can record this into version upgrade document (although we don't have it yet).

2. refactor connector, implement TableSourceFactory, TableSinkFactory interface and implement createSource, createSink, optionRule method.

In optionRule method, describe which parameter is necessary, which is optional, which are the conflicts with other parameter. (need action1 done)

Move the connector creation logic from Connector to ConnectorFactory.

The benefits:

• Know the relationship between parameters. Use the optionRule check the config before create connector instance.
• This can be used in SeaTunnelWeb.

Welcome to do this with me

Task list

type		assign to	status
ZetaConfig	EnvCommonOptions
EnvConfig	ServerConfigOptions
Connector	connector-activemq
Connector	connector-amazondynamodb
Connector	connector-amazonsqs
Connector	connector-assert
Connector	connector-cassandra
Connector	connector-cdc-mongodb
Connector	connector-cdc-mysql
Connector	connector-cdc-opengauss
Connector	connector-cdc-oracle
Connector	connector-cdc-postgres
Connector	connector-cdc-sqlserver
Connector	connector-cdc-tidb
Connector	connector-clickhouse
Connector	connector-console
Connector	connector-datahub
Connector	connector-dingtalk
Connector	connector-doris
Connector	connector-druid
Connector	connector-easysearch
Connector	connector-elasticsearch
Connector	connector-email
Connector	connector-fake
Connector	connector-file-cos
Connector	connector-file-ftp
Connector	connector-file-hadoop
Connector	connector-file-jindo-oss
Connector	connector-file-local
Connector	connector-file-obs
Connector	connector-file-oss
Connector	connector-file-s3
Connector	connector-file-sftp
Connector	connector-google-firestore
Connector	connector-google-sheets
Connector	connector-hbase
Connector	connector-hive
Connector	connector-http-feishu
Connector	connector-http-github
Connector	connector-http-gitlab
Connector	connector-http-jira
Connector	connector-http-klaviyo
Connector	connector-http-lemlist
Connector	connector-http-myhours
Connector	connector-http-notion
Connector	connector-http-onesignal
Connector	connector-http-persistiq
Connector	connector-http-wechat
Connector	connector-hudi
Connector	connector-iceberg
Connector	connector-influxdb
Connector	connector-iotdb
Connector	connector-jdbc
Connector	connector-kafka
Connector	connector-kudu
Connector	connector-maxcompute
Connector	connector-milvus
Connector	connector-mongodb
Connector	connector-neo4j
Connector	connector-openmldb
Connector	connector-paimon
Connector	connector-prometheus
Connector	connector-pulsar
Connector	connector-qdrant
Connector	connector-rabbitmq
Connector	connector-redis
Connector	connector-rocketmq
Connector	connector-s3-redshift
Connector	connector-selectdb-cloud
Connector	connector-sentry
Connector	connector-slack
Connector	connector-sls
Connector	connector-socket
Connector	connector-starrocks
Connector	connector-tablestore
Connector	connector-tdengine
Connector	connector-typesense
Connector	connector-web3j

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[Hisoka-X]

+1 from proposal1.
About proposal2, could you share more details about Move the connector creation logic from Connector to ConnectorFactory.? What it would be look like?

[liunaijie]

+1 from proposal1. About proposal2, could you share more details about Move the connector creation logic from Connector to ConnectorFactory.? What it would be look like?

About proposal2
now some connector implement the TableSourceFactory, but they not implement createSource method. so we still has 2 way to create the Connector instance.

• one is use tableSourceFactory.createSource/Sink() it return the connector instance.
• one is use ServiceLoader.load to load all implement and use pluginIdentifier to find the connector instance.

So I suggest we unified the behavior, implement the createSource/Sink method. use Factory to create the connector instance.

    private static boolean isFallback(Optional<Factory> factory) {
        if (!factory.isPresent()) {
            return true;
        }
        try {
            ((TableSourceFactory) factory.get()).createSource(null);
        } catch (Exception e) {
            if (e instanceof UnsupportedOperationException
                    && "The Factory has not been implemented and the deprecated Plugin will be used."
                            .equals(e.getMessage())) {
                return true;
            }
        }
        return false;
    }

[Hisoka-X]

Oh I got it. That's right.

[liunaijie]

example pr : #8580

[akulabs8]

Hey , I would like to contribute

[liunaijie]

Hey , I would like to contribute

Hi, thank you want to do this with me. And Just comment which connector you want to update, I will assign it to you.

[akulabs8]

Hey @liunaijie ,
I would like to start with Redis one

P.S. I looked into kafka but I couldn't find the Client sub directory there, any reasons why so?

[akulabs8]

Hey @liunaijie , I have come with a draft PR: #8594

Please take a look and let me know