Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use meituan hdfs to read or write train data and model #1088

Merged
merged 4 commits into from
Mar 20, 2024
Merged

Use meituan hdfs to read or write train data and model #1088

merged 4 commits into from
Mar 20, 2024

Conversation

henshy
Copy link

@henshy henshy commented Mar 15, 2024

  1. Add Meituan Hadoop File and Requirements:
    Add the fedlearner-sgx-dev.dockerfile and requirements.txt for Meituan Hadoop integration.
  2. Add Meituan Hadoop Environment in Gramine Template:
    Incorporate Meituan Hadoop environment settings into the python.manifest.template for Gramine.
  3. Perform Kerberos Authentication in hdfs_common.sh:
    Execute Kerberos authentication within the hdfs_common.sh script.
  4. Implement Meituan HDFS File Read and Write in the Training Step:
    Carry out file read and write operations for Meituan HDFS within the training step, specifically in trainer_master.py and trainer_worker.py.

@zeuson0
Copy link

zeuson0 commented Mar 20, 2024

LGTM

@gejielun gejielun merged commit c3ea95d into bytedance:fix_dev_sgx Mar 20, 2024
2 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants