Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

storage systems #186

Open
sean9528 opened this issue Oct 17, 2022 · 12 comments
Open

storage systems #186

sean9528 opened this issue Oct 17, 2022 · 12 comments
Assignees
Labels

Comments

@sean9528
Copy link

Can HSDS support Tencent Cloud object storage? How should I operate it

@LiaoSW
Copy link

LiaoSW commented Oct 17, 2022

Have the same question, there are plenty of object storage service providers in China such as AliCloud and Tencent Cloud, not sure what's the differences between them and Amazon S3, could you please provide some information about the requirement for objective storage?

@jreadey
Copy link
Member

jreadey commented Oct 17, 2022

Hi, most object storage providers (Azure being the notable exception) support the S3 API, so I suspect HSDS will work find with Tencent Cloud. Follow the instructions for setting up HSDS with AWS S3: https://github.com/HDFGroup/hsds/blob/master/docs/docker_install_aws.md, and (hopefully!) it will just work.

Let me know how it goes.

@sean9528
Copy link
Author

sean9528 commented Oct 20, 2022

Hi, thanks for your suggestion. I found a document on how to use COS Common Configuration in a third-party application that is compatible with Amazon S3(https://cloud.tencent.com/developer/article/1609495), and I ran some container in docker. I don't know the difference between hsds_dn, hsds_rangeget, hsds_sn and hsds_head. I tried to request the port of the container, but encountered the following problem. Do you know the possible reason.

~# curl http://127.0.0.1:<port=hsds_dn_1> 404(not found)

~# curl http://127.0.0.1:<port=hsds_dn_1>/about {"start_time": 1666087088, "state": "READY", "hsds_version": "0.7.0", "name": "Highly Scalable Data Service (HSDS)", "greeting": "Welcome to HSDS!", "about": "HSDS is a webservice for HDF data", "node_count": 4, "dn_urls": ["http://172.18.0.5:6101", "http://172.18.0.6:6101", "http://172.18.0.7:6101", "http://172.18.0.8:6101"], "dn_ids": ["dn-aa21d2725666-f54ac", "dn-c7094b1faecf-0046f", "dn-6e28539c041f-9c7e1", "dn-d9d30d0b2beb-aef10"], "username": "anonymous", "isadmin": false}(base)
~# curl http://127.0.0.1:<port=hsds_rangeget_1> 400: missing bucket

~# curl http://127.0.0.1:<port=hsds_sn_1> 400: invalid domain name

~# curl http://127.0.0.1:<port=hsds_head_1> 500: service unavailable

@jreadey
Copy link
Member

jreadey commented Oct 24, 2022

That looks as expected. You don't need to worry about the dn, rangeget, and head containers, the service endpoint will be: http://127.0.0.1:<port=hsds_sn_1>. By default this would be: http://127.0.0.1:5101. The /about request returns a service status JSON, and in your curl I see: "state": "READY", so that's good.

Have you tried the post install setup? https://github.com/HDFGroup/hsds/blob/master/docs/post_install.md

It's here that the service will need to talk to the storage system, so that's where I'd expect any issues with the Tencent cloud storage to come up.

@sean9528
Copy link
Author

I'm really sorry for replying to you so late. I tried the post install setup. After I run $ hsconfigure, The program keeps reporting error: "ERROR:root:got <class 'TypeError'> exception: init() got an unexpected keyword argument 'allowed_methods'"

@jreadey
Copy link
Member

jreadey commented Nov 14, 2022

No problem. Re: the error, it looks like the version of the requests package you are using is not compatible with the version h5pyd is expecting.
If you run: "pip freeze | grep requests", what do you get?

@sean9528
Copy link
Author

No problem. Re: the error, it looks like the version of the requests package you are using is not compatible with the version h5pyd is expecting. If you run: "pip freeze | grep requests", what do you get?

I get " requests @ file:///tmp/build/80754af9/requests_1592841827918/work ,requests-oauthlib==1.3.1, requests-unixsocket==0.3.0". and the verison of the requests package is 2.24.0.

@jreadey
Copy link
Member

jreadey commented Nov 14, 2022

I have requests version 2.26.0. Try: pip install requests --upgrade and see if that works.
Looks like the h5pyd setup should be updated with a min version limit for the requests package.

@sean9528
Copy link
Author

I upgraded the version of requests to 2.26.0,but It still didn't work. I want to confirm whether "state: READY" means that HSDS support Tencent Cloud object storage.

@jreadey
Copy link
Member

jreadey commented Nov 23, 2022

I setup HSDS on a Tencent VM and it worked fine. There was one test failure where HSDS had a 500 error rather than 404 when a invalid bucket parameter was used. You can verify that data is really getting written to the bucket by going to "bucket list" in the console, selecting the bucket, and then "File List". You should see top level folders of "db/" and "home/" after going through the post install steps and running the test suite.

I've created a setup doc for Tencent here: https://github.com/HDFGroup/hsds/blob/master/docs/docker_install_tencent.md. Appreciate feedback on anything that is incorrect or unclear.

@jreadey
Copy link
Member

jreadey commented Nov 23, 2022

This is the fix for the one test failure I saw: ebec5dd.

@okz
Copy link

okz commented Aug 17, 2023

upgrading requests and urllib3 worked for me

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

5 participants