storage systems

[sean9528]

Can HSDS support Tencent Cloud object storage? How should I operate it

[LiaoSW]

Have the same question, there are plenty of object storage service providers in China such as AliCloud and Tencent Cloud, not sure what's the differences between them and Amazon S3, could you please provide some information about the requirement for objective storage?

[jreadey]

Hi, most object storage providers (Azure being the notable exception) support the S3 API, so I suspect HSDS will work find with Tencent Cloud. Follow the instructions for setting up HSDS with AWS S3: https://github.com/HDFGroup/hsds/blob/master/docs/docker_install_aws.md, and (hopefully!) it will just work.

Let me know how it goes.

[sean9528]

Hi, thanks for your suggestion. I found a document on how to use COS Common Configuration in a third-party application that is compatible with Amazon S3(https://cloud.tencent.com/developer/article/1609495), and I ran some container in docker. I don't know the difference between hsds_dn, hsds_rangeget, hsds_sn and hsds_head. I tried to request the port of the container, but encountered the following problem. Do you know the possible reason.

~# curl http://127.0.0.1:<port=hsds_dn_1> 404(not found)

~# curl http://127.0.0.1:<port=hsds_dn_1>/about {"start_time": 1666087088, "state": "READY", "hsds_version": "0.7.0", "name": "Highly Scalable Data Service (HSDS)", "greeting": "Welcome to HSDS!", "about": "HSDS is a webservice for HDF data", "node_count": 4, "dn_urls": ["http://172.18.0.5:6101", "http://172.18.0.6:6101", "http://172.18.0.7:6101", "http://172.18.0.8:6101"], "dn_ids": ["dn-aa21d2725666-f54ac", "dn-c7094b1faecf-0046f", "dn-6e28539c041f-9c7e1", "dn-d9d30d0b2beb-aef10"], "username": "anonymous", "isadmin": false}(base)
~# curl http://127.0.0.1:<port=hsds_rangeget_1> 400: missing bucket

~# curl http://127.0.0.1:<port=hsds_sn_1> 400: invalid domain name

~# curl http://127.0.0.1:<port=hsds_head_1> 500: service unavailable

[jreadey]

That looks as expected. You don't need to worry about the dn, rangeget, and head containers, the service endpoint will be: http://127.0.0.1:<port=hsds_sn_1>. By default this would be: http://127.0.0.1:5101. The /about request returns a service status JSON, and in your curl I see: "state": "READY", so that's good.

Have you tried the post install setup? https://github.com/HDFGroup/hsds/blob/master/docs/post_install.md

It's here that the service will need to talk to the storage system, so that's where I'd expect any issues with the Tencent cloud storage to come up.

[sean9528]

I'm really sorry for replying to you so late. I tried the post install setup. After I run $ hsconfigure, The program keeps reporting error: "ERROR:root:got <class 'TypeError'> exception: init() got an unexpected keyword argument 'allowed_methods'"

[jreadey]

No problem. Re: the error, it looks like the version of the requests package you are using is not compatible with the version h5pyd is expecting.
If you run: "pip freeze | grep requests", what do you get?

[sean9528]

No problem. Re: the error, it looks like the version of the requests package you are using is not compatible with the version h5pyd is expecting. If you run: "pip freeze | grep requests", what do you get?

I get " requests @ file:///tmp/build/80754af9/requests_1592841827918/work ,requests-oauthlib==1.3.1, requests-unixsocket==0.3.0". and the verison of the requests package is 2.24.0.

[jreadey]

I have requests version 2.26.0. Try: pip install requests --upgrade and see if that works.
Looks like the h5pyd setup should be updated with a min version limit for the requests package.

[sean9528]

I upgraded the version of requests to 2.26.0，but It still didn't work. I want to confirm whether "state: READY" means that HSDS support Tencent Cloud object storage.

[jreadey]

I setup HSDS on a Tencent VM and it worked fine. There was one test failure where HSDS had a 500 error rather than 404 when a invalid bucket parameter was used. You can verify that data is really getting written to the bucket by going to "bucket list" in the console, selecting the bucket, and then "File List". You should see top level folders of "db/" and "home/" after going through the post install steps and running the test suite.

I've created a setup doc for Tencent here: https://github.com/HDFGroup/hsds/blob/master/docs/docker_install_tencent.md. Appreciate feedback on anything that is incorrect or unclear.

[jreadey]

This is the fix for the one test failure I saw: ebec5dd.

[okz]

upgrading requests and urllib3 worked for me