Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docker_web3-indexer_1 Exit 1 (Error: error returned from database: sorry, too many clients already) #248

Open
gpBlockchain opened this issue May 12, 2022 · 5 comments
Open

docker_web3-indexer_1 Exit 1 (Error: error returned from database: sorry, too many clients already) #248

gpBlockchain opened this issue May 12, 2022 · 5 comments
Assignees
@classicalliu
Labels
bug Something isn't working

Comments

@gpBlockchain
Copy link

env

git log --format="%H" -n 1

guopenglin@192 godwoken-kicker % git log --format="%H" -n 1
14bfd45f0a1337ad2023ccedaf781781eefd1806

./kicker ps

guopenglin@192 godwoken-kicker % ./kicker ps
:> docker-compose -f docker/docker-compose.yml ps

                 Name                                Command                  State                           Ports                     
----------------------------------------------------------------------------------------------------------------------------------------
docker_check-ckb-started-successfully_1   /entrypoint.sh http://ckb: ...   Exit 0                                                       
docker_ckb-indexer_1                      /bin/ckb-indexer -c http:/ ...   Up             0.0.0.0:8116->8116/tcp                        
docker_ckb-miner_1                        /bin/docker-entrypoint.sh  ...   Up             8114/tcp, 8115/tcp                            
docker_ckb_1                              /bin/docker-entrypoint.sh  ...   Up             0.0.0.0:8114->8114/tcp, 0.0.0.0:8115->8115/tcp
docker_godwoken_1                         /var/lib/layer2/entrypoint.sh    Up (healthy)   0.0.0.0:8119->8119/tcp, 0.0.0.0:8120->8120/tcp
docker_postgres_1                         docker-entrypoint.sh postgres    Up             0.0.0.0:5432->5432/tcp                        
docker_redis_1                            docker-entrypoint.sh redis ...   Up             0.0.0.0:6379->6379/tcp                        
docker_web3-indexer_1                     /var/lib/web3-indexer/entr ...   Exit 1                                                       
docker_web3_1                             /var/lib/web3/entrypoint.sh      Up (healthy)   3000/tcp, 0.0.0.0:8024->8024/tcp

docker_web3-indexer_1 logs

[2022-05-12T03:42:24Z INFO  gw_web3_indexer::runner] Sync block 3787
[2022-05-12T03:42:57Z INFO  gw_web3_indexer::runner] Sync block 3788
[2022-05-12T03:43:33Z INFO  gw_web3_indexer::runner] Sync block 3789
[2022-05-12T03:44:01Z INFO  gw_web3_indexer::runner] Sync block 3790
[2022-05-12T03:44:25Z INFO  gw_web3_indexer::runner] Sync block 3791
[2022-05-12T03:45:14Z INFO  gw_web3_indexer::runner] Sync block 3792
[2022-05-12T03:45:48Z INFO  gw_web3_indexer::runner] Sync block 3793
[2022-05-12T03:46:27Z INFO  gw_web3_indexer::runner] Sync block 3794
[2022-05-12T03:47:04Z INFO  gw_web3_indexer::runner] Sync block 3795
[2022-05-12T03:47:47Z INFO  gw_web3_indexer::runner] Sync block 3796
[2022-05-12T03:48:34Z INFO  gw_web3_indexer::runner] Sync block 3797
[2022-05-12T03:49:19Z INFO  gw_web3_indexer::runner] Sync block 3798
[2022-05-12T03:49:46Z INFO  gw_web3_indexer::runner] Sync block 3799
[2022-05-12T03:50:38Z INFO  gw_web3_indexer::runner] Sync block 3800
[2022-05-12T03:51:36Z INFO  gw_web3_indexer::runner] Sync block 3801
[2022-05-12T03:52:24Z INFO  gw_web3_indexer::runner] Sync block 3802
[2022-05-12T03:52:55Z INFO  gw_web3_indexer::ws_client] receive error tx receipt: 0x0x2c3cecc164f7ff0ab8c45fd3071d8c75733e8c9f0ef9f0dafb7679ad75b7ab1e
[2022-05-12T03:52:55Z WARN  gw_web3_indexer::error_receipt_indexer] unsupported polyjuice reason []
[2022-05-12T03:52:55Z INFO  gw_web3_indexer::error_receipt_indexer] delete error tx receipt 1
[2022-05-12T03:53:04Z INFO  gw_web3_indexer::runner] Sync block 3803
[2022-05-12T03:53:12Z INFO  gw_web3_indexer::ws_client] receive error tx receipt: 0x0x0ebf723b13989b2fa38f52b72116a666ec30a3c9789b7dd5870fe644a3e4f1ba
[2022-05-12T03:53:12Z WARN  gw_web3_indexer::error_receipt_indexer] unsupported polyjuice reason []
[2022-05-12T03:53:12Z INFO  gw_web3_indexer::error_receipt_indexer] delete error tx receipt 0
Error: error returned from database: sorry, too many clients already

Caused by:
    sorry, too many clients already
[2022-05-12T03:53:56Z INFO  gw_web3_indexer::ws_client] Successfully connected
[2022-05-12T03:53:56Z INFO  gw_web3_indexer::ws_client] Message send ok
[2022-05-12T03:53:56Z INFO  gw_web3_indexer::ws_client] ErrorTxReceipt type mismatch: invalid type: string "0x1", expected struct ErrorTxReceipt
[2022-05-12T03:53:58Z INFO  gw_web3_indexer::runner] Sync block 3804
[2022-05-12T03:54:28Z INFO  gw_web3_indexer::runner] Sync block 3805
[2022-05-12T03:55:23Z INFO  gw_web3_indexer::runner] Sync block 3806
[2022-05-12T03:56:09Z INFO  gw_web3_indexer::runner] Sync block 3807
[2022-05-12T03:57:09Z INFO  gw_web3_indexer::runner] Sync block 3808
[2022-05-12T03:57:53Z INFO  gw_web3_indexer::ws_client] receive error tx receipt: 0x0x323a475fd0e32d148d644970ca5839cca01bbe156ca33f5a48f1cdab0959672b
[2022-05-12T03:57:53Z WARN  gw_web3_indexer::error_receipt_indexer] unsupported polyjuice reason []
[2022-05-12T03:58:07Z INFO  gw_web3_indexer::runner] Sync block 3809
[2022-05-12T03:58:11Z INFO  gw_web3_indexer::ws_client] receive error tx receipt: 0x0x6cafbb0cb7ca6f6dfa547a4a0e9661fc4c850878d85c1692614f0971c14e750b
[2022-05-12T03:58:11Z WARN  gw_web3_indexer::error_receipt_indexer] unsupported polyjuice reason []
[2022-05-12T03:58:11Z WARN  sqlx::query] DELETE FROM error_transactions WHERE …; rows: 0, elapsed: 17.937s
    
    DELETE FROM
      error_transactions
    WHERE
      block_number <= $1
    
[2022-05-12T03:58:11Z INFO  gw_web3_indexer::error_receipt_indexer] delete error tx receipt 2
[2022-05-12T03:58:22Z INFO  gw_web3_indexer::ws_client] receive error tx receipt: 0x0x768f2a6956dfa83bf6075ae378d4151a3d56a84ebf32e446c8cb99ee52136d47
[2022-05-12T03:58:22Z WARN  gw_web3_indexer::error_receipt_indexer] unsupported polyjuice reason []
[2022-05-12T03:59:05Z INFO  gw_web3_indexer::runner] Sync block 3810
[2022-05-12T03:59:56Z INFO  gw_web3_indexer::runner] Sync block 3811
[2022-05-12T04:00:45Z INFO  gw_web3_indexer::runner] Sync block 3812
[2022-05-12T04:01:21Z INFO  gw_web3_indexer::ws_client] receive error tx receipt: 0x0x234dbdc5b4f0e564f1a17f0f97102289ffcc8c1bad2401c693f8d34449685e1f
[2022-05-12T04:01:21Z WARN  gw_web3_indexer::error_receipt_indexer] unsupported polyjuice reason []
[2022-05-12T04:01:21Z ERROR gw_web3_indexer::error_receipt_indexer] clear expired block error receipt error communicating with the server: connection aborted
[2022-05-12T04:01:28Z INFO  gw_web3_indexer::runner] Sync block 3813
Error: error returned from database: sorry, too many clients already

Caused by:
    sorry, too many clients already
[2022-05-12T04:02:30Z INFO  gw_web3_indexer::ws_client] Successfully connected
[2022-05-12T04:02:30Z INFO  gw_web3_indexer::ws_client] Message send ok
[2022-05-12T04:02:30Z INFO  gw_web3_indexer::ws_client] ErrorTxReceipt type mismatch: invalid type: string "0x2", expected struct ErrorTxReceipt
Error: error returned from database: sorry, too many clients already

Caused by:
    sorry, too many clients already
[2022-05-12T04:02:34Z INFO  gw_web3_indexer::ws_client] Successfully connected
[2022-05-12T04:02:34Z INFO  gw_web3_indexer::ws_client] Message send ok
[2022-05-12T04:02:34Z INFO  gw_web3_indexer::ws_client] ErrorTxReceipt type mismatch: invalid type: string "0x3", expected struct ErrorTxReceipt
Error: error returned from database: sorry, too many clients already

Caused by:
    sorry, too many clients already
@keroro520
Copy link
Contributor

keroro520 commented May 12, 2022 •
edited
Loading

Thanks for your report. @gpBlockchain

It seems this report involves several issues related to web3-indexer:

  • Unsupported Polyjuice reason

    [2022-05-12T03:52:55Z WARN  gw_web3_indexer::error_receipt_indexer] unsupported polyjuice reason []

  • Too many connections to the database

    Error: error returned from database: sorry, too many clients already

Caused by:
    sorry, too many clients already

  • Invalid ErrorTxReceipt

    [2022-05-12T04:02:34Z INFO  gw_web3_indexer::ws_client] ErrorTxReceipt type mismatch: invalid type: string "0x3", expected struct ErrorTxReceipt

Could you please confirm these issues? @classicalliu cc @Flouse

@Flouse
Copy link
Collaborator

Flouse commented May 12, 2022

Is that a clean env?
Would you clean your workspace and try again?

@classicalliu
Copy link
Contributor

unsupported polyjuice reason

unsupported polyjuice reason and ErrorTxReceipt type mismatch could be ignored.

sorry, too many clients already How to reproduce this error ?

@gpBlockchain
Copy link
Author

unsupported polyjuice reason

unsupported polyjuice reason and ErrorTxReceipt type mismatch could be ignored.

sorry, too many clients already How to reproduce this error ?

stress test ？i grep log " too many clients already"

guopenglin@192 godwoken-kicker % cat kick.log | grep " too many clients already" | wc -l
   57786

too many client aleady Up and down related logs

web3_1                            | [1]     at TCP.onStreamRead (internal/stream_base_commons.js:188:23)
web3_1                            | [1] 2022-05-12T05:04:57.117Z [info]: POST / 200 1872ms @eth_getBlockByNumber
web3_1                            | [1] 2022-05-12T05:04:57.129Z [info]: POST / 200 3580ms @eth_getBlockByNumber
web3_1                            | [1] 2022-05-12T05:04:57.129Z [error]: JSONRPC Server Error: [bound getBlockByNumber] error: sorry, too many clients already error: sorry, too many clients already
web3_1                            | [1]     at Parser.parseErrorMessage (/godwoken-web3/node_modules/pg-protocol/dist/parser.js:287:98)
web3_1                            | [1]     at Parser.handlePacket (/godwoken-web3/node_modules/pg-protocol/dist/parser.js:126:29)
web3_1                            | [1]     at Parser.parse (/godwoken-web3/node_modules/pg-protocol/dist/parser.js:39:38)
--
web3_1                            | [1] 2022-05-12T05:04:57.217Z [info]: POST / 200 3314ms @eth_getBlockByNumber
web3_1                            | [1] 2022-05-12T05:04:57.217Z [info]: POST / 200 2693ms @eth_getBlockByNumber
web3_1                            | [1] 2022-05-12T05:04:57.222Z [info]: POST / 200 3159ms @eth_getBlockByNumber
web3_1                            | [1] 2022-05-12T05:04:57.222Z [error]: JSONRPC Server Error: [bound getBlockByNumber] error: sorry, too many clients already error: sorry, too many clients already
web3_1                            | [1]     at Parser.parseErrorMessage (/godwoken-web3/node_modules/pg-protocol/dist/parser.js:287:98)
web3_1                            | [1]     at Parser.handlePacket (/godwoken-web3/node_modules/pg-protocol/dist/parser.js:126:29)
web3_1                            | [1]     at Parser.parse (/godwoken-web3/node_modules/pg-protocol/dist/parser.js:39:38)
--
web3_1                            | [1] 2022-05-12T05:04:57.236Z [info]: POST / 200 2568ms @eth_getBlockByNumber
web3_1                            | [1] 2022-05-12T05:04:57.240Z [info]: POST / 200 1873ms @eth_getBlockByNumber
web3_1                            | [1] 2022-05-12T05:04:57.244Z [info]: POST / 200 2573ms @eth_getBlockByNumber
web3_1                            | [1] 2022-05-12T05:04:57.250Z [error]: JSONRPC Server Error: [bound getBlockByNumber] error: sorry, too many clients already error: sorry, too many clients already
web3_1                            | [1]     at Parser.parseErrorMessage (/godwoken-web3/node_modules/pg-protocol/dist/parser.js:287:98)
web3_1                            | [1]     at Parser.handlePacket (/godwoken-web3/node_modules/pg-protocol/dist/parser.js:126:29)
web3_1                            | [1]     at Parser.parse (/godwoken-web3/node_modules/pg-protocol/dist/parser.js:39:38)
--
web3_1                            | [1] 2022-05-12T05:04:57.309Z [info]: POST / 200 3086ms @eth_getBlockByNumber
web3_1                            | [1] 2022-05-12T05:04:57.313Z [info]: POST / 200 2357ms @eth_getBlockByNumber
web3_1                            | [1] 2022-05-12T05:04:57.313Z [info]: POST / 200 2347ms @eth_getBlockByNumber
web3_1                            | [1] 2022-05-12T05:04:57.315Z [error]: JSONRPC Server Error: [bound getBlockByNumber] error: sorry, too many clients already error: sorry, too many clients already
web3_1                            | [1]     at Parser.parseErrorMessage (/godwoken-web3/node_modules/pg-protocol/dist/parser.js:287:98)
web3_1                            | [1]     at Parser.handlePacket (/godwoken-web3/node_modules/pg-protocol/dist/parser.js:126:29)
web3_1                            | [1]     at Parser.parse (/godwoken-web3/node_modules/pg-protocol/dist/parser.js:39:38)
--
web3_1                            | [1] 2022-05-12T05:04:57.321Z [info]: POST / 200 2985ms @eth_getBlockByNumber
web3_1                            | [1] 2022-05-12T05:04:57.321Z [info]: POST / 200 2339ms @eth_getBlockByNumber
web3_1                            | [1] 2022-05-12T05:04:57.322Z [info]: POST / 200 2136ms @eth_getBlockByNumber
web3_1                            | [1] 2022-05-12T05:04:57.322Z [error]: JSONRPC Server Error: [bound getBlockByNumber] error: sorry, too many clients already error: sorry, too many clients already
web3_1                            | [1]     at Parser.parseErrorMessage (/godwoken-web3/node_modules/pg-protocol/dist/parser.js:287:98)
web3_1                            | [1]     at Parser.handlePacket (/godwoken-web3/node_modules/pg-protocol/dist/parser.js:126:29)
web3_1                            | [1]     at Parser.parse (/godwoken-web3/node_modules/pg-protocol/dist/parser.js:39:38)
--
web3_1                            | [1] 2022-05-12T05:04:57.324Z [info]: POST / 200 1965ms @eth_getBlockByNumber
web3_1                            | [1] 2022-05-12T05:04:57.325Z [info]: POST / 200 1051ms @eth_getBlockByNumber
web3_1                            | [1] 2022-05-12T05:04:57.326Z [info]: POST / 200 2338ms @eth_getBlockByNumber
web3_1                            | [1] 2022-05-12T05:04:57.327Z [error]: JSONRPC Server Error: [bound getBlockByNumber] error: sorry, too many clients already error: sorry, too many clients already
web3_1                            | [1]     at Parser.parseErrorMessage (/godwoken-web3/node_modules/pg-protocol/dist/parser.js:287:98)
web3_1                            | [1]     at Parser.handlePacket (/godwoken-web3/node_modules/pg-protocol/dist/parser.js:126:29)
web3_1                            | [1]     at Parser.parse (/godwoken-web3/node_modules/pg-protocol/dist/parser.js:39:38)
--

@gpBlockchain
Copy link
Author

gpBlockchain commented May 12, 2022 •
edited
Loading

cat kick.log| grep 2022-05-12T04:02:3 > time.log
time.log

docker_web3-index should try to catch the too many clients exception and try again？

@Flouse Flouse added the bug Something isn't working label May 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@classicalliu classicalliu

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@Flouse @keroro520 @classicalliu @gpBlockchain