障害後にノードを再起動するためのTezosノードCPRスクリプトの問題

Question

障害後にノードを再起動するためのTezosノードCPRスクリプトの問題

- 5
- 78 2019-03-23
Tezos Node CPR スクリプトでいくつかの問題に直面しています Etom Knudsenから.障害が発生した後、ノードを自動的に再起動します.スクリプトには、35行目のネットワーク統計を呼び出す関数にバグがあります.これは、 GitHub .

スクリプトはノードからネットワーク統計を呼び出して、ノードが他のピアに接続されているかどうかを確認します.

curl -s 127.0.0.1:8732/network/stat/

そうでない場合、呼び出しは次のようなものを返します.

{"total_sent":"8898729","total_recv":"9544788","current_inflow":0,"current_outflow":0}

また、次の呼び出しによって、ノードがネットワークから最後のブロックをいつ受信したかを確認します.

curl -s 127.0.0.1:8732/chains/main/blocks/head/header/

ネットワーク障害時にrpc呼び出しが空の行を返し、スクリプトが最後のブロックがいつ到着したかを判断できないため、後者は機能しなかったと思われます.デフォルトでは、スクリプトは、この時間中にブロックが到着しなかった180秒後にノードを再起動するように構成されています.ただし、たとえば次の行で示されているように、これは失敗します.

2019-03-23T06:55:56Z Waiting. Last block was 515 secs ago

最新のノード更新(コミットハッシュ:366f64f3 ...)以降、ノードが他のピアに接続されていない場合にもデータを送受信しているように見えるため、ネットワーク統計機能も失敗すると思います.これは次のように示されます.

2019-03-23T06:55:35Z Waiting. Last block was 494 secs ago) 2019-03-23T06:55:40Z Found p2p activity 2019-03-23T06:55:40Z Looking for p2p avtivity - will wait for max 90 secs 2019-03-23T06:55:40Z Waiting. Last block was 499 secs ago) 2019-03-23T06:55:45Z Found p2p activity 2019-03-23T06:55:45Z Looking for p2p avtivity - will wait for max 90 secs 2019-03-23T06:55:45Z Waiting. Last block was 504 secs ago) 2019-03-23T06:55:51Z Found p2p activity

どちらの問題でも、スクリプトが意図したとおりに機能せず、ノードが自動的に再起動しなくなります.

誰かがこれらの問題を観察し、それに応じてスクリプトを修正したことはありますか?または、Tezosノード用のより適切に機能する別の再起動スクリプトを知っている人はいますか?
I'm facing some issues with the Tezos Node CPR script from Etom Knudsen, which serves to automatically restart the node after failure. The script has a bug in the function to call network stats in line 35, which I solved using the solution suggested by a user on GitHub.

The script calls the network statistics from the node to check if it is connected to other peers:

curl -s 127.0.0.1:8732/network/stat/

If it is not, the call will return something like:

{"total_sent":"8898729","total_recv":"9544788","current_inflow":0,"current_outflow":0}

It also checks when the node received the last block from the network, by calling:

curl -s 127.0.0.1:8732/chains/main/blocks/head/header/

I suspect that the latter has never worked, as during network failures the rpc call seems to return an empty line, preventing the script to determine when the last block arrived. By default the script is configured to restart the node after 180s when no block arrived during this time. However, it fails to do so, as indicated, for instance, by the following line:

2019-03-23T06:55:56Z Waiting. Last block was 515 secs ago

I believe since the most recent node update (commit hash: 366f64f3...) also the network stat function fails, as the node seems to receive and send some data also when it is not connected to any other peer. This is indicated by the following:

2019-03-23T06:55:35Z Waiting. Last block was 494 secs ago) 2019-03-23T06:55:40Z Found p2p activity 2019-03-23T06:55:40Z Looking for p2p avtivity - will wait for max 90 secs 2019-03-23T06:55:40Z Waiting. Last block was 499 secs ago) 2019-03-23T06:55:45Z Found p2p activity 2019-03-23T06:55:45Z Looking for p2p avtivity - will wait for max 90 secs 2019-03-23T06:55:45Z Waiting. Last block was 504 secs ago) 2019-03-23T06:55:51Z Found p2p activity

Both issues result in the script not working as intended and the node does not automatically restart any longer.

Has anyone also observed these issues and revised the script accordingly? Or does anyone know another restart script for the Tezos node that works better?
node rpc

1 回答

投票

cryptodad · Answer 1 · 2019-03-24

1

2019-03-24

Etom Knudsen のGitHubページにも記載されているように、 33行目のgetTimeSinceLastBlock関数にバグがある可能性があります.

不思議なことに、ノードを手動で再起動すると、CPRスクリプトも起動し、そのログに次のエラーメッセージが表示されます.

 Looking for p2p avtivity - will wait for max 90 secs
Waiting. Last block was 510 secs ago)
Waiting. Last block was 515 secs ago)
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/lib/python3.6/json/__init__.py", line 299, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "/usr/lib/python3.6/json/__init__.py", line 354, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.6/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
date: invalid date '+%s'
/usr/local/bin/tezos-node-cpr.sh: 33: /usr/local/bin/tezos-node-cpr.sh: arithmetic expression: expecting primary: "1553324161-"
Network OK. No p2p activity and/or too long ( secs since last block. Restarting node!`

したがって、Pythonで使用される秒から文字列の時間への変換にバグがあり、スクリプトが最後のブロックからの時間を決定できないようです.

As I also noted on the GitHub page of Etom Knudsen, it could be that there is a bug in the getTimeSinceLastBlock function in line 33.

Strangely, the moment I manually restart the node, the CPR script also fires and its log shows the following error message:

Looking for p2p avtivity - will wait for max 90 secs
Waiting. Last block was 510 secs ago)
Waiting. Last block was 515 secs ago)
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/lib/python3.6/json/__init__.py", line 299, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "/usr/lib/python3.6/json/__init__.py", line 354, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.6/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
date: invalid date '+%s'
/usr/local/bin/tezos-node-cpr.sh: 33: /usr/local/bin/tezos-node-cpr.sh: arithmetic expression: expecting primary: "1553324161-"
Network OK. No p2p activity and/or too long ( secs since last block. Restarting node!`

So it seems that there is a bug in the conversion of seconds to string time to be used in python, preventing the script to determine the time since the last block.

考えてみると、起動時にp2pアクティビティが発生しないため、ノードを手動で再起動するとcprスクリプトが起動し、スクリプトがノードを再起動するための基準が満たされます.

When I think about it, it makes sense that the cpr script fires when the node is restarted manually, as there will be no p2p activity during startup, fulfilling a criterium for the script to restart the node.
- 0
- 2019-03-25
- cryptodad