国立天文台・天文シミュレーションプロジェクト
XC50システム利用者各位
いつも国立天文台・天文シミュレーションプロジェクトのXC50システムをご利用いただき、誠にありがとうございます。
XC50システムにおきまして、大規模なジョブを実行した際にI/Oによりネットワーク通信での遅延が生じてLustreファイルシステムがタイムアウトとなる現象が確認されました。
このエラーが発生したことにより、XC50システムより出力されたファイルの一部が破損した事例が報告されています。
以下に5/29の定期保守作業以降に記録されたエラーログとそのノードで実行されていたジョブを記載いたしますので、
該当するジョブを投入していたユーザは出力ファイルの確認をお願いいたします。
また、ログインノードにおいて生じたエラーにつきましてはユーザの特定ができませんでした。
エラーの発生原因もまだ完全に特定できてはおりませんので、全てのユーザの皆様に今一度出力ファイルに破損がないかをご確認いただきますようお願い申し上げます。
あわせて、本エラーによる被害規模を把握するため、出力ファイルに破損が確認された場合にはお手数ですがご報告いただけますと幸いです。
ユーザの皆様に多大なご迷惑おかけいたしますことを深くお詫び申し上げます。
今後とも天文シミュレーションプロジェクトXC50システムのご利用をよろしくお願いいたします。
本メールに関するご質問やご意見は以下のページからお寄せ下さい。
http://www.cfca.nao.ac.jp/inquiry/
---------------------------------------------
console-20180603:2018-06-03T17:10:07.722107+09:00 c4-0c2s11n1 LustreError: 167-0: lustrefs-OST0047-osc-ffff885e9240d000: This client was evicted by lustrefs-OST0047; in progress operations using this service will fail.
+++ 06/03/2018 8487.sdb user=asahnayt jobname=GRboltza queue=ex-s
console-20180608:2018-06-08T00:05:21.137081+09:00 c1-0c0s12n3 LustreError: 167-0: lustrefs-OST0009-osc-ffff885e90f08800: This client was evicted by lustrefs-OST0009; in progress operations using this service will fail.
+++ 06/08/2018 12307.sdb user=sugiymnn jobname=B0079 queue=bulk-b
console-20180608:2018-06-08T00:22:01.894654+09:00 c0-0c0s9n2 LustreError: 167-0: lustrefs-OST0023-osc-ffff882e8381c800: This client was evicted by lustrefs-OST0023; in progress operations using this service will fail.
+++ 06/08/2018 12947.sdb user=kanagwkz jobname=ma12q13 queue=bulk-b
console-20180608:2018-06-08T02:31:20.075603+09:00 c3-0c0s9n2 LustreError: 167-0: lustrefs-OST001e-osc-ffff885e91838000: This client was evicted by lustrefs-OST001e; in progress operations using this service will fail.
console-20180608:2018-06-08T02:31:20.081794+09:00 c4-0c2s9n2 LustreError: 167-0: lustrefs-OST0020-osc-ffff882e82d51800: This client was evicted by lustrefs-OST0020; in progress operations using this service will fail.
console-20180608:2018-06-08T02:31:20.081808+09:00 c4-0c2s9n0 LustreError: 167-0: lustrefs-OST0018-osc-ffff882e8335d000: This client was evicted by lustrefs-OST0018; in progress operations using this service will fail.
console-20180608:2018-06-08T02:31:20.112598+09:00 c5-0c0s1n3 LustreError: 167-0: lustrefs-OST0026-osc-ffff885e90feb800: This client was evicted by lustrefs-OST0026; in progress operations using this service will fail.
console-20180608:2018-06-08T02:31:20.112626+09:00 c5-0c0s1n0 LustreError: 167-0: lustrefs-OST0019-osc-ffff882e903ea000: This client was evicted by lustrefs-OST0019; in progress operations using this service will fail.
+++ 06/08/2018 9842.sdb user=kurodatk jobname=Z70R10B12 queue=bulk-a
console-20180609:2018-06-09T04:41:51.521464+09:00 c2-0c0s12n3 LustreError: 167-0: lustrefs-MDT0000-mdc-ffff881fc9d01800: This client was evicted by lustrefs-MDT0000; in progress operations using this service will fail.
+++ Login node: xc04
console-20180614:2018-06-14T00:48:54.731923+09:00 c5-0c0s13n1 LustreError: 167-0: lustrefs-OST0052-osc-ffff882e82952000: This client was evicted by lustrefs-OST0052; in progress operations using this service will fail.
console-20180614:2018-06-14T00:48:54.732357+09:00 c5-0c0s13n2 LustreError: 167-0: lustrefs-OST0041-osc-ffff885e922cb800: This client was evicted by lustrefs-OST0041; in progress operations using this service will fail.
console-20180614:2018-06-14T00:48:54.732733+09:00 c5-0c0s13n2 LustreError: 167-0: lustrefs-OST0046-osc-ffff885e922cb800: This client was evicted by lustrefs-OST0046; in progress operations using this service will fail.
+++ 06/14/2018 19796.sdb user=takhshri jobname=gzip queue=large-bp
console-20180614:2018-06-14T17:44:24.832393+09:00 c2-0c0s12n3 LustreError: 167-0: lustrefs-OST0025-osc-ffff881fc9d01800: This client was evicted by lustrefs-OST0025; in progress operations using this service will fail.
+++ Login node: xc04
console-20180615:2018-06-15T23:05:30.871822+09:00 c0-0c0s12n3 LustreError: 167-0: lustrefs-OST0041-osc-ffff881f5352b800: This client was evicted by lustrefs-OST0041; in progress operations using this service will fail.
console-20180615:2018-06-15T23:05:36.140311+09:00 c0-0c0s12n3 LustreError: 167-0: lustrefs-OST0050-osc-ffff881f5352b800: This client was evicted by lustrefs-OST0050; in progress operations using this service will fail.
console-20180615:2018-06-15T23:05:36.140318+09:00 c0-0c0s12n3 LustreError: 167-0: lustrefs-OST0046-osc-ffff881f5352b800: This client was evicted by lustrefs-OST0046; in progress operations using this service will fail.
+++ Login node: xc01
console-20180623:2018-06-23T19:39:16.825093+09:00 c3-0c1s14n2 LustreError: 167-0: lustrefs-OST001c-osc-ffff885e9e60b000: This client was evicted by lustrefs-OST001c; in progress operations using this service will fail.
+++ 06/23/2018 32460.sdb user=ishiymtm jobname=Phi4096Re queue=large-a
console-20180625:2018-06-25T10:15:45.504548+09:00 c3-0c0s11n1 LustreError: 167-0: lustrefs-OST0052-osc-ffff885e8dfff000: This client was evicted by lustrefs-OST0052; in progress operations using this service will fail.
console-20180625:2018-06-25T10:16:06.496732+09:00 c3-0c0s11n1 LustreError: 167-0: lustrefs-OST0000-osc-ffff885e8dfff000: This client was evicted by lustrefs-OST0000; in progress operations using this service will fail.
console-20180625:2018-06-25T10:22:04.658183+09:00 c3-0c0s11n1 LustreError: 167-0: lustrefs-MDT0000-mdc-ffff885e8dfff000: This client was evicted by lustrefs-MDT0000; in progress operations using this service will fail.
console-20180625:2018-06-25T10:22:37.027947+09:00 c3-0c0s11n1 LustreError: 167-0: lustrefs-OST0004-osc-ffff885e8dfff000: This client was evicted by lustrefs-OST0004; in progress operations using this service will fail.
console-20180625:2018-06-25T10:23:03.119610+09:00 c3-0c0s11n1 LustreError: 167-0: lustrefs-OST004c-osc-ffff885e8dfff000: This client was evicted by lustrefs-OST004c; in progress operations using this service will fail.
console-20180625:2018-06-25T10:23:24.168457+09:00 c3-0c0s11n1 LustreError: 167-0: lustrefs-OST0007-osc-ffff885e8dfff000: This client was evicted by lustrefs-OST0007; in progress operations using this service will fail.
+++ 06/25/2018 33333.sdb user=kurodatk jobname=S20_0.30 queue=bulk-a
console-20180626:2018-06-26T01:35:43.076079+09:00 c2-0c0s4n1 LustreError: 167-0: lustrefs-OST001a-osc-ffff881fcadcd800: This client was evicted by lustrefs-OST001a; in progress operations using this service will fail.
+++ Login node: xc02
console-20180701:2018-07-01T07:51:48.374287+09:00 c2-0c0s12n3 LustreError: 167-0: lustrefs-MDT0000-mdc-ffff881fc9d01800: This client was evicted by lustrefs-MDT0000; in progress operations using this service will fail.
+++ Login node: xc04