http://stab.st-andrews.ac.uk/wiki/index.php?title=Node9_network_failure_incident_16-20.03.2017&feed=atom&action=historyNode9 network failure incident 16-20.03.2017 - Revision history2024-03-28T23:43:06ZRevision history for this page on the wikiMediaWiki 1.30.0http://stab.st-andrews.ac.uk/wiki/index.php?title=Node9_network_failure_incident_16-20.03.2017&diff=1252&oldid=prevRf: Created page with "= Introduction = Node9 usually a perfectly behaving machine suddenly loses network connections. = Troubleshooting = Going through all the procedures to see which component..."2017-03-20T18:05:55Z<p>Created page with "= Introduction = Node9 usually a perfectly behaving machine suddenly loses network connections. = Troubleshooting = Going through all the procedures to see which component..."</p>
<p><b>New page</b></p><div>= Introduction =<br />
<br />
Node9 usually a perfectly behaving machine suddenly loses network connections.<br />
<br />
= Troubleshooting =<br />
<br />
Going through all the procedures to see which component was to blame took an entire day, unfortunately during quite a busy period for projects. <br />
<br />
<br />
It turns out the machine nor its network interfaces were not to blame. This was time wasted because it turned out to be a Network Services issue. They were chaning around network hubs and some cabling did not get re-seated properly, it would seem.<br />
<br />
Here are the details<br />
<br />
The Butts Wynd Data Centre Core Switches' (BWDC Cisco Nexus 5672s) port-channel linking your High Performance Computing System (Viglen HX425Ca - Asset 03267, rack 17 position 35) is now up and running (see output included below), having re-seated the copper cabling connecting up both of its Ethernet interfaces. Please do let us know if this problem re-occurs.<br />
<br />
bwdc-n5672-0# show port-channel sum | inc Po146<br />
146 Po146(SU) Eth LACP Eth108/1/28(P)<br />
<br />
bwdc-n5672-0# show logging | inc Ethernet108/1/28<br />
...<br />
2017 Mar 20 16:15:03 bwdc-n5672-0 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel146: first operational port changed from none to Ethernet108/1/28<br />
2017 Mar 20 16:15:03 bwdc-n5672-0 %ETHPORT-5-IF_UP: Interface Ethernet108/1/28 is up in mode access<br />
<br />
bwdc-n5672-1# show port-channel sum | inc Po146<br />
146 Po146(SU) Eth LACP Eth109/1/28(P)bwdc-n5672-1#<br />
...<br />
2017 Mar 20 16:15:00 bwdc-n5672-1 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel146: first operational port changed from none to Ethernet109/1/28<br />
2017 Mar 20 16:15:00 bwdc-n5672-1 %ETHPORT-5-IF_UP: Interface Ethernet109/1/28 is up in mode access</div>Rf