Debugging EC2 Status Check Failed due to Networking Issue

Recently I was modifying the EBS encryption of an instance, from the default AWS EBS KMS key to CMK. In order to do so, I created AMI of the instance and noted down other important parameters like Instance type, VPC, Subnet, Security Group, IAM role, Key, and most importantly Private IP of the instance. Then I terminated the instance and launched a new one using the AMI we created earlier and specified all the parameters similar to our original instance.

Now here comes the interesting part as the instance started failing its instance reachability status check. I tried stopping/starting it a couple of times, but it didn’t help.

1. Detach root volume from the instance

2. Launch another instance(let’s call it debugging instance) in the same AZ where my detached root volume is.

3. Attach that root volume to the debugging instance, as a secondary volume

4. Execute lsblk command to find out the partition name for the secondary volume. In my case, it was /dev/xvdf1.

5. Mount the secondary volume to /mnt, using sudo mount /dev/xvdf1 /mnt

6. Check /etc/sysconfig/network-scripts/ifcfg-eth0 and /mnt/etc/sysconfig/network-scripts/ifcfg-eth0

And I found that there is some difference between these 2 files. One stark difference was that in /mnt/etc/sysconfig/network-scripts/ifcfg-eth0, HWADDR parameter was hardcoded(which I suspect was causing the issue).

7. Take backup of /mnt/etc/sysconfig/network-scripts/ifcfg-eth0 and copy /etc/sysconfig/network-scripts/ifcfg-eth0 to /mnt/etc/sysconfig/network-scripts/ifcfg-eth0

sudo cp /mnt/etc/sysconfig/network-scripts/ifcfg-eth0 /mnt/etc/sysconfig/network-scripts/ifcfg-eth0-backup
sudp cp /etc/sysconfig/network-scripts/ifcfg-eth0 /mnt/etc/sysconfig/network-scripts/ifcfg-eth0
sudo chattr +i /mnt/etc/sysconfig/network-scripts/ifcfg-eth0
sudo touch /mnt/.autorelabel
umount /mnt

8. Now detach the secondary volume from our debugging instance and attach it back to our instance as root volume.

Now when I started the instance, it was able to come up without any issue.

Experienced Cloud Consultant with a knack of automation. Linkedin profile: https://www.linkedin.com/in/vinayakpandeyit/