SMTP Mail Sending Failure - Connection troubleshooting

Background:

I have encounter issue when sending email from my application server via SMTP server. The syndrome is only smaller size email is able to send out successfully but not the larger size email. I tried to upgrade OS on the application server from windows 2012 to 2016, using different module/plugin and still encountering the same issue. The issue was going nowhere until i start tracing the network using wireshark.

Summary:

Through network tracing it is able to identify the issue was causing by IP-Sec encryption that transmitting packet with size larger than 1400 will be failing. By limiting the MTU to 1400, the email is able to send out successfully now.

Details of the process:

1. Download wireshark
2. Install wireshark on the server that is having connection issue
• I am installing with all default options including Npcap
3. Open wireshark
4. Select the network interface that you want wireshark to monitor. 
• Can ignore all those without traffic 






5. Stop the tracing and add in the filtering criteria to remove all the noise
• For example in my case, i want to connect from my application server to SMTP server for sending email and lets say my SMTP server ip address is 10.198.1.133 then my filter will be:
• ip.dst==10.198.1.133 or ip.src==10.198.1.133
• In this case the wireshark will capture the network traffic from my application to SMTP server and also all network traffic from the SMTP server to my application server




6. Start the tracing again once the filter is entered



7. Initiate the connection attempt, in my case is trying to send email via the SMTP server
8. Stop the tracing once the connection attempt is completed

9. Examine the logs
• From the log, we will be able to see all traffic between the application server and SMTP server.
10. In my case, i can see that the connection went smoothly all the way until Data Segment started to send over to server (Those in black background)
11. We can see that the SMTP server is not responding and the application server keep on transmitting
12. We have also noticed that the length of the data packet is 5894 while the Retransmission packet length is 1514

13. Then we clean up the logs and start the tracing again on wireshark
14. This round we try with ping command
• ping -f -l 1350 SMTP_SERVER_IP
• Ping managed to go through with packet size of 1350 bytes
• ping -f -l 1400 SMTP_SERVER_IP
• Ping failed to go through with packet size of 1400 bytes

15. Check on the wireshark log and we are able to see all the ping with length of 1342 went through successfully
16. All the ping with length 1442 is failing to get response from SMTP server
    

17. From further checking it is because of the network between application server and SMTP server has been encrypted using IPSEC and any packet that is larger than 1400 will be splitting into different packet but couldn't handle at the SMTP server side.
18. We have then try to limit the maximum transmission unit (MTU) to 1400 and the problem got resolved.
19. To check the MTU size can use the command below
• netsh int upv4 show int
• Also take note of the idx, we will need it in later command

20. Issue the command to change the MTU size to 1400
• netsh interface ipv4 set subinterface XX MTU=1400 store=persistent
• XX refer to the Idx from step above

21. Check again the MTU after running the command
• The MTU value should now be 1400


22. Test sending email again via SMTP and it has now gone through successfully

5.