You have everything ready for deployment day. You have a list of projects that went thru development, quality assurance testing, broken builds, good builds, and demos. After business owners signed off on these projects, it is time to deploy to production.
What could go wrong?
In this post, I want to share how a CTO saved a deployment day. In this company, deployments took place at 5 am CST. Most of the time, we deployed bug fixes, new features and enhancements every 2 weeks. Our team was ready for deployment day. We had 5 major products but only 2 projects needed a release.
Before the deployment day, we had a short meeting to discuss any deployment risks. Everyone is comfortable and we set a deployment day for Wednesday at 5 am.
During the deployment, we triggered 2 production builds using Octopus Deploy and everything worked as expected. The team was happy and we went downstairs to celebrate with coffee and breakfast burritos. Yummy.
At 9 am, our IT operations team started getting emails that none of our applications worked. Everyone was shocked. We deployed with no issues this morning. What is going on? One of our application servers was unresponsive since CPU utilization approached 100%. We also noticed that SQL server was not performing at optimal speeds.
At 9:30 am, our CTO got notified about our current deployment situation. He started asking questions like “What is currently running on our application server?” IT director says, “just .NET sites”. We checked every single web site for any issues but nothing. They looked correct. CTO started looking at windows services and saw a dotnetnuke executable. No one knew that we had a very old dotnetnuke installation. IT director stopped dotnetnuke.exe and we started seeing CPU back to normal levels. We checked SQL server and it is also back to normal. At this point, our team was notified of the issue and how the CTO was able to identified the issue.
Remember this story when you run into deployment issues. Ask questions until you find answers to these problems.