Author: Raymond

Categories
General

Using AWS Python SDK to Read Simple Queue Service Messages

In this post, I want to share how to use AWS Simple Queue Service to manage email bounces and complaints. In my previous post, I wrote about sending emails using AWS Python SDK. First, we have to setup 2 SNS topics to handle email bounces and complaints. Go to the Simple Notification Service console and add a topic. After creating a SNS topic, you should see something similar to:

Pay attention to the protocol column above. The protocol for my email-bounce is set to SQS. This is very important since we want to process SQS messages every time SES receives a bounce or complaint. The endpoint is set to a SQS queue.

Now, let’s go back to the SES console and add a configuration set. Here is my SNS destination named email-bounces which is link to the email-bounce topic we created before.

During the sns setup, we also have to create a sqs queue to hold our messages.

Since SNS will be responsible to act as a proxy, we have to setup permissions for this queue to receive SNS messages.

After completing these steps, we are ready to read messages using Python.

I created a very simple Python script. At the top, I’m importing boto3 and my constructor is creating a sqsClient. To read messages, I created a method that takes an url as an argument. In line 24, you can see that the queueUrl is the location of our sqs email bounce. To actually read the message, we call receive_message and pass the sqs url and also MaxNumberOfMessages. In this case, I’m just using 1 to keep things simple. And finally we return the message.

I also have another method called GetEmailAddress that takes in a message as an argument. First, I retrieve the body string and start parsing it to find the email address. At the end of this method, I return the email address.

With this email address, I can clean up my records so I don’t send emails to it. That’s it for now. Next week, I’ll continue using Python scripts to interact with DynamoDB.

Categories
General

Sending emails with AWS Python SDK


Houston we have a problem. Not Houston. I’m the one with the problem. I need to send emails to my list of users but I know that some of these users were spam users. I need a way to send emails and also remove invalid emails from my system.

In the past I have used AWS .NET SDK to send emails and also track bounces and complaints. But I want to challenge myself so I want to play with the Python SDK. In this post, I want to share how to send emails using the AWS Python SDK.

Install the python SDK by following these instructions. Make sure to setup a profile that has permissions to send emails. In this example, I’m using a profile name “python-scripts” that has permissions to send ses emails. Make sure you set this name based on your setup.

After installing boto3, we are ready to create a class responsible to send emails. Take a look at EmailController.py below:


Now let’s create a class that uses EmailController.py. Take a look at EmailRunner.py.

I like to keep things simple. I have a csv file with all the email addresses. Each line in the csv file has an email address. Next, my class reads each line and sends an email. If there are any errors while sending the emails, I’ll use another csv file to keep track of these errors.

If we were able to send emails successfully, we will receive a valid response with status of 200.

That’s all for today. In a future post, I’m going to show you how to setup sns notifications for bounces and complaints. Have a nice day!

Categories
.Net AWS CodeDeploy General

Creating AWS CodeDeploy Deployments Using .NET SDK

This is part 3 in a series dedicated to AWS CodeDeploy API using .NET SDK. In part 1, I created the CodeDeploy application using ASP.NET Core MVC, C#, and DynamoDB. In part 2, I created the deployment group which has settings for alarms, load balance, auto scaling groups, deployment styles, and other settings.

In this post, I want to concentrate on creating the deployment. In the deployment request, you can specify revision information (Github or S3 settings), deployment group, auto rollback configuration, and other settings.

Talk is cheap. Show me the code!

If you want to follow along, you can visit my github repo at https://github.com/agileraymond/DotNetDeployments. Now that you have a reference to the repo, let’s modify our controller so we can display our view.

This controller action displays our AddDeployment view.

There is a limitation with this view since it only displays S3 settings. In a future post, I will revisit this view and add the Github repo as well. I wanted to get something working in a short amount of time. When the user clicks on Add Deployment button, it will call a Post controller action to trigger a new deployment.

If the user entered all required information, a new deployment will be created in the AWS CodeDeploy console.

If you want to read more about the AWS .NET SDK, follow this link https://docs.aws.amazon.com/sdkfornet/v3/apidocs/Index.html.

If you have any questions or issues with this code, contact me via twitter @agileraymond.

Have a nice day!

Categories
.Net AWS CI Code Deployment CodeDeploy Continuous Delivery

Creating AWS CodeDeploy Deployment Groups Using .NET SDK

In a previous post, I shared how to create codedeploy applications using the AWS .NET SDK. Adding an application is the foundation to get codedeploy working correctly. In this post, I want to continue this series and show you how to add deployment groups.

To see what parameters we need to add deployment groups, I’m going to read the official documentation here. Find the Amazon.CodeDeploy documentation on the left of the page, and then click on AmazonCodeDeployClient. All codedeploy operations will be handled by the AmazonCodeDeployClient. The method we need is CreateDeploymentGroupAsync. Since we are using .NET Core 2, we need to use the Async methods. CreateDeploymentGroupAsync takes 2 parameters: CreateDeploymentGroupRequest and CancellationToken.

These are CreateDeploymentGroupRequest’s properties:

– AlarmConfiguration: Gets and sets the property AlarmConfiguration. Information to add about Amazon CloudWatch alarms when the deployment group is created.

– ApplicationName: Gets and sets the property ApplicationName. The name of an AWS CodeDeploy application associated with the applicable IAM user or AWS account.

– AutoRollbackConfiguration: Gets and sets the property AutoRollbackConfiguration. Configuration information for an automatic rollback that is added when a deployment group is created.

– AutoScalingGroups: Gets and sets the property AutoScalingGroups. A list of associated Auto Scaling groups.

– BlueGreenDeploymentConfiguration: Gets and sets the property BlueGreenDeploymentConfiguration. Information about blue/green deployment options for a deployment group.

– DeploymentConfigName: Gets and sets the property DeploymentConfigName. If specified, the deployment configuration name can be either one of the predefined configurations provided with AWS CodeDeploy or a custom deployment configuration that you create by calling the create deployment configuration operation. CodeDeployDefault.OneAtATime is the default deployment configuration. It is used if a configuration isn’t specified for the deployment or the deployment group. For more information about the predefined deployment configurations in AWS CodeDeploy, see Working with Deployment Groups in AWS CodeDeploy in the AWS CodeDeploy User Guide.

– DeploymentGroupName: Gets and sets the property DeploymentGroupName. The name of a new deployment group for the specified application.

– DeploymentStyle: Gets and sets the property DeploymentStyle. Information about the type of deployment, in-place or blue/green, that you want to run and whether to route deployment traffic behind a load balancer.

– Ec2TagFilters: Gets and sets the property Ec2TagFilters. The Amazon EC2 tags on which to filter. The deployment group will include EC2 instances with any of the specified tags. Cannot be used in the same call as ec2TagSet.

– Ec2TagSet: Gets and sets the property Ec2TagSet. Information about groups of tags applied to EC2 instances. The deployment group will include only EC2 instances identified by all the tag groups. Cannot be used in the same call as ec2TagFilters.

– LoadBalancerInfo: Gets and sets the property LoadBalancerInfo. Information about the load balancer used in a deployment.

– OnPremisesInstanceTagFilters: Gets and sets the property OnPremisesInstanceTagFilters. The on-premises instance tags on which to filter. The deployment group will include on-premises instances with any of the specified tags. Cannot be used in the same call as OnPremisesTagSet.

– OnPremisesTagSet: Gets and sets the property OnPremisesTagSet. Information about groups of tags applied to on-premises instances. The deployment group will include only on-premises instances identified by all the tag groups. Cannot be used in the same call as onPremisesInstanceTagFilters.

– ServiceRoleArn: Gets and sets the property ServiceRoleArn. A service role ARN that allows AWS CodeDeploy to act on the user’s behalf when interacting with AWS services.

– TriggerConfigurations: Gets and sets the property TriggerConfigurations. Information about triggers to create when the deployment group is created. For examples, see Create a Trigger for an AWS CodeDeploy Event in the AWS CodeDeploy User Guide.

To keep my code example concise, I’m going to only use required properties to add a deployment group. Let’s start by adding the controller actions. Take a look at the gist below:

The first action returns a view so we can fill out application name, deployment group name, and service role arn. Take a look at the view:

I’m only displaying the required fields to create a new deployment group. This is how I like to develop my applications: add small features that work and then add more features and keep improving those features. It is very difficult to add perfect code at first. It is constant improvements that will yield better applications.

When the user clicks on add button, the post action will take care of sending the request to the codedeploy client. If the call to CreateDeploymentGroupAsync is successful, we will see a new deployment group in the aws console. To be able to understand deployment groups, we have to understand development environments. We usually have dev, test, and production environments. These environments are usually separated from each other. Dev environment is usually open for all developers. Test might be use to test actual deployments. And production only a couple of engineers should have access to that environment. In CodeDeploy, deployment groups allow you to mirror your development environment when it comes to deployment. For 1 application, you can setup 3 deployment groups (dev, test, and production). Each group will be linked to an EC2 instance(s) or on-premises servers. In a future post, I will provide examples with all these properties. Stay tuned!

Next: Creating AWS CodeDeploy Deployments Using .NET SDK

Categories
General

How to be Consistent with your Writing

I started writing on this blog on April 2014. This was my second attempt in starting a blog. During my first attempt back in 2013, I was very excited to start writing and sharing with my fellow software developers. I bought a domain and paid for hosting thru GoDaddy. I had big dreams with this new project but I could not write. I didn’t produce a single article. When it came time to renew my domain, I decided to cancel my hosting account and also not to renew my domain. I came up with so many excuses not to write anymore.

However, with the start of the new year, I decided to create a new blog. This time I decided to use solutionsByRaymond.com because I wanted to share solutions to common software problems. My goal was to create 1 article for month. So far I have only missed couple of months. To stay consistent on my blog, I read everyday. Currently I’m concentrating on cloud computing articles. AWS is my favorite cloud provider and there are blog posts everyday. My second favorite topic is .NET framework, C#, and its related tools. Let’s not forget about HTML, CSS, and JavaScript.

After reading, I like to write for a few minutes using my cell phone. I use Evernote and take notes without paying attention to grammar or errors. During this time, I want to capture my ideas because I will forget later. When I’m using my MacBook Pro, I write directly on my blog. I also use Visual Studio Code to take notes. When I’m ready to publish an article, I read it multiple times to find any errors.

This is my advice to other developers starting with a blog. Make it a habit to write everyday. It doesn’t have to be 500 or 1000 words. Your goal should be to write something everyday. This will help you stay consistent with your writing. If you write something this week and don’t produce any content the following 2 weeks, it’s going to be difficult to stay consistent.

Stop reading now and go write something. See you next time.

 

 

Categories
.Net ASP.NET MVC AWS Code Deployment CodeDeploy Continuous Delivery

Creating AWS CodeDeploy Application Using .NET SDK

I’m a big fan of AWS and its cloud services. S3 has changed the way we store objects. EC2 has enabled us to spin up instances quickly and in a cost effective way. CodeDeploy helps developers deploy applications to EC2 instances and also on-premises servers. In this post, I want to share how to create a CodeDeploy application using the AWS .NET SDK.

First, create a new asp.net mvc project using Visual Studio or Visual Studio Code. Make sure to target .NET Core 2.0. Now that we have a new project, let’s add codedeploy nuget package. If you are using VS Code, use the built-in terminal and type “dotnet add package AWSSDK.CodeDeploy”. This will add the latest version of CodeDeploy. We also need to add an AWS nuget package to inject codedeploy service in our Startup.cs file. Run in the terminal “dotnet add package AWSSDK.Extensions.NETCore.Setup” to add this package.

Let’s modify our Startup.cs file to look like below:

In order to have access to AWS CodeDeploy api calls, we need to setup our credentials. For this example, I’m using a profile to store AWS access key id and secret access key. I’m storing the credentials outside my source code in a profile file so I can keep them secure. Also those credentials will be different between developers. To help you with this setup, follow this document to setup your AWS credentials and make sure to pay special attention to the profile section.

With the AWS profile in place, we need to add a reference to it. Take a look at my appsettings.json file. I named mine dotnetdeployments-profile since you can have multiple profiles.

We are ready to start looking that controller. Take a look at the controller below:

In connection with the controller, we also need to look at the view:

The add action in our controller displays the add view only. The view has a reference to a model called CreateApplicationRequest and this class has a property named ApplicationName. When the user clicks on the add button, a post will be triggered back to our controller. And finally, it will call the api method CreateApplicationAsync. If everything is setup correctly, we will receive a successful response and our application will be visible on the AWS console.

If you want to see a fully functional example, go to the github project dotnetdeployments and clone it locally to see a working example. Creating a CodeDeploy application is the first step in using this api. We also need to create deployment groups, deployment configs and other settings. Stay tuned for the next post in this series.

Next: Create deployment groups

Categories
General

Solutions Not Problems

If you listen to the news, most of the time it’s bad news. If you listen to a co-worker, they are always complaining about other co-workers. Sometimes they complain about their own jobs. They complain about their boss. They complain about rules and regulations. We seem to be inundated with problems and complains. Before I started writing on this blog, I took days to come up with a good domain name. I use raymondsanchez.com and others domain names but they were not perfect for the content I wanted to share.

In the software world, we need business owners, business analyst, software developers, quality assurance testers, and other personnel to create solutions, mobile apps, software products. In my role as a software developer, I need to build solutions not problems. I need to contribute high quality software. I need to fix bugs in a timely manner. My point is that I have to concentrate more on the solutions than on problems.

That’s why I decided to purchase SolutionsByRaymond.com where I’ve shared articles on software development. Most of my articles are related to the .NET ecosystem and tools that have been useful in my 16 year career as a software developer.

Remember this: concentrate your efforts in solutions and not your problems.

Categories
.Net Agile General

Last Deployment of 2017


Yesterday we completed our last deployment of 2017. Web sites, WCF services, Web Api, Windows Services, and a PowerBuilder app were deployed without major issues. Our sprints run for 2 weeks and we tried to deploy at least once a month.

In this post, I want to share some of the tools we use at MDBuyline to ship high quality software.

Atlassian Jira
Our business analysts meet with our internal users on a regular basis. Sometimes these conversations take place informally during lunch or when they meet at the hall. The conversation goes something like this: “hey Bob, how are you? By the way, I wanted to show you an issue that I’m having with our website.” During these conversations, our analysts will gather enough information to create a user story. Bob will login to Jira and provide a brief description of the issue or feature request. The more information they provide the better. Adding visual aids also helps developers understand these stories better.

These stories will go to a backlog. Every other week we (developers and BA) will meet to go over these stories and also to set priorities for our next sprint. Currently we are in Sprint 47 and it will end next Thursday. Our sprints are 2 weeks long. We also have a scrum master that helps the team manage stories, priorities and deadlines.

Continuous Integration

We currently use subversion for our source control but will be migrating to git soon. Before we start working on a new story, we create a new branch based on the latest change of trunk (master in git). To continuous build our projects, we use Jenkins to handle our .NET builds and also our deployments. All of our projects have a Jenkins job that is set to build against the trunk version. The hassle with working with branches is that we have to ask our team lead to add a new Jenkins job and point it to our new branch. Hopefully with out Git migration, these setup will be handle automatically.

Continuous Deployment

After Jenkins completes building our projects, we use different PowerShell scripts to deploy web sites and web services to the right environment. We have 3 environments: dev, test, and prod. Let me give you an example so this becomes clear. Let’s say that we have a .NET website and it needs to be setup with Jenkins. Our team lead will add a new Jenkins job and set it up with subversion authentication, msbuild arguments, and PowerShell scripts. This job is build against the trunk version. Since we have to create a branch, our team lead will create a new Jenkins job based on the job that points to trunk. After creating the new job, the team lead will modify the url to point to the branch instead of the trunk url. At this point, we should be able to trigger a build and create artifacts. To move these artifacts from our Jenkins server to the correct environment, we use PowerShell scripts to copy files from Jenkins to server dev-win for example.

Summary

This year is coming to an end. For me personally, it has been a great year adding new features to our software products. Our production environment is in great shape since we ship high quality code. Jira helps us create sprints with the right items to work on. Jenkins helps us with our continuous integration. PowerShell scripts deploy our software efficiently and in a timely manner.

 

Categories
General

How to be an Effective Support Engineer

effective support engineer

For the last 2 months I’ve been in charge of our production support. If there is a bug or something needs investigation, a ticket is created to start this work. We have websites, web services and windows services to manage all these products. We are a .NET shop but also run linux servers. To be an effective support engineer, you have to familiar with your codebase. Some will say that you have to be an expert. The more familiar you are with your code, the easier it becomes to resolve issues.

How to become a project expert?

It is going to be very difficult to become an expert without having spent enough time on a given project. One way to become a project expert is to be familiar with the code. You have to be familiar with the architecture and how the project is structured. First step is to add new features to the project. By adding new features, you will understand the project better than if you start by working on bugs. Pair programming is another way to gain project knowledge. Ask other developers for help. During this knowledge transfer session, take notes and share it with other developers. The next time you receive a bug or ticket on this project, you will be in a better position to resolve it in a timely manner.

Take notes

On many occasions, I receive the same ticket within months and I had to spend hours doing the same research. This is because I didn’t take any notes when I received the original ticket. That’s why it is very important to take notes. When I start my research, I take notes as I look in the code or sql query. Sometimes I use notepad++ to take simple notes like text. For more detailed notes, I use Microsoft Word and take pictures. These pictures get annotated so we can focus on a specific area. No matter what program you use, take notes and share them with your team.

Transfer Knowledge

Just imagine keeping this knowledge to yourself. The internet was invented to share documents with other people. The same concept applies to support engineers. We have to share this information with other developers. Sharepoint or Confluence are tools that help software teams share important information. During pair programming sessions, take the time to share project architecture and any potential areas where the software can be made better.

When you take the role of a support engineer, you have to be familiar with the project. As you work on bugs, take notes so this knowledge gets share with other developers. Wikis are essential to transfer knowledge. Having a central place where information is shared will pay off dividends in the long run.

Categories
.Net AWS C# CI Github

3 Things I learned During Hacktoberfest

Hacktoberfest is a month long initiative to promote collaboration using Github. DigitalOcean created this event a few years back. This year I decided to participate in this event. In this post I want to share the 3 things I learned during Hacktoberfest.

Learn something new

During our busy schedules at work, we’re focused on maintaining existing products. In many occasions, these products are using old technology. We have a very small web site running ASP.NET MVC 2. MVC 2 was released on March 2010 so this web site is using old technology that is 7 years old. We tried to upgrade this website to run a more recent version but we ran into migration issues and the effort was abandoned. Since this site is not a critical product in our company, we decided not to spent more time on it.

With the recent release of .NET Core 2, it’s very important that .NET developers stay on top of these changes. Last month I created a new project in github called  DotNetDeployments. This project was created to automate .NET deployments. No more copy and paste files between servers. In order for me to learn something new, I decided to base this project on .NET Core 2. Core 2 was released on August 2017 and there are major changes in relationship to previous versions. In addition to learning .NET Core 2, I also learned DynamoDB high level operations using the AWS SDK.

Solve your own problems

Before Hacktoberfest took place, I started brainstorming ideas for a new project. I wrote down some ideas but I was not happy with those projects. I wanted to solve bigger problems. I’ve worked in different industries and companies and there is always areas to improve. In my current position, we are using Jenkins for our continuous integration server and powershell scripts to deploy our applications. With this setup, we are able to deploy 95% of our projects. The other 5% are deployed by copy and paste. It is not fun. So I decided to create a new project to solve this problem. DotNetDeployments will handle our deployments using AWS CodeDeploy and powershell will be use to create IIS sites, and create Windows Services. The beauty of this project is that it can handle on-premises servers and also AWS EC2 instances. Since this is an open source project, I’m expecting the community to get involved and make this project even better.

People are willing to help

After creating DotNetDeployments, I created github issues to keep track of all things I wanted to accomplish. I added “hacktoberfest” and “help wanted” labels to my issues so I can communicate with the community that I needed help. It didn’t take long and I was receiving small pull requests. I was so excited that developers were willing to help a new project. I reviewed the code and was able to accept those pull requests. After the first pull requests, I decided to add AppVeyor to handle my automated builds. AppVeyor is really easy to use and their documentation is awesome. Now with CI in place, I created more issues to handle unit tests, and also to rearrange the folder structure. I received more pull requests and was happy to review and accept them. Some of these changes broke the build but I merged those changes since I had a different issue to update AppVeyor config file. These changes were necessary because our folder structure changed. I just want to thank all the contributors that are taking the time to make this project better. We’re not done yet but during Hacktoberfest we made a lot of progress.

In summary, Hacktoberfest was a very successful initiative by DigitalOcean and GitHub. During this month, I was able to learn new technologies and solve real problems that developers face every day. DotNetDeployments could not be possible without the help of the community. Thanks to all contributors.