t2.small CPU Credit Drop

AWS T2 Instance CPU Credits

I recently quoted out a estimated AWS environment using t2 instances. I recommended for development t2.nano servers due to the single developer only working eight hours a day. I then recommended t2.medium instances for a soft production launch. Total user base will be around 30 users, so we were after redundancy at a low cost. I was asked “Why not use t2.small or t2.nano for production?”. I knew it was against best practices but I did not have any data to show why until I installed TeamCity on a t2.small sever for internal use.

AWS Cloudwatch CPU Usage

Above is the CPU graph for the t2.small instance that runs the head node for TeamCity. By default, AWS servers start in UTC time, also a good best practice if your applications support it. Windows Update Active Hours

By default, Windows, and Windows Server 2016 sets the active hours from 8AM to 5PM local time to the server.

5PM UTC time is 12 PM EST, which will kick off Windows Update during your lunch hour. In our case, it dropped all of our CPU credits before I could shut off windows updates. t2.small CPU Credit Drop

Each t2 instance has a baseline of vCPU, t2.Small is 20%, but the vCPU is not the best with intensive applications. Thankfully now we have Windows Update set correctly for business hours in the East Coast and our TeamCity server is now running fine. Lesson learned with t2 instance, be careful with any automatic process.

 

How to solve Elasticsearch Unassigned Shards on Reboot

We have had our ElasticSearch cluster in production for a few weeks. Checking on drive space we decided we wanted to increase it since it was sitting at 67, 88 and 95 for each of our three nodes. The sysadmin added the space, and rebooted the box (ok, this was the first mistake, you can do this online now days). When it rebooted, horror set in, all three shards were unassigned and appeared gone. Cluster status, yellow.

Elasticsearch Unallocated Shards

Quick online search lead to a shard allocation setting.

Elasticsearch Shard Allocation

Tried it, nothing. I would just use the same image above, but it is a waste of pixels.

Then, I hit pay dirt, found a way to ask ElasticSearch, or explain itself, to you on why a shard for an index didn’t exist. You can pass in the node or shard numbers to it, this is a great API.

Elasticsearch Allocation Explain

Error, drive space watermark “NO”. Bingo! Alright, let us fix that drive now.

Drive space added, shards are back online. Now the thing to remember is everything was fine, but it appears with ElasticSearch 1.6 that if you hit the water mark, no matter what happens, you have to go back under it before that node will come fully online. It was processing data, serving up content and overall working, just the shards were missing. Migrating to ElasticSearch 2.3 soon, will find out if it has the same watermark very soon.

 

 

ElasticSearch Online ReIndex With Writes C#

We just now deployed a application solely using ElasticSearch (Elastic.co) as our backend full-text search provider. We migrated away from SQL Full Text search for reasons I will post later. ElasticSearch runs on top of Lucene, but adds a REST api and cluster goodness on top.

Quick setup,
You need Java, and the Java Runtime(JRE) in your path. I cheat and update the elasticsearch batch files to point at a download.
You need to run elasticsearch. On windows/linux, you just run it from the command line. Later you can register it as a background service.
Install a management plugin. I use a plugin from mobz “plugin install mobz/elasticsearch-head” in the elasticsearch bin folder should do the trick.
Port is 9200 and the plugin’s run on the “_plugin” folder. If you get it up and running you should get this screen minus the indexes.

ElasticSearch-Head-Home

Terms
Index==Table in the SQL terms
Mappings==I want to tell you what this field means and how to store it. In SQL, we just have data types, in ElasticSearch you have much more control than “This is a string”.
Alias==A view in SQL terms, this is important later.
Replica==Backup copy
Shard==A way to split one really large index into smaller indexes. Keep in mind you need to know details of how to update a document if you shard.

The first problem you run into with ElasticSearch is “How do I change my mappings?”. Easy, you make a new index and point everything to it. In SQL Server, SSMS will do this for you by creating a new table, moving the data, dropping the old table, renaming the current table. Elasticsearch doesn’t have an official client, so you have to code this yourself. Second problem is how to handle this while the data is live. ElasticSearch doesn’t have the concept of “locks” in the SQL server sense. Plus relying on locking data to perform an update is so 2004 of an application. Large ElasticSearch instances contains billions of documents, hard to lock or file copy all of them.

Source Code C# (If you want to cut to the chase)
https://github.com/joshbartley/ElasticSearch_ReIndex

ElasticSearch-Single-Index
You always start with one index.

You get some data into ElasticSearch, you find out your mapping is wrong and you need to change it. Four records is easy, four million would be harder. First step you should verison your indexes, dates, numbers, random word generator, doesn’t matter. Whatever makes the most sense to you to understand.

You create your new secondary index, and add in a lowercase mapping, this is superfluous to this post, just an example.

ElasticSearch-Second-Index-ReIndex

If you move all the data now, you may miss the records that were being put into “datav1” during the Re-Index. Here is where the aliases come into play. You can create them however you like, I named them *_R(ead) and *_R(ead)W(rite). You cannot write to an alias with more than one index.

ElasticSearch-Read-Write-Alias

The “data_rw” alias is only to let your application know where to write the data. You have to code your application to accept the alias, grab the indexes, and write to both. In this example I am not using the bulk query as I am only dealing with two records on each operation. Anyone want to PR a change to the C# Client?

[csharp]
private static void WriteSecondaryObjects(ElasticClient client)
{
var indexes = client.GetAlias(x => x.Name("data_rw"));

foreach (var index in indexes.Indices)
{
client.Index(new Company() { Name = "Mega Acme Corp" }, idx => idx.Index(index.Key));
client.Index(new Company() { Name = "Global World Domination Corp Acme LLC" }, idx => idx.Index(index.Key));
}
}
[/csharp]

After I wrote two records, the document count goes from 2 to 4 on the first index, and 0 to 2 on the second. Notice the “data_r” alias is still pointed at the old index, this is because the new index doesn’t have all the data yet.

chrome_2016-04-05_21-23-44

The following code I grabbed from a StackOverflow answer and updated it to the latest NEST (C# Client) for ElasticSearch. http://stackoverflow.com/a/34867857/32963

[csharp]
public static void Reindex(ElasticClient client, string aliasName, string currentIndexName, string nextIndexName)
{
Console.WriteLine("Reindexing documents to new index…");
var searchResult = client.Search<object>(s => s.Index(currentIndexName).AllTypes().From(0).Size(100).Query(q => q.MatchAll()).SearchType(Elasticsearch.Net.SearchType.Scan).Scroll("2m"));
if (searchResult.Total <= 0)
{
Console.WriteLine("Existing index has no documents, nothing to reindex.");
}
else
{
var page = 0;
IBulkResponse bulkResponse = null;
do
{
var result = searchResult;
searchResult = client.Scroll<object>(new Time("2m"), result.ScrollId);
if (searchResult.Documents != null && searchResult.Documents.Any())
{
ThrowOnError(searchResult,"reindex scroll " + page);
bulkResponse = (IBulkResponse)ThrowOnError(client.Bulk(b =>
{
foreach (var hit in searchResult.Hits)
{
b.Index<object>(bi => bi.Document(hit.Source).Type(hit.Type).Index(nextIndexName).Id(hit.Id));
}

return b;
}),"reindex page " + page);
Console.WriteLine("Reindexing progress: " + (page + 1) * 100);
}

++page;
}
while (searchResult.IsValid && bulkResponse != null && bulkResponse.IsValid && searchResult.Documents != null && searchResult.Documents.Any());
Console.WriteLine("Reindexing complete!");
}

Console.WriteLine("Updating alias to point to new index…");
client.Alias(a => a
.Add(aa => aa.Alias(aliasName).Index(nextIndexName))
.Remove(aa => aa.Alias(aliasName).Index(currentIndexName)));
}

[/csharp]

The NEST ReIndex function creates a new index and requires a mapping at create time, that is not what we need. Once it runs, it will grab all the old data. Now if you had data being pumped into ElasticSearch it should be writing it to both indexes.

elasticsearch-reindex

At this point, you would use some feature toggles/flags to test out the new index’s mappings. If all is well, switch over the read alias.

elasticsearch-reindex-complete

DONE!

If you made it this far and want to know what the rollback plan is, well we didn’t delete anything. So just swap the alias back over. Ideally if you used the feature toggle/flag, you would have reduced your risk for the rollback. We are testing this out on Thursday, so we will see how it works with half a million records 🙂

 

How to Diagnose a Test Failure on TeamCity

Have you ever had that unit test that works locally, but fails on team city. Turns out you can debug them and find out that it isn’t team city’s fault.

Failing Team City Builds

You run the test locally, pass, run them on team city, fail. Alright, time to get serious, 1, pull the TeamCity artifact down locally, 2, run MSTest from command line like TeamCity does, 3, start debugging. Most unit test frameworks throw exceptions when a test fails, great way to debug into the test.

Make sure you have an artifact from TeamCity or you can’t pull down the test. Well, unless you are an admin on the box.

TeamCity Test Artifact

TeamCity Download Artifact

Visual Studio Attach to Process

Open up Visual Studio Command Prompt for VS2015.

[code]
mstest /testcontainer:MyTests.dll
[/code]

Next, attach to the MSTest process QUICKLY. MSTest launches QTAgent32.exe for running the actual unit tests but it won’t stop to wait for you.

Attach To Process QTAgent32.exe

In the end, it turned out another test ran first, and our second tests couldn’t set a private static readonly because of its if null check.

[code langauge=”csharp”]
private static readonly List<int> Codes = new List<int>();
[/code]

Benchmarking MVC6 Beta 8

I have always been curious how fast MVC can be, and a previous article (http://blog.bitdiff.com/2012/06/performance-comparison-iis-75-and-iis-8.html) detailed the Web API  speeds with IIS 7.5 to IIS 8 along with self-hosted options. So I spun up a MVC 6 web  app with Beta-8 and used the Apache BenchMark to test it out. Running a i7-3740QM CPU with Windows 7.

My controller contained only this,

[csharp]

[Route("api/[controller]")]
public class ValuesController : Controller
{
// GET: api/values
[HttpGet]
public IEnumerable<string> Get()
{
return new string[] { "value1", "value2" };
}
}

[/csharp]

I fired up Apache Benchmark using the command below.

[code]ab -n 1000 -c 2 http://localhost:5000/api/Values [/code]

3.064MS! It beats out the 17.818 from Windows 8 Self Host and I am running Windows 7! This is a localhost call, so adding in SSL or a network would likely increase this. but it helps if it starts really low first.

The first “Time per request” is for each request * concurrent request number. The second “Time per request” is the actual individual requests. Wonderfully explained at this ServerFault question http://serverfault.com/questions/274252/apache-ab-please-explain-the-output

mvc6-beta8-batch1

For giggles, I pumped it up to 10,000 concurrent requests, I also tried 5,000 concurrent requests and got the same ~7ms response time.

mvc6-beta8-crazyconcurrent

So basically MVC6 is FAST! VS2015 diagnostic session for those who wanted to take a peek. I might have had around 100 chrome tabs open when running this benchmark.

 

 

vs2015 diagnostic session

 

Feature Flags Patterns and Applications

The purpose of a feature flag, or known as a feature toggle, is to add code to an existing system without modifying its current behavior. If a feature flag is added to a system and the system breaks while the feature flag is turned off, it was not done properly. The end goal of a feature flag is to provide a low risk way to introduce new code to a production environment to let select users test the new feature, and for the feature to be easily deployed. The term mainstream feature will apply to a feature flag that is now turned on for all users. A feature that has the possibility of being turned off again is not considered a mainstream feature at that point. A more in depth look can be found on Martin Fowler’s site Martin Fowler’s Feature Toggle

Style sheet change Pattern (MVC)

In the MVC world, the controller is responsible for gathering and presenting a model to the view. This should include the results of any feature flags. In ASP.Net MVC, you can modify the model, or you can use a ViewBag, it is up to personal preference as both are acceptable. For CSS that is embedded in static files, you may need to create a new static file and use the feature flag to determine if you want to add the stylessheet as a page resource. Do not attempt to bundle the new style sheet, as if you have more than one feature in the system, you will have too many combinations of bundled and minified code to make this happen. Future web technologies, like WebRTC will make additional files no longer be a extra network request.

Controller Code

[code language=”csharp”]
public class HomeController : Controller
{
public ActionResult StyleSheet()
{
var feature = new US1234StyleSheetChange(HttpContext.User.Identity);

ViewBag.US1234StyleSheetChange = feature.Enabled();

return View();
}
}
[/code]

View File
[code language=”csharp”]
@section stylesheets{
<style>
@if (ViewBag.US1234StyleSheetChange)
{
<text>
.ok-button{
background-color: green;
weight:bold;
}
</text>
}
else
{
<text>
background-color: yellow;
color: red;
</text>
}
</style>
}

@if (ViewBag.US1234StyleSheetChange)
{
@*we couldn’t make all the changes with style sheets alone*@
<button id="go" class="ok-button confirm">Ok</button>
}
else
{
<button id="go" class="ok-button">Ok</button>
}
[/code]

Javascript Pattern

In the previous Stylesheet Pattern, we saw that it was easy to add to the view any changes to the stylesheet. In the below example, you can see we wrap the entire jQuery event around the same feature flag variable. In the case of minified or bundled javascript, it would be best to leave it out of the bundled files until it makes it as a mainstream feature.

[code language=”javascript”]
<script>
$(function () {
@if (ViewBag.US1234StyleSheetChange)
{
<text>
$(".confirm").click(function () {
var userAction = confirm("Are you sure you want to continue");
});
</text>
}

});
</script>
[/code]

Code only feature change

In the case of a code only change wrapped in a feature flag, a simple if statement suffices in most cases.
[code language=”csharp”]
public class HomeController : Controller
{
public ActionResult Index()
{
var feature = new US1234TitleChange(HttpContext.User.Identity);

if (feature.Enabled())
{
ViewBag.Title = "Home Page Changed for US1234";
return View();
}

ViewBag.Title = "Home Page";
return View();
}
}
[/code]
In extreme cases where code clarity may be results in a more unmanageable feature, you can break the feature into a private function and split the code path at the beginning of the function as shown below.
[code language=”csharp”]
public class HomeController : Controller
{
public ActionResult Index()
{
var feature = new US1234TitleChange(HttpContext.User.Identity);

if (feature.Enabled())
{
return US1234TitleChange();
}

//Additional changes made here should be made in the US1234TitleChange
//until it is a mainstream feature
ViewBag.Title = "Home Page";
return View();
}
/// <summary>
/// Please replace Index() with this function once the feature is mainstream
/// </summary>
/// <returns></returns>
private ActionResult US1234TitleChange()
{
ViewBag.Title = "Home Page Changed for US1234";
return View();
}
}
[/code]

SQL – Stored procedure pattern (Selects)

Some frameworks do not do well with additional columns returning. If you have one of those, you can apply the below pattern to make sure the new column does not come back from the select statement.

[code language=”sql”]
CREATE PROCEDURE [dbo].[usp_GetSalesVolume]
@param1 int = 0,
@param2 int
AS

if (Exists(Select 1 from FeatureFlags where id=4 and active=1))
begin
select ‘new data elements here’, @param1, @param2
return
end

SELECT @param1, @param2

RETURN 0
[/code]

Sql stored procedure (Updates)

In the case where logic changes for an update and the change has to occur in the stored procedure. You can add a new parameter that MUST have a default value set. Then break the code path into two and return or use an else statement. In this case a new table has been added, one the feature is a mainstream feature, the old table should be deleted along with the previous code.

[code language=”sql”]
CREATE PROCEDURE [dbo].[usp_UpdateSalesVolume]
@param1 int = 0,
@param2 int,
@param3 int =0 /*added per feature Goal Column */
AS
if (Exists(Select 1 from FeatureFlags where id=4 and active=1))
begin
update Goalsv2 Set [email protected], [email protected] where [email protected]
return
end
update Goals Set [email protected] where [email protected]
RETURN 0
[/code]

Bar Build

Bar Build – Part 1

There comes a point where you have enough parties and your friends leave enough drinks at your place that you must build a bar. Oh, this is easy, well, maybe not.

Step 1. Decide if you want a fridge in your bar or not. There are two types of fridges, expensive vent out the front commercial fridge, or the clear glass front fridges you have to leave two inches of clearance around. Go with the second option. We bought a nice Frigidare mini fridge with a glass front that fits in very well.

Step 2. Buy base cabinets from a local home improvement store (you can build your own). Base cabinets are what your entire kitchen is made of and are different from wall cabinets because they are deeper and have a toe kick. They usually have built-in supports to hold a counter top. Once you have the fridge, add in the four inches (two on each side) to properly vent the fridge, or what is recommended in the fridge manual. Also make sure your bar is not too long, you are not opening a restaurant here.

Step 3. Build a 2×4 wall, including the studs being 16″ on center to help screw in the cabinets. Since base cabinets are mounted to a wall in a kitchen to keep them from coming apart or moving, you should do the same. This also gives you an area to put the actual bar top where you put your food or drinks. The base cabinets are around 34″ tall, with a countertop. No one wants to lean up on a kitchen counter with their elbows, it feels horrible. Normal bar height is 42″ excluding the bar moulding. REMEMBER, 2×4″ are 1 1/2 ” x 3 1/2″, don’t mess up and be off by 1/2″ in your calculations. You will have 1 1/2″ for the bottom 2×4, and another 1 1/2″ for the top 2×4. Most bars use two layers of 3/4 plywood, so subtract 1 1/2″ (3/4″ X 2) for the plywood for the 42″ height.

Bar Wall

Screwing the cabinets to the base of the wall. Use shims so you don’t pull the plywood apart.

IMG_20140825_194842Cabinets screwed to 2x4 Wall

Step 4. Add in some paneling. No one wants to see those 2×4 or the back of those cabinets. Spend the extra few dollars and get some nice sanded or baltic birtch plywood. 1/4″ will work just fine if put the 2×4″ 16″ on center unless someone kicks your bar. If someone kicks your bar, you would have issues with 3/4 plywood too, you just spent more.

IMG_20140830_185906
Construction Adhesive to attach a panel to hide the 2×4 wall.

 

1/8 plywood on cabinet end
Apply a sheet of 1/8 plywood, then trip to the cabinet.
Bar Build
Same for all sides, micro-pin nailer so I didn’t have to clamp it.
Fluted molding, cut on 45 and on top of a 3" trip piece.
Fluted molding, cut on 45 and on top of a 3″ trip piece.
IMG_20140919_181959
Fluted molding, dry fit.

 

Added more trim with glue and a pin nailer, corbels to better support weight on the bar.
Added more trim with glue and a pin nailer, corbels to better support weight on the bar.

 

Next up, the countertop, paint and bar moulding.

WCF Custom tool warning “Cannot import wsdl:portType”

Yet another odd problem to deal with today. “Custom tool warning: cannot import wsdl:portType Could not load file or assembly”Capture

Thanks to Nuget, it appears that it tries to load all assemblies when “Reuse types in all referenced assemblies” is marked.

Capture

The WSDL tool doesn’t use the below setting so it fails.

[code langauge=”xml”]
<runtime>
<assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
<dependentAssembly>
<assemblyIdentity name="RabbitMQ.Client" publicKeyToken="89e7d7c5feba84ce" culture="neutral" />
<bindingRedirect oldVersion="0.0.0.0-3.4.3.0" newVersion="3.4.3.0" />
</dependentAssembly>
</assemblyBinding>
</runtime>
[/code]

Un-check, deselect, “Reused types in referenced assemblies” and your problems will go away, just don’t check the offending assembly again 🙂 If you have to use that assembly, for your project file to the correct version.

SSMS – Application Intent ReadOnly

If you have an always on MSSQL availability group you can add
applicationintent=readonly;

to your connection string to connect as a read only user on the secondary mirror of your availability group. If you want to connect using Microsoft Sql Server Management Studio (SSMS) using the same intent, open a new window, hit connect, type in your secondary DB instance.

SSMS_CONNECT

Click “Options” and then “Additional Connection Parameters” and viola! You can now run read only queries without taxing your production SQL Sever instance!

SSMS_applicationintent_readonly

Unit Testing Web API Controllers with Headers

I couldn’t find a good example of how to unit test a controller with

[code langauge=”csharp”] var headerValue = Request.Headers.GetValues("HEADER");[/code]

in the controller. Not too common, but happens enough it can be expected. Also the controller inheirts “ApiController” which doesn’t inheirt from “ControllerBase” so the usual controller context call won’t work.

Here is how I coded the unit test

[code language=”csharp”]
var config = new HttpConfiguration();
//add in the actual route or method if your are looking at it in your code
var request = new HttpRequestMessage(HttpMethod.Post, "http://localhost/random");
//DefaultApi maps to your web api route setup
var route = config.Routes.MapHttpRoute("DefaultApi", "api/{controller}/{id}");
//add in the extra data, similar to when you call @Url.Route
var routeData = new HttpRouteData(route, new HttpRouteValueDictionary { { "controller", "products" } });
var controller = new MyProject.Controllers.Api.ExampleController();
controller.ControllerContext = new HttpControllerContext(config, routeData, request);
controller.Request = request;
controller.Request.Headers.Add("X-Your-Header","HeaderValue");
[/code]

In the controller, do a

[code langauge=”csharp”] var headerValue = Request.Headers.GetValues("HEADER");[/code]

do not do

[code langauge=”csharp”]
//DON’T DO THIS
var headerValue = HttpContext.Current.Request.Headers.GetValues("HEADER");[/code]

This will hit the old IIS style objects and your unit test will fail.

If you don’t have Resharper, you might need these using statements.

[code langauge=”csharp”]

using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Web.Http;
using System.Web.Http.Controllers;
using System.Web.Http.Routing;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using Moq;

[/code]