At DrDoctor there is a very strong belief that the business should be Lean (note the capital 'L'), so we analyse data on how our services are used in order to guide product development. An important aspect of that is that we need to get features out there in order to see how they're used, which fits neatly with a Scrum-based, Agile approach. We do favour working software over comprehensive documentation and we aim to deliver that working software every sprint (i.e. every two weeks). Worse than that, following lean principles, we don't just value responding to change over following a plan we acknowledging that a core strength is our ability to change our plans. Yes there is a vision, but no, I don't pretend to know for certain what I'll be working on in two months' time.
What does this mean in practice? At a very high level, it means that we need to be able to release features easily. Or put another way, having a 'release sprint' (which I've seen in lots of places) means we've failed. We haven't always succeeded in this, but we're getting there and we are now at the stage where we have decent CI and automated deployment, we can ...
Push changes to our central code repository
Which triggers a build on our build server
Which runs our test suite (unit tests, integration tests that exercise the DB and bigger 'Spec' tests that lean on even more infrastructure)
We're notified if the build passes or fails
With 5 button clicks we can deploy the build to our 'smoke test' environment where we can do a full end-to-end test - our product owners get to run through the feature they've asked for
5 more button clicks and we've deployed to production
However, the main point isn't about the tools, it's about the mindset. Lots of companies talk about Lean and Agile but don't deliver (if you've ever sat through a 30 minute "stand-up", you know what I'm talking about), but at DrDoctor the value of delivering software rapidly is understood, which translates into investing time into making delivery easy. There is a lot more to do, but the whole company agrees that we don't say a story is delivered unless the code is running somewhere.
I realised that in my previous post I forgot to mention log4net on the list of OSS that we use at DrDoctor. This is probably because it's like Léon - does the job quietly, professionally and you barely notice it's there. Until you it's really important that it's there.
So yeah, just a bit of love for log4net because it got overlooked because we don't have to think about it.
At DrDoctor we're on a .NET stack (and we're Bizspark members) but that doesn't mean we don't love open source tools - what we want is the right tools - and the .NET world is full of them. We've used a few and I thought it might be useful for people to see what choices we've made. So here goes:
1) RabbitMQ. Correct, it's not MS, but at no point did we ever consider using MSMQ. It's an awesome messaging infrastructure with a great .NET client. It works and it works well.
2) EasyNetQ Fantastic light-weight, open source Bus implementation on top of RabbitMQ. It's actively developed, well documented and we got it working in about two days. It's at a 0.x release at the moment, so breaking changes do happen, which means we've taken a cut of the code that we're happy with (I possibly managed to push a breaking change into our build before I realised this ;) ). Hopefully at some point we can contribute something back.
3) Topshelf Makes working with Windows Services a pleasure. It's used by a lot of other projects (e.g. NServiceBus) because it's great.
4) Castle Windsor Powerful IoC with a tonne of well written documentation.
5) Magnum library We only we use one thing from this library (the State Machine) but it's at the core of our business logic. Created by the same guys who brought you Topshelf (see above). Almost non-existent documentation, but really handy utilities.
6) Simple Data Mark Rendle's rather excellent data access library (we use it on top of SQL Server).
7) NancyFX Nice light-weight web framework (an alternative to MS's WebAPI and MVC frameworks).
Of those six seven tools I'd say that easyNetQ, Topshelf, Magnum, NancyFX & Simple Data are all about developer productivity. Simply put, they are tools that allow us as developers to work on business features rather than infrastructure.
RabbitMQ is a fundamental architectural choice. DrDoctor is event driven by its very nature, so a messaging infrastructure fits perfectly. It also leads to a clean separation of concerns with components that communicate via RabbitMQ in an asynchronous fashion. This in turn leads to a much easier development and deployment cycle: if we're working on our appointment management logic and need to deploy an update it's one component that needs to come down, while our entire application remains up. SOA baked in from the start.
NancyFX feels more like the Python/Rails world and doesn't carry all the kludge that comes with the .NET MVC/WebAPI frameworks. Seriously, I don't need the Entity Framework stuff. I really, really don't.
Which really just leaves Castle. There are loads of IoC containers out there, I just happen to feel most comfortable with Castle.
Update: In the time it took me to make the change to our code and write the blog post below, EasyNetQ got updated, thus rendering this post obsolete. I'll soon be doing a new version that uses the latest version.
At my current company (DrDoctor) we're using EasyNetQ as a lightweight bus implementation on top of RabbitMQ. EasyNetQ has a lot of very nice features, one of which is auto-creation of exchanges and queues. It uses the fully qualified name of the message type as its basis, but we have quite a deep hierarchy of messages, which makes sense in the tree-structure of our solution, but makes looking at the queues painful as we end up with long names.
The 'CreateBus' method has an overload that allows you to pass in your own services. When 'CreateBus' is called, if you pass in your own service (for instance IEasyNetQLogger from the example in the documention), your service gets registered first and so the default service registration doesn't happen (take a look at DefaultServiceProvider.Register for details).
In our case we wanted to alter the way queues and exchanges are named, which is slightly more complex because there isn't a separate service for each of those, instead there is a 'Conventions' service which sets these. However, that service is itself easy to poke from the outside. Here's some code:
The method 'GetNameFromType' is a static method that returns a string, based on the type passed in.
public static string GetNameFromType(Type type)
{
if (type.GetInterface("IMyMessage") == null)
throw new ArgumentException("Type must implement IMyMessage");
string category = null;
foreach (var attr in type.GetCustomAttributes(false).OfType<MessageCategoryAttribute>())
{
category = attr.Category;
}
if (category == null)
throw new ArgumentException("Implementation must be marked with MessageCategoryAttribute");
return category + "_" + type.Name;
}
This suits our needs as it means we can (and indeed have to) mark up our messages with an attribute that categorises them.
There is a bit of gotcha with this, however. We have effectively replaced the use of the EasyNetQ method 'TypeNameSerializer.Serialize' for our exchanges and message queue names, which is fine, but this method is used elsewhere that could cause a problem. There is another service that is registered
As part of the greenfield project I'm currently working on I decided that I'd use the chain of responsibility design pattern, but it took me a little while to work out how to set up Castle Windsor to support this. It's pretty simple, but the stuff I found on the interwebs all seemed to be slightly out of date, so I thought I'd explain it here.
The short version is you can use the 'DependsOn' and 'Dependency.OnComponent<T, U>' methods when registering your components to do this pretty easily. You can use an installer that looks something like this:
public class ChainOfResponsibilityInstaller : IWindsorInstaller
{
public void Install(IWindsorContainer container, IConfigurationStore store)
{
container.Register(Component.For<IChainItem>()
.ImplementedBy<ChainItemOne>()
.DependsOn(Dependency.OnComponent<IChainItem,ChainItemTwo>()),
Component.For<IChainItem>()
.ImplementedBy<ChainItemTwo>()
.DependsOn(Dependency.OnComponent<IChainItem, ChainItemThree>()),
Component.For<IChainItem>()
.ImplementedBy<ChainItemThree>());
}
}
In this post I'm going to run through how to use the SessionAuthenticationModule (SAM) (part of Windows Identity Foundation (WIF) in .Net 4.5) for authentication and authorization in a simple MVC application, replacing Forms Authentication. It won't be production code, but if after reading this you'd like to see a more complete Membership and Identity management library take a look at Brock Allen's Membership Reboot on GitHub https://github.com/brockallen/BrockAllen.MembershipReboot
The solution that goes along with this post is available at https://github.com/StephenFriend/AuthWithWIFAndSAM and it will probably be easier to look at the code as you read this. In the walkthrough below, I've used Visual Studio 2012 and there are different tagged commits, so that it's easy to see the changes that are made as we move from Forms Authentication to using the Session Authentication Module.
First, I'll set up an extremely simple solution using forms auth. If you've cloned from github, simply type git checkout -f formsAuth
The first step is to create a new MVC4 app (imaginatively called 'DemoSite' in the code), using the 'Internet Application' Project template. Next we want to amend the Web.config file at the root of the MVC site.
In the controllers folder create a Home controller using the 'Empty MVC Controller' template from the drop-down in the scaffolding options section of the dialogue. The code for the Home Controller should look like this:
[Authorize]
public class HomeController : Controller
{
[AllowAnonymous]
public ActionResult Index()
{
return View();
}
public ActionResult MyStuff()
{
return View();
}
}
Note the use of the 'Authorize' attribute on the whole class and the 'AllowAnonymous' attribute on the 'Index()' method. This should mean that anyone is able to access the Index view, but only users that are authorized can access the MyStuff view.
Next up let's create an Account controller. The code for this will look like this initially:
public ActionResult Login()
{
return View();
}
[HttpPost]
[ValidateAntiForgeryToken]
public ActionResult Login(LoginInput model)
{
if (ModelState.IsValid && LoginDetailsAreValid(model))
{
FormsAuthentication.SetAuthCookie(model.Email, false);
return RedirectToAction("MyStuff", "Home");
}
ModelState.AddModelError("", "The user name or password provided is incorrect.");
return View(model);
}
// In the real world this would be likely be a call to the database to get the user details, with a validation
// that the user exists and that their password is correct. See WebMatrix.WebData.SimpleMembershipProvider.ValidateUser()
// for an example (download the ASP.Net web stack here: http://aspnetwebstack.codeplex.com/)
private bool LoginDetailsAreValid(LoginInput loginDetails)
{
return (string.Compare(loginDetails.Email, "you@example.com",
StringComparison.InvariantCultureIgnoreCase) == 0 &&
string.Compare(loginDetails.Password, "password", StringComparison.Ordinal) == 0);
}
In order for this to compile you'll also need to create a LoginInput class (in the Models folder), which is a simple View Model that we'll use to create the login view and user input. It should look like this:
public class LoginInput
{
[Required]
public string Email { get; set; }
[Required]
[DataType(DataType.Password)]
public string Password { get; set; }
}
The end point of this first stage is to create the views that go with the various controller actions. Home/Index looks like this:
@if (User.Identity.IsAuthenticated)
{
You are already logged in as @User.Identity.Name
}
else
{
using (Html.BeginForm())
{
@Html.ValidationSummary()
}
}
and Home/MyStuff looks like this:
@{
ViewBag.Title = "MyStuff";
}
Congratulations, you're authenticated
If you start up the website now, you should be able to log in (as you@example.com, password = 'password') and if you check your cookies, you will see that a cookie called .ASPXAUTH has been created. If you try to access Home/MyStuff without logging in you will be automatically redirected back to Account/Login
Now, we're going to stop using Forms Authentication, initially not replacing it with anything. (Use git checkout -f noAuth to see the code in the repo).
To start, go to the web.config file and change the authentication node from:
to:
This means that the FormsAuthenticationProvider module will no longer be loaded at application start-up. Once you've done this, you'll see that if you log on you'll be presented with a 401 error when you are redirected to 'MyStuff'. If, however, you check your cookies, you'll see that the call to FormsAuthentication.SetAuthCookie in the AccountController has set a cookie, but there is no provider specified so the 401 error is displayed on actions that have an 'Authorize' attribute.
In order to use the SAM module you will need to alter your web.config so that includes the following elements in the <config sections> element:
The first of these is to "configure a service or application to use Windows Identity Foundation" [1], the second names the configuration section for federation configuration, which you'll need to add to the config file:
The default configuration requires SSL, so you will need this section if you don't want to sort out setting up SSL for this example.
Under <system.webServer><modules> you will need to add the following so that the SessionAuthenticationModule is added to the ASP.Net pipeline.:
In order to have access to the classes you have just configured, you will also need to add references to System.IdentityModel and System.IdentityModel.Services to your project.
Finally, you need to alter the Account controller to look like this:
public class AccountController : Controller
{
public ActionResult Login()
{
return View();
}
[HttpPost]
public ActionResult Login(LoginInput model)
{
if (ModelState.IsValid && LoginDetailsAreValid(model))
{
WriteAuthCookie(model.Email);
return RedirectToAction("MyStuff", "Home");
}
ModelState.AddModelError("", "The user name or password provided is incorrect.");
return View(model);
}
// In the real world this would be most likely be a call to a database to get the user details, with validation
// that the user exists and that their password is correct. See WebMatrix.WebData.SimpleMembershipProvider.ValidateUser()
// for an example (download the ASP.Net web stack here: http://aspnetwebstack.codeplex.com/)
private bool LoginDetailsAreValid(LoginInput loginDetails)
{
return (string.Compare(loginDetails.Email, "you@example.com",
StringComparison.InvariantCultureIgnoreCase) == 0 &&
string.Compare(loginDetails.Password, "password", StringComparison.Ordinal) == 0);
}
private void WriteAuthCookie(string userEmail)
{
var claims = new List();
claims.Insert(0, new Claim(ClaimTypes.Name, userEmail));
var claimsId = new ClaimsIdentity(claims, "Password");
var cp = new ClaimsPrincipal(claimsId);
var sam = FederatedAuthentication.SessionAuthenticationModule;
var token = new SessionSecurityToken(cp);
sam.WriteSessionTokenToCookie(token);
}
}
As you can see the new WriteAuthCookie method deals with persisting the authentication details to a cookie. In it we create a new Name claim (the only claim we use in this example), that is then used to create a ClaimsIdentity, which in turn is used to create a new ClaimsPrinciple that is then persisted to a cookie.
Once you have logged in, if you check your cookies, you should see that you have a new 'FedAuth' cookie, which is what the SAM module uses. If you then go back to Account/Login you should see that the claim of 'Name' is used for the User.Identity.Name property that is displayed on that page if you're logged in.
which implies that this is standard practice for the dev that wrote it, rather than the response to a problem. It's conceivable that this person has done a lot of performance tests and could demonstrate that all the calls to GC.Collect are justified. Or more formally, it's a logical possibility, like the sky being green or everyone from Birmingham having six fingers on their right hands.
Unfortunately because the code is proprietary I can't share the full horror and give the performance metrics to demonstrate how truly awful this is - for instance, the application that contains the top snippet calls GC.Collect() at least four times for every iteration of the while loop. However what I can do is examine this code, look at why you might do something like this and try to show that the code tells you that the person who wrote it doesn't really understand it.
There are many sources that tell devs to not call GC.Collect() [1, 2], however there are some circumstances when it might be the right thing to do. According to Microsoft you should "Use the Collect method when there is a significant reduction in the amount of memory being used at a defined point in your application's code."[3] Rico Mariani also offers this piece of advice for when you might call GC.Collect, "if some non-recurring event has just happened and this event is highly likely to have caused a lot of old objects to die."[4]
At first glance this code doesn't seem to fit that last piece of advice, since it sits in a while loop, however the method that is used for the loop termination criterion (MainApplicationWork()) effectively returns false when there are no more records for the application to process. Those records do not arrive in a regular fashion, so it is arguable that this is an event that occurs repeatedly, but is irregular (perhaps akin to the canonical example of a user closing a windows form - it might happen multiple times, but it's not a predictable event and so the GC algorithm self-tuning will not work).
When the call to GC.Collect() is made it "Forces an immediate garbage collection of all generations."[5] Part of this process involves moving pointers from the finalization queue to the freachable queue for all objects with Finalizer methods that have been found to be garbage (i.e. are unreachable). Interestingly this resurrects the objects you are trying to get rid of because there is now a reachable reference to them (on the freachable queue, which is considered a root) and they will only become unreachable again once their finalizers have been run by the Finalization thread (which runs automatically whenever there are objects in the freachable queue).
Because of this it is recommended[6] to call GC.WaitForPendingFinalizers after GC.Collect as this forces the current thread to wait until all the finalizers in the objects you are trying to clear up have been executed and thus avoids the objects getting promoted to Gen1. Given that we've just resurrected some objects, it is recommended to call GC.Collect again to finally release the memory of all the objects that have just been finalized. So you should end up with:
But in this codebase there is a further call to GC.WaitForPendingFinalizers which does, ummmm, precisely nothing since the only compaction/garbage collection that takes place as part of the second call to GC.Collect() is of the objects on the heap that have just been finalized. Nothing else has changed in the application in the meantime since both GC.Collect() and GC.WaitForPendingFinalizers() block the current thread.
Does it do any harm? Probably not (although that requires some actual performance metrics to be confident on), but what it does shout is that this snippet is a piece of cargo-cult programming, repeated through a codebase, no doubt with the best of intentions, but almost certainly for no good reason and very likely damaging application performance.
* more or less - the application method names have been changed to protect the innocent
1. http://blogs.msdn.com/b/ricom/archive/2003/12/02/40780.aspx - Rico Mariani "Two things to avoid for better memory usage"
2. http://msdn.microsoft.com/en-us/library/ff647790.aspx - MS Patterns & Practices "Improving Managed Code Performance"
3. http://msdn.microsoft.com/en-us/library/bb384155.aspx - MSDN "Induced Collections"
4. http://blogs.msdn.com/b/ricom/archive/2004/11/29/271829.aspx - Rico Mariani "When to call GC.Collect()"
5. http://msdn.microsoft.com/en-us/library/xe0c2357.aspx - MSDN Library GC.Collect definition
6. http://msdn.microsoft.com/en-us/library/ff647790.aspx - MS Patterns & Practices "Improving Managed Code Performance"
