CacheRepository Use Case - Reading From (Somewhat) Unstructured Text File

Yesterday I was given a text file with a rather weird format that contained several hundred lines like this.



I needed to pull out the first sequence of numbers after the "work place name". I was able to use CacheRepository for this quite easily. I created an entity with the same name as the text file. Then it was just a matter of parsing a string (line) to get out the data I wanted.

CacheRepository Introduction (Why I Made It)

From time to time I have the need to write ETL programs to move data from one data source (sometimes sql server) to another data source (often times sql server). If things are going good this usually is because I'm migration data from a legacy app to one of our newer applications. Performance tends to be very high on my list of requirements. Ensuring a quick turn around time between making a change and seeing the result is very important in these types of apps.

Although these applications are typically .net console apps that are created to ultimately run one time the reality is that they can turn into pretty complex programs and I feel they must be coded with "production" code. I don't typically create unit test for them but that has not always been out of the question.

With that said though I don't want to use "heavy" tooling like NHibernate or Entity Framework to extract and load data from the data sources. When I first started writing these programs I used ADO .Net but quickly fell victim to loads of duplication from copy and pasting. When .net Micro ORM's started to become more popular it seemed like these would be a perfect fit and for some time I felt they were.

Switching to a micro ORM help my code writing and reading significantly which allowed me to focus on other aspects of these programs. Performance bottlenecks from querying and inserting data became more and more frustrating. I found myself writing caches for my query results which helped but often seemed clumsy and inconsistent. Caching my query results did speed up my queries but it then became very apparent that I needed to start bulk inserting data because my inserts were taking way too long.

That's when I created CacheRepository. The tool is essentially just a wrapper over top the micro orm Dapper .Net and it extension Dapper-Extensions. The truth is though I could have wrote this using just about any micro orm.

I've tried to stress usability as much as possible so go try it out.

https://nuget.org/packages/CacheRepository

Sending Anonymous Emails

I'm creating an application within Appharbor and wanted to configure Elmah to send emails to me when an application error occurs. Configuration for this is done from the errorMail tag within the Web.config file. The problem is that this tag expects a user name and password and I obviously don't want to set these in plain text.

While researching this I discovered this article which stated that you didn't need to authenticate when sending an email to a Google apps account.

I really didn't believe that this was possible but I gave it a try and it worked. I happen to have a Goggle Apps account so I was able to use the same MX record that was used in the article.

I looked into this more and made the following observations:

1. Anonymous emails can be sent to all gmail and Google apps users. When I say anonymous I mean that you don't have to supply credentials at all. All you need is a valid MX record which can be looked up quite easily.

2. Not only can anonymous emails be sent, you can actually send an email with a from address of gmail.com or of another Google apps domain (or any domain for that matter). This is interesting because when I tried this the Google gravatar of the fake address was actually displayed right in my gmail app. I would have thought that Google would have at least authenticated its' own users.

3. You can't use the smtp server "smtp.gmail.com" without enabling ssl and authenticating yourself with a valid user name and password. This is true for either relaying emails to outside domains (not gmail or a Google apps account) which makes perfect sense but also for sending a an email within the domain.

I haven't tried this with other popular email hosts but I suspect that this behavior is not unusual. It was just surprising to me. Use the following code to try this on your own:

Helpful Links

http://blog.dantup.com/2011/05/configuring-elmah-to-send-emails-without-putting-your-password-in-the-config-file

Coding Beliefs #1

Code is read far more often than it is written. Creating code that is easy to read and easy to understand gives it a higher likelihood of being easier to change. That is far more important than how quickly you can write the code the first time.

Application Startup Tasks in FubuMVC

Within my applications I often have a need to perform some kind of initialization whenever the application starts up. I typically call these application startup tasks. Because I use Structuremap as my dependency injection tool it is really easy to register and scan for specific implementations of an interface. So I usually create the following:

public interface IApplicationStartupTask
{
    void Execute();
}

Now when I want all of the implementation of this interface I can just inject an enumeration into some type using constructor injection.

IEnumerable<IApplicationStartupTask> tasks

But here's the problem. The only way I know how to instantiate an object on application startup without new-ing it up explicitly is by doing this:

ObjectFactory.GetInstance<SomeStartupType>()

Really what I end up doing is this though:

ObjectFactory.GetAllInstances<IApplicationStartupTask>().Each(x => x.Execute);

Throughout my application I try very hard not to use ObjectFactory explicitly but I've never found a way around it when trying to execute these startup tasks. Because this hard dependency has "just worked" for me I've just moved on and not worried about it. But today I changed the way I register Structuremap within FubuMVC and my hard dependency no longer worked. I usually configure FubuMVC like this:

FubuApplication.For<ConfigureFubuMVC>().StructureMap(ObjectFactory.Container)

But I changed it to this



Now a new Container is used per unit of work instead of the global Container. I went out searching for the "right way" of doing this and with some help from Jeremy Miller I discovered that what I was trying to do is already built into FubuMVC!

Jeremy says..." Use the IActivator interface from Bottles and register that with your container. Or really, just convert your IApplicationStartupTask's to IActivator's and you're good to go."

Auto-Start ASP .NET Applications

Reminder!

I always forget that since ASP.NET 4 (IIS 7.5 - Windows 7 / Windows Server 2008 R2) we can enable the web application to auto-start which makes start up times less of an issue if the app is not used frequently throughout the day. This tends to be important for me as we do a lot of application initialization on app start up.

I pulled the following from Scott Guthrie...

Configuring an ASP.NET 4 Application to Auto-Start

To use the ASP.NET 4 auto-start feature, you first configure the IIS “application pool” worker process that the application runs within to automatically startup when the web-server first loads.  You can do this by opening up the IIS 7.5 applicationHost.config file (C:\Windows\System32\inetsrv\config\applicationHost.config) and by adding a startMode=”AlwaysRunning” attribute to the appropriate <applicationPools> entry:

<applicationPools>

     <add name="MyAppWorkerProcess" managedRuntimeVersion="v4.0" startMode="AlwaysRunning" />

</applicationPools>

If you load up the Windows task manager, click the “show processes from all users” checkbox, and then hit save on a startMode attribute change to the applicationHost.config file, you’ll see a new “w3wp.exe” worker process immediately startup as soon as the file is saved.

A single IIS application pool worker process can host multiple ASP.NET applications.  You can specify which applications you want to have automatically start when the worker process loads by adding a serviceAutoStartEnabled="true" attribute on their <application> configuration entry:

<sites>

     <site name="MySite" id="1">

          <application path="/" serviceAutoStartEnabled="true" serviceAutoStartProvider="PreWarmMyCache" />

     </site>

</sites>

<serviceAutoStartProviders>

     <add name="PreWarmMyCache" type="PreWarmCache, MyAssembly" />

</serviceAutoStartProviders>

The serviceAutoProvider="PreWarmMyCache" attribute above references a provider entry within the config file that enables you to configure a custom class that can be used to encapsulate any "warming up" logic for the application.  This class will be automatically invoked as soon as the worker process and application are preloaded (before any external web requests are received), and can be used to execute any initialization or cache loading logic you want to run before requests are received and processed:

public class PreWarmCache : System.Web.Hosting.IProcessHostPreloadClient {

    public void Preload(string[] parameters) {

        // Perform initialization and cache loading logic here...

    }

}

IIS will start the application in a state during which it will not accept requests until your "warming up" logic has completed.  After your initialization code runs in the Preload method and the method returns, the ASP.NET application will be marked as ready to process requests. 

You can optionally combine the new auto-start "warming up" feature with the load-balancing capabilities of the IIS7 Application Request Routing (ARR) extension, and use it to signal to a load-balancer once the application is initialized and ready to accept HTTP traffic – at which point the server can be brought into the web farm to process requests.

Helpful Links

http://weblogs.asp.net/scottgu/archive/2009/09/15/auto-start-asp-net-applications-vs-2010-and-net-4-0-series.aspx

Sync SVN Repository to a Bitbucket Git Repository

At my current job we have multiple Subversion repositories that are hosted internally with VisualSVN Server. This has worked great for us for many years now. Before subversion (and many many years ago) we used CVS as our version control tool and WinCVS for our client.

Well just as it was previously obvious that we needed to switch from CVS to Subversion, I'm starting to get a similar feeling with switching from Subversion to Git. So we are now interested in the possibilities of hosting private repositories in Bitbucket. We aren't experts in the GIT ecosystem and we run windows throughout here. Because of this I want to start with baby steps. I figured a good start would be to just get one of our internal repositories out into Bitbucket so the following are the steps I took (minus the many hours) to do just that.

Notice: I'm assuming there are many ways to do what I was trying to do. This is just the way I did it :) Also, these steps are unfortunately not complete. I did not record everything that I did here. I just had some notes after my success and thought I would share.

Step 1- Install Git for Windows

This project provides a fork of Git on Windows. I actually installed this months ago so I can't remember any specific issues that I may have had.

Step 2- Install Ruby

I can't remember if Git for Windows installs Ruby already for you or not. It probably doesn't but then I'm just not sure as I installed Ruby separately years ago but for some reason think that maybe Git for Windows has a dependency on it. Like I said though, it probably doesn't. None the less, Ruby must be installed.

Step 3- Install the Ruby gem svn2git

gem install svn2git

That should do it. I looked into other tools that supposedly migrated Subversion repositories to Git but this one seemed the best for me.

I tried several times to get svn2git to work but just couldn't. Then I realized that git already has git-svn built into it so I should just use that! At this point I'm not even sure that we needed Ruby installed. Like I stated earlier, these steps aren't compete.

Step 4- Clone the SVN Repository using git svn

Start up Git bash and type:

git svn clone -s https://server.local/svn/someProject

This clones the SVN repository (including the trunk, tags, and branches) into the directory you are in. The -s flag states that the repo structure is in the standard format. Be warned that if you are cloning an old/active repository that this will take a long time to complete. I actually recommend doing this once and then making a copy of the resulting folder. This way if you end up messing up something in the clone (like I did several times) you can just delete it and use the copy you created instead of having to run this command again.
Note: If you have a repository that you just can't do a complete clone of because it is too big you can run the following command:

git svn clone -s -r HEAD https://server.local/svn/someBigProject

The -r HEAD means to just grab the most recent version of the trunk so this will not include any history. If you do this then you can skip Step 5 below.

Step 5- Execute a Git Checkout

git checkout -b trunk remotes/trunk

This is needed because if you look at the folder that was created from the clone, you'll notice it is empty except for a .git folder. This command is checking out the trunk to the working tree. After this you'll see all of the code that was in your trunk.

Step 6- Change name of branch from trunk to master

git branch -m master

I don't think I had to do this but I thought it was a good idea as master seems to be the convention for Git.

Step 7- Update the Working Tree

git svn rebase

The documentation states that this fetches revisions from SVN and "rebases" the current tree and that it is similar to an svn update. The thing is though is that when I did this I expected nothing really to happen because I just pulled from svn and there weren't any commits to it but for some reason this acted like it was doing a bunch of modifications.
Now things get even weirder. I ran the same command again:

git svn rebase

This seemed to do even more updates and took a little bit of time to complete. At this point when I ran the same command again I received the following response:

'Current branch trunk is up to date.'

That's what I was expecting the first time I ran that command. I don't understand but this is what worked for me.

Step 8- Create a New Bitbucket Repository

I created a private repository.

Step 9- Generate & Install SSH Keys

I would assume that you can following this documentation to do this. I used it to install the keys that I created but I didn't use PuTTygen to generate them. Instead I used ssh-keygen. I basically followed this Github documentation except for step 4. A good way to test that you have the keys set up properly is to execute the following command:

ssh -T hg@bitbucket.org

If you receive a message similar to the following then you are good to go:

conq: logged in as gbrunton.
You can use git or hg to connect to Bitbucket. Shell access is disabled.

Step 10- Prepare repository for Bitbucket

git remote add origin ssh://git@bitbucket.org/gbrunton/someProject.git

This adds a remote for the repository. Whatever that means :) I noticed though that it updates the config file within the .git folder.

Step 11- Push to Bitbucket

git push -u origin master

This command pushes the local repo into Bitbucket with all of the trunk master history!

Step 12- Schedule Daily Task

Because all I want right now is to just copy what gets committed into SVN over to Bitbucket, I created a batch file that gets ran by the scheduler.

Conclusion

This took quite a bit of time to figure out and really isn't that useful as is. In order to completely remove our SVN usage we would have to be able to migrate our Bugzilla bugs over to Bitbucket's Issues. We would also have to find a replacement for the current commit hook we have between SVN and Bugzilla. At this point I think this was a good first step.

Helpful Links

http://stackoverflow.com/a/3240146/384853
http://trac.parrot.org/parrot/wiki/git-svn-tutorial