Friday night was a great start. After a group gathering to go over the site, we split up into pairs to get a set of story cards created. This was a relatively new experience for me in terms of planning a development cycle.  After tinkering with Cucumber over the last month, it seems to fit right into the Agile/BDD way and it’s something that certainly helps me focus on knowing what I plan on doing before I start coding.

Saturday was a different story. I literally spent the first 5 hours or so struggling with the Rails environment chosen for the weekend. I’m no Rails expert and I understand why some gems were chosen, like Bundler in a forward looking approach to the impending Rails3 release.  But it seems a few of us had issues with it in combination with Authlogic because of which gems loaded first and if we had previously installed any gems using sudo instead of installing them into local gems. When it’s all said and done, I could run rake spec but Command-R in Textmate never worked right. Even after moving on I spent a lot of time dealing with schema.rb merge conflicts when pulling from upstream. This isn’t a terribly surprise given that file is a bottleneck of sorts when you have a lot of people adding migrations in rapid fire.

All in all, I got nothing done Saturday and I’m not proud of it. I lost my will to live at around 3PM. If I were to start an event like this, I would give serious thought to removing some of these pain points for the developers, esp for people who aren’t Rails experts. Stick to the stock Rails gem management setup. The less gems you load, the less shenanigans you have to fight, esp ones that try and manage gems/paths. At list for the first iteration. Also, I would either get the migrations/schema done before the event, or at least funnel migration creation through 1-2 people. Let everyone else mock their way down from the outside in.

Sunday was awesome. I did what I should’ve done Saturday. Grab a card and get something out the door and stop worrying if I’m doing it “right” the first time. It’s easier to re-factor than it is to create. Move forward. By the end of the night I managed to get a calendar page up, styled with the help of Parag including next/previous and month dropdown selection support.

I’m definitely looking forward to the next event like this. Hopefully by then I can get my current Rails project done so I won’t suck so much at the next event.

I stumbled across something today that seems obvious in retrospect that leads me to the opinion that the default view/partial view search patterns are broken. If you crack open Reflector.NET and point it towards WebFormViewEngine, you’ll see that ViewLocationFormats and PartialViewLocationFormats are declared the same:

~/Views/{1}/{0}.aspx

~/Views/{1}/{0}.ascx

~/Views/Shared/{0}.aspx

~/Views/Shared/{0}.ascx

In other words, for the Index action on the Home controller, the following paths are searched:

~/Views/Home/Index.aspx

~/Views/Home/Index.ascx

~/Views/Shared/Index.aspx

~/Views/Shared/Index.ascx

This all seems well and good until you happen to have a partial view that has the same name as an action that either happens to be shared, or the parti is being overridden in the local controllers folder. Here’s a somewhat plausible example: A Search controller with an Items action:

~/Views/Search/Items.aspx

    (the view page for the Items action)

~/Views/Home/Index.ascx

    (possibly an items partial view customized just for this controller/action)

~/Views/Shared/Items.ascx

    (the shared partial view to render items in many controllers/actions/views)

This is where the problems start. By default, a call to RenderPartial("Items") within the Items.aspx page is going to find the Items.aspx file first and not the Items.ascx. I’ve never ever seen anyone in the wild put partial views in an aspx file nor have I ever seen anyone put a full action view in an ascx file. So why on earth did they all them to be interchangeable in the location paths? Broken I say. Broken.

The good news is that it’s super easy to tweak these kinds of things in ASP.NET MVC. Inherit from WebFormViewEngine and set your own ViewLocationFormats and PartialViewLocationFormats. I my case I decided to force the issue explicitly. ViewLocationFormats only looks for aspx files and PartialViewLocationFormats only look for ascx files. Now we can override shared partial views locally in controller folders without worrying about name collisions.

Not knowing anything about what it takes to convert a template language to .NET I started from ground up learning how to use ANTLR to create a lexer/parser automatically from a grammar definition. As if that wasn’t enough, my parser output is an Abstract Syntax Tree, which I then use another grammar to template it into C# code using StringTemplate templates, which ANTLR supports natively.

It’s slow going, but I’m learning a lot along the way. At some point I’m going to output VB.NET and it still needs a lot of work for compilation, dll caching and such. The code is in a pre beta state, but it does support basic GET/SET/DEFAULT/IF statements and I’m slowly adding more as I get more comfortable with ANTLR.

Since ReSharper only has a few macros that will make copies of other template variables, I started scouring the internet for clues and found this post. It’s a little outdate for ReSharper for 4.5, so here’s what I had to do to make my custom macro.

You can download the source or fork your own on GitHub.

Creating the Macro

1. Create a new C# class library project.
2. Add references to the following dlls:

   JetBrains.Platform.ReSharper.UI.dll
   JetBrains.ReSharper.Feature.Services.dll

3. Add a new class to your project and mark it with the Macro attribute and implement the IMacro interface:

   using System;
   using System.Collections.Generic;
   using System.Text.RegularExpressions;
   using JetBrains.ReSharper.Feature.Services.LiveTemplates.Macros;
   using JetBrains.ReSharper.Feature.Services.LiveTemplates.Hotspots;
   
   namespace ChrisLaco.ReSharper.Macros
   {
       // {#0:pattern}, etc are arguments that are passed to EvaluateQuickResult arguments[]
       [Macro("applyRegex", ShortDescription = "Replace string matching {#0:pattern} with {#1:string} in {#2:variable}", LongDescription = "Applies regex to replace value in another template variable.")]
       class ApplyRegexMacro : IMacro
       {
           #region IMacro Members
   
           public string EvaluateQuickResult(IHotspotContext context, IList<string> arguments)
           {
               if (arguments.Count != 3)
               {
                   return null;
               }
               else
               {
                   try
                   {
                       // Using the regex argument, replate the value of the specified template variable with the specified string
                       var result = Regex.Replace(arguments[2], arguments[0], arguments[1], RegexOptions.IgnoreCase);
   
                       return result;
                   }
                   catch (Exception e)
                   {
                       return "<" + e.Message + ">";
                   }
               }
           }
   
           public HotspotItems GetLookupItems(IHotspotContext context, IList<string> arguments)
           {
               return null;
           }
   
           public string GetPlaceholder()
           {
               return "a";
           }
   
           public bool HandleExpansion(IHotspotContext context, IList<string> arguments)
           {
               return false;
           }
   
           // Must return the same # of parameters declared in the Macro
           public ParameterInfo[] Parameters
           {
               get
               {
                   return new[] { new ParameterInfo(ParameterType.String), new ParameterInfo(ParameterType.String), new ParameterInfo(ParameterType.VariableReference) };
               }
           }
   
           #endregion
       }
   }

4. In AssemblyInfo.cs, apply assembly attributes for the ReSharper plugin manager:

   using System.Reflection;
   using System.Runtime.InteropServices;
   using JetBrains.UI.Application.PluginSupport;
   
   [assembly: PluginTitle("ChrisLaco.ReSharper.Macros")]
   [assembly: PluginDescription("Provides additional template macros: ApplyRegexMacro.")]
   [assembly: PluginVendor("Christopher H. Laco")]

   This information will be displayed in the ReSharper -> Plugins dialag:

   

5. To install plugin, simply copy the compiled dll into one of the follow directories:

   C:\Documents and Settings\username\Application Data\JetBrains\ReSharper\v4.5\vs9.0\Plugins
   C:\Program Files\JetBrains\ReSharper\v4.5\Bin\Plugins

Using the Macro

1. Select the macro from the list of available macros:
2. Click on the macro variables to set them:
3. Finish up the template text and savr your changes:

Off I went to the Apple store Sunday to get it fixed. Aside from the wait due to the place being packed as always, the process was pretty easy. My Genius called over another Genius and said “Hey, come look at this.” Oh no I thought. Apparently, one thing that gets the Genius folk excited are laptops that are old but in mint condition. They seemed pleased that my old 2007 was cherry. Of course it was. I pamper the thing. The Speck case doesn’t hurt either. And it doesn’t hurt my chances of getting the screen fixed without issue as well.

Luckily for me, the LCD replacement was an in store repair. Once the part came in Tuesday the repair was done and they called me this evening to come pick it up.

Nice. Thank god I spent the $ I saved on an Apple store refurb on AppleCare.

Introduction

In this article, I’ll show you how to monitor a local directory using Siphon and go over some of the different options that are available.

The Scenerio

Your mission, should you chose to accept it, is to check a local file system directory for new CSV files twice a day at 8am and 3pm. Upon completion, or failure of processing each file, we will rename then move them to separate directories for a human to review. In order to achieve this, we will need to:

1. Create a custom processor class to process our csv files
2. Configure a new monitor in app.config
3. Run the monitor

Creating a Processor

In order to have Siphon process data, we must create a custom processor class. More often than not, this will probably just be an adapter  between the Siphon API and your other existing software. The only requirements for a processor in Siphon is that the class:

1. The class must have a constructor that takes no parameters (private, protected or public)
2. The class must implement the IDataProcessor interface

For the sake of me getting my C# skills up to par, we’ll use C# for this example. I’m going to assume that you have a basic understanding of starting new projects, classes, and adding references to other assemblies. Once you’ve created a new class and added a reference to System.Configuration and Siphon.dll, we want to implement the IDataProcessor interface. This consists of 3 methods: Initialize, Process and Dispose:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using ChrisLaco.Siphon;

namespace CustomProcessor
{
    public class CustomProcessor : IDataProcessor
    {
        public void Initialize(ProcessorElement config)
        {
        }

        public bool Process(IDataItem item)
        {
        }

        public void Dispose()
        {
        }
    }
}

A data processor has three basic actions:

1. Initialize the process using the configuration supplied from the service or console where this monitor is running
2. Process the new data item, returning True if it was processed successfully or False if processing failed
3. Cleanup any resources

First, a word from our sponsor about IDataItem and IDataItem<T>

When a monitor finds new items to process, it wraps them in any number of classes that implement the IDataItem<T> interface. This interface exposes three properties:

1. Name: A friendly name for the data item
2. GetString: The contents of the data item
3. Data: The underlying data item in it’s native format

What gets returned from the Data property depends on what type of data each specific monitor understands. The LocalDirectoryProcessor returns FileInfo objects, as does the FtpMonitor. The DatabaseMonitor returns TableRow objects. Some monitors, like Imap and Pop3 monitors, return simple string that also has a reference to the temp file where it’s stored. It is assumed, wisely or otherwise that the monitor and the processor have to both understand and work with the same Data types. It wouldn’t make much sense for the DatabaseMonitor to send an IDataItem<TableRow> to a processor that assumed it was getting IDataItem<FileInfo>. Of course, there’s nothing stopping you from having an uber Process() method that looks to see what types it’s getting understand multiple item types. The Process method on the IDataProcessor interface is actually set to receive IDataItem [without a specific type] for just this reason and GetString will suffice it you’re only inspecting contents.

Back to implementing IDataProcessor

For the sake of simplicity, let’s simply inspect the name of the new files and return True if the file has one name, and False for any other file:

public bool Process(IDataItem item)
{
    IDataItem<FileInfo> fileItem = (IDataItem<FileInfo>)item;

    if (fileItem.Data.Name == "success.csv")
    {
        return true;
    }
    else
    {
        return false;
    }
}

First, we cast the item to an instance of IDataItem<FileInfo>. Next, we check the file name using the Name property of Data, which is a FileInfo object of course. If it is success.csv, return True. Otherwise, return False.

Configuration

Now that we our simple processor,  we need to configure a monitor in our app.config:

<siphon>
  <monitors>
    <monitor name="CsvMonitor" type="ChrisLaco.Siphon.LocalDirectoryMonitor, Siphon">
      <settings>
        <add name="Path" value="C:\" />
        <add name="Filter" value="*.csv" />
        <add name="ProcessCompleteActions" value="Rename, Move" />
        <add name="ProcessFailureActions" value="Rename, Move" />
        <add name="CompletePath" value="Completed" />
        <add name="FailurePath" value="Failed" />
        <add name="CreateMissingFolders" value="True" />
      </settings>
      <schedule type="ChrisLaco.Siphon.DailySchedule, Siphon">
        <daily>
          <time value="8:00" />
          <time value="15:00" />
        </daily>
      </schedule>
      <processor type="CustomProcessor, CustomProcessor" />
    </monitor>
  </monitors>
</siphon>

First we give the monitor a name and tell Siphon what class to use:

<monitor name="CsvMonitor" type="ChrisLaco.Siphon.LocalDirectoryMonitor, Siphon">

Next, we need to configure the monitor using the settings group:

• Path: The full path where to scan for new files.
• Filter: The file name filter to use; only returning files that match the filter.
• ProcessCompleteActions: The actions to perform on the files when processing is successful.
• ProcessFailureActions: The actions to perform on the files when processing fails.
• CompletePath: The relative or full path to the folder to move files into when processing succeeds if the Move action is set.
• FailurePath: The relative or full path to the folder to move files into when processing fails if the Move action is set.
• CreateMissingFolders: Create the Path, CompletePath and FailurePath folders if they do not already exist.

In Siphon, whenever possible, the Path, FailurePath and CompletePath options accept and convert to use Uri objects internally. C:\ is converted to file:///C:/. This is because the same code is used for remote folders as well in FtpMonitor, where Path could be ftp://example.com/ instead. Just like internet uris, the CompletePath and FailurePath settings can be absolute (ftp://example.com/failure/), root relative (/failure/) or relative to the path directory (../failure). This allows you to easily nest directories if necessary.

Now that we have our monitor configured, we need to give it a schedule:

<schedule type="ChrisLaco.Siphon.DailySchedule, Siphon">
  <daily>
    <time value="8:00" />
    <time value="15:00" />
  </daily>
</schedule>

Here we’ve selected the DailySchedule class and set two times of day to schan for new files: 8:00 and 15:00. The times are in the local server time zone and based on 24 hour notation. The only thing left to do is send the data to our custom processor above:

<processor type="CustomProcessor, CustomProcessor" />

Running Your Monitor

We have a custom processor and a fresh config. Now we need to run it. There are two ways to make use of your monitors in Siphon:

1. SiphonService: Windows Service that loads monitors from app.config at startup and runs the monitors based on their schedules.
2. SiphonConsole: Command line utility that loads monitors from app.config and runs the monitors manually.
3. Both: Use the command line utility to run a monitor immediately that is hosted on a local/remote SiphonService.

If you want to run your monitor automatically, simply add it to SiphonService.exe.config whever SiphonService is installed and restart the service. If you want to run your monitor manually fomr the command line, add it to SiphonConsole.exe.config.

The run a monitor, or all monitors from the command line, simple do:

C:\Siphon>SiphonConsole.exe CsvMonitor
C:\Siphon>SiphonConsole.exe *

To run a monitor hosted by SiphonService, simply supply the location of the service, then the monitor name:

C:\Siphon>SiphonConsole.exe http://localhost/ServiceAdminPath CsvMonitor
C:\Siphon>SiphonConsole.exe http://myotherserver/ServiceAdminPath *

If you want to run monitors on hosted under SiphonService, you must enable the remote administration in the config:

<siphon enableRemoteAdministration="True">

We’ll cover how to control the administration url/port from config as wel as using endpoint configurations instead of the http url i the console in a future post. Happy monitoring!

Introduction

In this article, I’m going to cover how Siphon configuration works, the options and how to use the configuration classes to read/write configuration information.

Configuration Section

Here’s a sample Siphon configuration section from the source code:

<configSections>
  <section name="siphon" type="ChrisLaco.Siphon.SiphonConfigurationSection, Siphon" />
</configSections>

<siphon>
  <monitors>
    <monitor name="IntervalMonitor" type="ChrisLaco.Siphon.LocalDirectoryMonitor, Siphon">
      <settings>
        <add name="Path" value="C:\" />
        <add name="Filter" value="*.tmp" />
      </settings>
      <schedule type="ChrisLaco.Siphon.IntervalSchedule, Siphon">
        <interval value="1.2:3:4" />
      </schedule>
      <processor type="ChrisLaco.Siphon.DummyProcessor, Siphon" />
    </monitor>
  </monitors>
</siphon>

Let’s break this down piece by piece. First, we need to tell .NET who’s in charge of the siphon configuration section:

<configSections>
 <section name="siphon" type="ChrisLaco.Siphon.SiphonConfigurationSection, Siphon" />
</configSections>

Next, we add a new monitor element to the monitors collection:

<monitor name="IntervalMonitor" type="ChrisLaco.Siphon.LocalDirectoryMonitor, Siphon">

Here we have a monitor named IntervalMonitor and told Siphon what type to load. If you’re not familiar with using type strings in .NET configuration,  the string above tells .NET that we want to use the class ChrisLaco.Siphon.LocalDirectoryMonitor in the Siphon assembly. Note that if you want to use types that are installed into the GAC you will need to use the full type string including the Culture, PublicKeyToken and Version.

Next, we configure setting required by the LocalDirectoryMonitor:

<settings>
  <add name="Path" value="C:\" />
  <add name="Filter" value="*.tmp" />
</settings>

The settings consist of name/value pairs that are passed to the monitors Initialize() method. The number of options and their values vary depending on the monitor being used. Next, we’ll tell the monitor how often to look for new data:

<schedule type="ChrisLaco.Siphon.IntervalSchedule, Siphon">
  <interval value="1.2:3:4" />
</schedule>

Just like with the monitor element above, we need to tell Siphon what schedule class we want to use. In our case, we’re going to use the IntervalSchedule, and set the interval using a string TimeSpan.Parse() understands: 1 day, 2 hours, 3 minutes and 4 seconds. If you are interested in a daily schedule, you can also use the daily element to specify a daily schedule:

<schedule type="ChrisLaco.Siphon.DailySchedule, Siphon">
  <daily>
    <time value="8:00" />
    <time value="15:00" />
  </daily>
</schedule>

Note: whether you use an interval or a daily schedule, the schedule type you use must understand the configuration you’ve chosen. As we’ll see in the Configuration classes later, the configuration format is setup to handle the most common use cases, like daily schedule values. Just like the monitor element, the schedule element also has a settings element used to pass name/value options to the specified schedule type.

Now that we’ve chosen a monitor and a schedule, the last thing to do is to tell Siphon who will process the newly found data as it arrives:

<processor type="ChrisLaco.Siphon.DummyProcessor, Siphon" />

Here we’ve loaded the DummyProcessor, which does does nothing with new data and always returns True. As with the monitor and schedule elements, the processor element also has a settings element collection.

Reading Configuration

Siphon comes with a set of classes to read/write siphon configuration data. Once you have finished setting up your app.config, you can use the SiphonConfigurationSection class to load the section and loop through all of the configured monitors:

REM Load config section: defaults to 'siphon'
Dim section As SiphonConfigurationSection = SiphonConfigurationSection.GetSection

REM Loop through monitor elements
For Each monitor As MonitorElement In section.Monitors

    REM Get a setting
    Dim path As String = monitor.Settings("Path").Value

    REM create an instance of the specified type
    Dim instance As IDataMonitor = monitor.CreateInstance
    instance.Path = path

    REM Set the schedule
    instance.Schedule = monitor.Schedule.CreateInstance

    REM Set the processor
    instance.Processor = monitor.Processor.CreateInstance

    REM Process new data
    instance.Process()
Next

The code above is pretty straight forward. One item of interest: when CreateInstance is called, it creates an instance of the specified type and calls the Initialize() method on the new instance. While we set the Path setting manually, the monitors Initialize() could have also done the same thing.

All monitors must implement the IDataMonitor interface. All schedules must implement the IMonitorSchedule interface and all processors must implement the IDataProcessor interface. CreateInstance() looks for a constructor with no parameters in the type being created.

Writing Configuration

You can also use the configuration classes to programaticaly create configuration files. SImply create the settings in code, then save the config to the specified file:

Dim config As Configuration = ConfigurationManager.OpenExeConfiguration("C:\Path\To\SomeApp.exe")
Dim section As New SiphonConfigurationSection
Dim monitor As New MonitorElement("TestMonitor", "LocalDirectoryMonitor, Siphon")

monitor.Settings.Add(New NameValueConfigurationElement("Path", "C:\"))
monitor.Processor = New ProcessorElement("MockProcessor, SiphonTests")
monitor.Processor.Settings.Add(New NameValueConfigurationElement("Foo", "Bar"))
monitor.Schedule.Type = "IntervalSchedule, Siphon"
monitor.Schedule.Daily.Add(New TimeElement(New TimeSpan(100)))
monitor.Schedule.Daily.Add(New TimeSpan(200))
section.Monitors.Add(monitor)

config.Sections.Add("siphon", section)
config.Save()

This will update the app.config file for the specified executable path with the new configuration information. The code above will generate something like this:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
    <configSections>
        <section name="siphon" type="ChrisLaco.Siphon.SiphonConfigurationSection, Siphon, Version=1.0.0.0, Culture=neutral, PublicKeyToken=0576062738dd7ad9" />
    </configSections>
    <siphon>
        <monitors>
            <monitor name="TestMonitor" type="LocalDirectoryMonitor, Siphon">
                <schedule type="IntervalSchedule, Siphon">
                    <interval value="00:01:00" />
                    <daily>
                        <time value="00:00:00.0000100" />
                        <time value="00:00:00.0000200" />
                    </daily>
                </schedule>
                <processor type="MockProcessor, SiphonTests">
                    <settings>
                        <add name="Foo" value="Bar" />
                    </settings>
                </processor>
                <settings>
                    <add name="Path" value="C:\" />
                </settings>
            </monitor>
        </monitors>
    </siphon>
</configuration>

Introduction

For me, writing software seems to be about scratching an itch, filling a need by creating your own solution. Siphon is such a project. It seems that almost every project that come across my desk has the need to monitor some directory for files to process. Sometimes that’s a local directory. Sometimes that’s a remote ftp directory. Sometimes it’s even a pop3 mailbox. After watching numerous similar scheduled tasks, batch files, vbscript and command line utilities  being created I decided to create Siphon to help get all of these tasks back into more consistent manageable pieces.

Siphon is a set of tools to allow you to monitor data on a specific schedule and send that data off or processing. There are three basic parts used to accomplish this:

1. Data Monitors: What do you want to monitor?
2. Monitor Schedules: When do you want to check for new data?
3. Data Processors: What do you want to do with the new data?

These three parts are combined in code or in config files to accomplish the task of monitors and processing new data. These tasks can be configured and run manually using SiphonConsole, and command line utilities for running local/remote monitoring jobs, or SiphonService, a Windows Service that runs jobs based on the schedules supplied.

Data Monitors

The following data monitors are currently supported:

• LocalDirectoryMonitor: Monitor local file system directories for new files.
• FtpDirectoryMonitor: Monitor remote ftp directories for new files.
• ImapMonitor: Monitor IMAP mail boxes for new messages.
• Pop3Monitor: Monitor POP3 mail boxes for new messages.
• MessageQueueMonitor: Monitor MSMQ queues for new messages.
• DatabaseMonitor: Monitor any DbProviderFactory support database for new records.

Data monitors can take separate sets of actions against the data when processing complete/fails. These actions include:

• Delete: Delete the data when processing completes/fails.
• Move:  Move the data when processing completes/fails.
• Rename: Rename the data when processing completes/fails.
• Update: Update the data when processing completes/fails.
• Combined: combine any of these actions together, i.e. Rename then Move.

Monitor Schedules

The following monitor schedules are currently supported:

• IntervalSchedule: Run whenever a specified amount of time as passed.
• DailySchedule: Run at the same time(s) every day.

Requirements

In order to run Siphon you must have the following:

• .NET 3.5
• log4Net for logging (Free)
• Lumisoft.NET for FTP/IMAP/POP3 protocol support (Free)

Then came the upgrade to Parallels 4. From day 1, I’ve had nothing but problems. It was most definitely much slower just to boot XP. Even a fresh install of the software and even an new XP VM fixed nothing. It was still slow. Even more entertaining was that when running XP in VGA mode, Safe Mode, or at the boot options screen, Parallels took up 100% of 1 or both host CPUs. After KB articles, months of patches and support, Parallels tells me that eating up all that CPU is normal. Parallels 3 didn’t have that problem. No, I won’t disable all the options that worked in version 3, and yes, 4GB ram was just fine in version 3.

Well, I’m getting off this train now. Clearly, Parallels 4 has issues, and I’m not the only one having it. VirtualBox is free, installed and running a fresh new install of XP and Studio 2008 and what do you know, it even runs XP in Safe Mode without eating 100% of the host CPU. Go figure.

It’s wireless. I can put it anywhere. Any computer in the house can print to it. Any computer can send faxes through it. It can also scan/copy documents and images and send the results to any computer that is online.

Even better, the ink cartridges are cheap. Less than $10. Worth every penny and the wife is happy. I’m still a little sad that the LaserJet will be retired.