Christopher H. Laco » siphon

Monitoring Local Directories with Siphon

claco — Thu, 19 Mar 2009 02:00:58 +0000

This is the third in a series of posts about Siphon, a set of data monitoring utilities for .NET under the MIT license. The source code can be found on GitHub.

Introduction

In this article, I’ll show you how to monitor a local directory using Siphon and go over some of the different options that are available.

The Scenerio

Your mission, should you chose to accept it, is to check a local file system directory for new CSV files twice a day at 8am and 3pm. Upon completion, or failure of processing each file, we will rename then move them to separate directories for a human to review. In order to achieve this, we will need to:

Create a custom processor class to process our csv files
Configure a new monitor in app.config
Run the monitor

Creating a Processor

In order to have Siphon process data, we must create a custom processor class. More often than not, this will probably just be an adapter between the Siphon API and your other existing software. The only requirements for a processor in Siphon is that the class:

The class must have a constructor that takes no parameters (private, protected or public)
The class must implement the IDataProcessor interface

For the sake of me getting my C# skills up to par, we’ll use C# for this example. I’m going to assume that you have a basic understanding of starting new projects, classes, and adding references to other assemblies. Once you’ve created a new class and added a reference to System.Configuration and Siphon.dll, we want to implement the IDataProcessor interface. This consists of 3 methods: Initialize, Process and Dispose:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using ChrisLaco.Siphon;

namespace CustomProcessor
{
    public class CustomProcessor : IDataProcessor
    {
        public void Initialize(ProcessorElement config)
        {
        }

        public bool Process(IDataItem item)
        {
        }

        public void Dispose()
        {
        }
    }
}

A data processor has three basic actions:

Initialize the process using the configuration supplied from the service or console where this monitor is running
Process the new data item, returning True if it was processed successfully or False if processing failed
Cleanup any resources

First, a word from our sponsor about IDataItem and IDataItem

When a monitor finds new items to process, it wraps them in any number of classes that implement the IDataItem interface. This interface exposes three properties:

Name: A friendly name for the data item
GetString: The contents of the data item
Data: The underlying data item in it’s native format

What gets returned from the Data property depends on what type of data each specific monitor understands. The LocalDirectoryProcessor returns FileInfo objects, as does the FtpMonitor. The DatabaseMonitor returns TableRow objects. Some monitors, like Imap and Pop3 monitors, return simple string that also has a reference to the temp file where it’s stored. It is assumed, wisely or otherwise that the monitor and the processor have to both understand and work with the same Data types. It wouldn’t make much sense for the DatabaseMonitor to send an IDataItem to a processor that assumed it was getting IDataItem. Of course, there’s nothing stopping you from having an uber Process() method that looks to see what types it’s getting understand multiple item types. The Process method on the IDataProcessor interface is actually set to receive IDataItem [without a specific type] for just this reason and GetString will suffice it you’re only inspecting contents.

Back to implementing IDataProcessor

For the sake of simplicity, let’s simply inspect the name of the new files and return True if the file has one name, and False for any other file:

public bool Process(IDataItem item)
{
    IDataItem fileItem = (IDataItem)item;

    if (fileItem.Data.Name == "success.csv")
    {
        return true;
    }
    else
    {
        return false;
    }
}

First, we cast the item to an instance of IDataItem. Next, we check the file name using the Name property of Data, which is a FileInfo object of course. If it is success.csv, return True. Otherwise, return False.

Configuration

Now that we our simple processor, we need to configure a monitor in our app.config:

First we give the monitor a name and tell Siphon what class to use:

Next, we need to configure the monitor using the settings group:

Path: The full path where to scan for new files.
Filter: The file name filter to use; only returning files that match the filter.
ProcessCompleteActions: The actions to perform on the files when processing is successful.
ProcessFailureActions: The actions to perform on the files when processing fails.
CompletePath: The relative or full path to the folder to move files into when processing succeeds if the Move action is set.
FailurePath: The relative or full path to the folder to move files into when processing fails if the Move action is set.
CreateMissingFolders: Create the Path, CompletePath and FailurePath folders if they do not already exist.

In Siphon, whenever possible, the Path, FailurePath and CompletePath options accept and convert to use Uri objects internally. C:\ is converted to file:///C:/. This is because the same code is used for remote folders as well in FtpMonitor, where Path could be ftp://example.com/ instead. Just like internet uris, the CompletePath and FailurePath settings can be absolute (ftp://example.com/failure/), root relative (/failure/) or relative to the path directory (../failure). This allows you to easily nest directories if necessary.

Now that we have our monitor configured, we need to give it a schedule:

Here we’ve selected the DailySchedule class and set two times of day to schan for new files: 8:00 and 15:00. The times are in the local server time zone and based on 24 hour notation. The only thing left to do is send the data to our custom processor above:

Running Your Monitor

We have a custom processor and a fresh config. Now we need to run it. There are two ways to make use of your monitors in Siphon:

SiphonService: Windows Service that loads monitors from app.config at startup and runs the monitors based on their schedules.
SiphonConsole: Command line utility that loads monitors from app.config and runs the monitors manually.
Both: Use the command line utility to run a monitor immediately that is hosted on a local/remote SiphonService.

If you want to run your monitor automatically, simply add it to SiphonService.exe.config whever SiphonService is installed and restart the service. If you want to run your monitor manually fomr the command line, add it to SiphonConsole.exe.config.

The run a monitor, or all monitors from the command line, simple do:

C:\Siphon>SiphonConsole.exe CsvMonitor
C:\Siphon>SiphonConsole.exe *

To run a monitor hosted by SiphonService, simply supply the location of the service, then the monitor name:

C:\Siphon>SiphonConsole.exe http://localhost/ServiceAdminPath CsvMonitor
C:\Siphon>SiphonConsole.exe http://myotherserver/ServiceAdminPath *

If you want to run monitors on hosted under SiphonService, you must enable the remote administration in the config:

We’ll cover how to control the administration url/port from config as wel as using endpoint configurations instead of the http url i the console in a future post. Happy monitoring!

Siphon Configuration

claco — Mon, 16 Mar 2009 02:53:49 +0000

This is the second in a series of posts about Siphon, a set of data monitoring utilities for .NET under the MIT license. The source code can be found on GitHub.

Introduction

In this article, I’m going to cover how Siphon configuration works, the options and how to use the configuration classes to read/write configuration information.

Configuration Section

Here’s a sample Siphon configuration section from the source code:

Let’s break this down piece by piece. First, we need to tell .NET who’s in charge of the siphon configuration section:

Next, we add a new monitor element to the monitors collection:

Here we have a monitor named IntervalMonitor and told Siphon what type to load. If you’re not familiar with using type strings in .NET configuration, the string above tells .NET that we want to use the class ChrisLaco.Siphon.LocalDirectoryMonitor in the Siphon assembly. Note that if you want to use types that are installed into the GAC you will need to use the full type string including the Culture, PublicKeyToken and Version.

Next, we configure setting required by the LocalDirectoryMonitor:

The settings consist of name/value pairs that are passed to the monitors Initialize() method. The number of options and their values vary depending on the monitor being used. Next, we’ll tell the monitor how often to look for new data:

Just like with the monitor element above, we need to tell Siphon what schedule class we want to use. In our case, we’re going to use the IntervalSchedule, and set the interval using a string TimeSpan.Parse() understands: 1 day, 2 hours, 3 minutes and 4 seconds. If you are interested in a daily schedule, you can also use the daily element to specify a daily schedule:

Note: whether you use an interval or a daily schedule, the schedule type you use must understand the configuration you’ve chosen. As we’ll see in the Configuration classes later, the configuration format is setup to handle the most common use cases, like daily schedule values. Just like the monitor element, the schedule element also has a settings element used to pass name/value options to the specified schedule type.

Now that we’ve chosen a monitor and a schedule, the last thing to do is to tell Siphon who will process the newly found data as it arrives:

Here we’ve loaded the DummyProcessor, which does does nothing with new data and always returns True. As with the monitor and schedule elements, the processor element also has a settings element collection.

Reading Configuration

Siphon comes with a set of classes to read/write siphon configuration data. Once you have finished setting up your app.config, you can use the SiphonConfigurationSection class to load the section and loop through all of the configured monitors:

REM Load config section: defaults to 'siphon'
Dim section As SiphonConfigurationSection = SiphonConfigurationSection.GetSection

REM Loop through monitor elements
For Each monitor As MonitorElement In section.Monitors

    REM Get a setting
    Dim path As String = monitor.Settings("Path").Value

    REM create an instance of the specified type
    Dim instance As IDataMonitor = monitor.CreateInstance
    instance.Path = path

    REM Set the schedule
    instance.Schedule = monitor.Schedule.CreateInstance

    REM Set the processor
    instance.Processor = monitor.Processor.CreateInstance

    REM Process new data
    instance.Process()
Next

The code above is pretty straight forward. One item of interest: when CreateInstance is called, it creates an instance of the specified type and calls the Initialize() method on the new instance. While we set the Path setting manually, the monitors Initialize() could have also done the same thing.

All monitors must implement the IDataMonitor interface. All schedules must implement the IMonitorSchedule interface and all processors must implement the IDataProcessor interface. CreateInstance() looks for a constructor with no parameters in the type being created.

Writing Configuration

You can also use the configuration classes to programaticaly create configuration files. SImply create the settings in code, then save the config to the specified file:

Dim config As Configuration = ConfigurationManager.OpenExeConfiguration("C:\Path\To\SomeApp.exe")
Dim section As New SiphonConfigurationSection
Dim monitor As New MonitorElement("TestMonitor", "LocalDirectoryMonitor, Siphon")

monitor.Settings.Add(New NameValueConfigurationElement("Path", "C:\"))
monitor.Processor = New ProcessorElement("MockProcessor, SiphonTests")
monitor.Processor.Settings.Add(New NameValueConfigurationElement("Foo", "Bar"))
monitor.Schedule.Type = "IntervalSchedule, Siphon"
monitor.Schedule.Daily.Add(New TimeElement(New TimeSpan(100)))
monitor.Schedule.Daily.Add(New TimeSpan(200))
section.Monitors.Add(monitor)

config.Sections.Add("siphon", section)
config.Save()

This will update the app.config file for the specified executable path with the new configuration information. The code above will generate something like this:

An Introduction To Siphon

claco — Sun, 15 Mar 2009 17:27:00 +0000

This is the first in a series of posts about Siphon, a set of data monitoring utilities for .NET under the MIT license. The source code can be found on GitHub.

Introduction

For me, writing software seems to be about scratching an itch, filling a need by creating your own solution. Siphon is such a project. It seems that almost every project that come across my desk has the need to monitor some directory for files to process. Sometimes that’s a local directory. Sometimes that’s a remote ftp directory. Sometimes it’s even a pop3 mailbox. After watching numerous similar scheduled tasks, batch files, vbscript and command line utilities being created I decided to create Siphon to help get all of these tasks back into more consistent manageable pieces.

Siphon is a set of tools to allow you to monitor data on a specific schedule and send that data off or processing. There are three basic parts used to accomplish this:

Data Monitors: What do you want to monitor?
Monitor Schedules: When do you want to check for new data?
Data Processors: What do you want to do with the new data?

These three parts are combined in code or in config files to accomplish the task of monitors and processing new data. These tasks can be configured and run manually using SiphonConsole, and command line utilities for running local/remote monitoring jobs, or SiphonService, a Windows Service that runs jobs based on the schedules supplied.

Data Monitors

The following data monitors are currently supported:

LocalDirectoryMonitor: Monitor local file system directories for new files.
FtpDirectoryMonitor: Monitor remote ftp directories for new files.
ImapMonitor: Monitor IMAP mail boxes for new messages.
Pop3Monitor: Monitor POP3 mail boxes for new messages.
MessageQueueMonitor: Monitor MSMQ queues for new messages.
DatabaseMonitor: Monitor any DbProviderFactory support database for new records.

Data monitors can take separate sets of actions against the data when processing complete/fails. These actions include:

Delete: Delete the data when processing completes/fails.
Move: Move the data when processing completes/fails.
Rename: Rename the data when processing completes/fails.
Update: Update the data when processing completes/fails.
Combined: combine any of these actions together, i.e. Rename then Move.

Monitor Schedules

The following monitor schedules are currently supported:

IntervalSchedule: Run whenever a specified amount of time as passed.
DailySchedule: Run at the same time(s) every day.

Requirements

In order to run Siphon you must have the following:

.NET 3.5
log4Net for logging (Free)
Lumisoft.NET for FTP/IMAP/POP3 protocol support (Free)