Starfish ETL Monitor

Starfish ETL Monitor

Installation

Beginning with Starfish ETL 3.8, the StarfishETL Monitor service is included and created when you install the StarfishETL Scheduler. The Monitor service's Startup Type is set to 'Manual' by default. If you wish to use this service, you should configure the settings (defined below), change the Startup Type to 'Automatic' and start the service. If you are already running a previous version of Starfish ETL, it is recommended that you apply updates to all components of the software, then reinstall the StarfishETL Scheduler from the installation files to ensure the service is registered correctly.

Intended Purpose

The StarfishETL Monitor is intended to help ensure uptime for mission-critical systems integrations developed with our ETL tool. There are two components to this service, Monitor and Watchdog.

Monitor is intended to run on the same server as the StarfishETL Engine. StarfishETL maintains a log of each time a successful job is run. If your job is supposed to run once every night, then every 24 hours or so this log should be updated. Monitor reviews this log, and if a configured amount of time has past since it has seen an update (in our example say 25 hours) then it will send a notification email to the administrator you've configured. This is useful in cases where perhaps a Windows update has left IIS in a state of needing to be restarted.

Watchdog can be considered an extension of this by ensuring the system is constantly running. It's purpose is to occasionally ping a central server at StarfishETL, for example every 5 minutes. Our server checks to ensure we've heard from your server successfully in the past 20 minutes (configurable). If we haven't heard from your sever, we'll send a notification to your administrator. This is useful in the case where a server has been unintentionally shut down, loses internet connectivity, or other unforeseen circumstances. If you intend to enable Watchdog, you must inform StarfishETL Support before so we can enable this feature for your company.

When contacting us to enable Watchdog, please provide your the following pieces of information:

  • Your StarfishETL license code.
  • The kick interval that you configure below.
  • Your desired fault span (how much time we should allow to pass before notifying you).
  • Email address(es) where we should send fault notices to.

Monitor Service Configuration

The settings below must be configured before enabling the StarfishETL Monitor service. You will edit the StarfishMonitor.exe.config file which should be in the same folder where StarfishETL Scheduler was installed.

EnginePath
Location of the 'engine.db' file for the StarfishEngine instance you want to monitor.
ProblemInterval
In seconds, how much time should be allowed to pass since the last successful job was run before sending notifications.
CheckInterval
In seconds, how often the monitor's internal timer should check to see if there are any problems.
SMTPServer
SMTP hostname to use when sending emails. Leave defaults to use the provided Connect Creatio account.
SMTPPort
SMTP server port to use when sending emails. Typically 25 or 587.
SMTPUsername
Username to use for the SMTP connection if authentication is required.
SMTPPassword
Password to use for the SMTP connection if authentication is required.
EnableSSL
Whether the SMTP server's connections (configured above) require an SSL connection.
MailFromAddress
Address the email should appear to have been sent from.
MailFromDisplay
Display name for the address the email should appear to have been sent from.
EmailRecipients
Who to send the notifications to, separate multiples with a comma.
CompanyName
Name of your company - will be used in the notification email that gets sent. If you're using Watchdog (below), this MUST match the Company Name for your StarfishETL license.
Debug
Logs additional entries to the event log for use when debugging configuration issued. Ensure this is false for production environments.

Watchdog Configuration

The settings below are also configured in the same StarfishMonitor.exe.config file.

WatchdogEnabled
Boolean to enable watchdog. Inform StarfishETL support before enabling, we must also enable on our side.
LicenseCode
The license code that has been assigned to your company.
WatchdogKickInterval
In seconds, how often the watchdog should update the last time we heard from you. Please keep this at something reasonable like every 5 minutes (300 seconds).

Sample StarfishMonitor.exe.config file

When updating StarfishETL, the StarfishMonitor.exe.config is not automatically overwritten with a new version. In this case, you may be missing the configuration settings mentioned above. If you have an older version of this file you did not customize, you can overwrite it with this sample.

<?xml version="1.0" encoding="utf-8"?>
<configuration>
    <configSections>
        <sectionGroup name="applicationSettings" type="System.Configuration.ApplicationSettingsGroup, System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
            <section name="StarfishMonitor.Properties.Settings" type="System.Configuration.ClientSettingsSection, System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" requirePermission="false"/>
        </sectionGroup>
    </configSections>
    <startup> 
        <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.0"/>
    </startup>
    <applicationSettings>
        <StarfishMonitor.Properties.Settings>
            <setting name="EnginePath" serializeAs="String">
                <value>C:\Inetpub\wwwroot\StarfishEngine\engine.db</value>
            </setting>
            <setting name="ProblemInterval" serializeAs="String">
                <value>14400</value>
            </setting>
            <setting name="SMTPServer" serializeAs="String">
                <value>smtp.office365.com</value>
            </setting>
            <setting name="SMTPPort" serializeAs="String">
                <value>587</value>
            </setting>
            <setting name="EmailRecipients" serializeAs="String">
                <value>monitor@domain.com</value>
            </setting>
            <setting name="CompanyName" serializeAs="String">
                <value>Your Company Name</value>
            </setting>
            <setting name="MailFromAddress" serializeAs="String">
                <value>monitor@domain.com</value>
            </setting>
            <setting name="MailFromDisplay" serializeAs="String">
                <value>Starfish Monitor</value>
            </setting>
            <setting name="Debug" serializeAs="String">
                <value>False</value>
            </setting>
            <setting name="EnableSSL" serializeAs="String">
                <value>True</value>
            </setting>
            <setting name="CheckInterval" serializeAs="String">
                <value>900</value>
            </setting>
            <setting name="SMTPUsername" serializeAs="String">
                <value>monitor@domain.com</value>
            </setting>
            <setting name="SMTPPassword" serializeAs="String">
                <value>Password</value>
            </setting>
            <setting name="WatchdogEnabled" serializeAs="String">
                <value>False</value>
            </setting>
            <setting name="LicenseCode" serializeAs="String">
                <value />
            </setting>
            <setting name="WatchdogKickInterval" serializeAs="String">
                <value>300</value>
            </setting>
        </StarfishMonitor.Properties.Settings>
    </applicationSettings>
</configuration>

    • Related Articles

    • Starfish ETL Scheduler

      Installation The Scheduler is a run as Windows Service. It may installed on any server which has access to the web service, but for the sake of simplicity it is recommended to be installed the same server as the Starfish Engine. The Scheduler reads ...

    • Starfish ETL Admin

      You will install the Admin tool on the workstation where you plan to do your job development. The installation requires the .NET Framework 4.0, and will need the ability to access the web service set up in the previous step. An easy way to test this ...

    • StarfishETL iPaaS Logging Features and Settings

      StarfishETL iPaaS Logging Features and Settings StarfishETL iPaaS has a number of options for logging and monitoring integration activity. Logging is required at various stages of the integration mapping process, during the testing phase prior to ...

    • Starfish Ray

      Overview The Starfish Ray is a light-weight footprint service installed on a client machine which enables the movement of data from a system behind the firewall to other cloud-based systems. Our cloud hosted Starfish software facilitates this ...

    • Import Starfish Map file from a locally stored *.SPD file

      Import Starfish Map file from a locally stored *.SPD file Step Snapshot Go to project and select the Import view Select Browse to import the *.spd file Select *.spd file from directory folder. Once the import is complete, the project will be added to ...