Starfish ETL iPaaS Logging Features and Settings

Starfish ETL iPaaS Logging Features and Settings

Starfish ETL iPaaS Logging Features and SettingsStarfish ETL iPaaS has a number of options for logging and monitoring integration activity. Logging is required at various stages of the integration mapping process, during the testing phase prior to deployment, and during post deployment of the integration.

 

The logging and monitoring capabilities of Starfish ETL iPaaS allow administrators to monitor and track system availability, health of the connection between each end point, errors encountered due to poorly formatted data or errors due to mapping logic. Starfish ETL iPaaS will notify administrators as errors are encountered using email notifications.

Logging configuration and notification options

 

  • System level
  • Connection level
  • Runtime level
  • Scripting and workflow event level
  • Notification
  • Starfish Monitor - Keep-alive service
  • Locally deployed endpoint’s logs
  • Reading the log
  • Job history and past logs

 

System level settings

 

 

System level settings define execution behavior, connection timeout threshold and the expected behavior once the system encounters an error.

 

·       Halt on Error

o   Job execution will stop if any error occurs including row errors.

·       Log Error Rows to CSV

o   Errors will be logged to the Exception CSV files

o   Creates the Exception CSV files when any errors occur, logging the entire row's data, the row # and error description.

·       Command Timeout:

o   Default timeout, in seconds, connectors use when connecting.

System parameters are modified using the engine’s web.config settings file.

 

Cloud deployed systems configure the Starfish Designer using the system settings dialog found at the top-level Designer menu under Settings | Options.

 

Reference the web.config settings wiki page for additional information.

 

 

Establish retry parameters. Parameters establish the behavior encountered an issue with the endpoint connection.

 

·       RetryRowOnErrorAttempts

o   Options: 1 or 0. Retry Rows when an error is encountered.

·       RetryRowOnErrorAttemptWaitMS

o   Options: 1 or greater. The number of milliseconds to wait between retry attempts.

·       RetryRowReconnectAfter

o   Options: 0 or greater. Attempt reconnect after x number of failures.

 

Changes to the retry options must be made directly to the iPaaS engine’s web.config settings.

 

Inform Starfish Cloud ops team if your cloud deployed system requires system level changes to the Retry Row options. 

 

Connection level settings

 

Each connection may handle logging uniquely. It may be required to request each endpoint to provide additional logging that is unique to the connection. Not all connections can provide additional logging information.

 

If the connection can provide additional logging, you can add string parameters to the connection. Set this parameter for each connection.

 

Recommended for temporary use.  Highest level logging. Very verbose and potentially large files are created.

 

This information is recorded to a log file local to the iPaaS server. Access to this information will require assistance from the Starfish ETL Cloud Ops team.

 

Reference the web.cofig settings wiki page for more information about connection level settings. 

 

If the connection can provide additional logging, you can add string parameters to the connection. Set this parameter for each connection.

 

·       LogPath

  • Full text file path name
  • Path to save log files in when LogToFile is enabled.

 

·       LogToFile

  • Boolean, Default False. If enabled, a log file (at the “High” level) with be created within the specified LogPath location.

Not all connections are support additional logging. 

 

If additional logging is required, add the following to the Additional Connection Sting Parameters in the Edit Connection dialog.

 

logfile=d:\logs\xxx\xxx.txt;verbosity=3

 

 

 

Runtime level settings

 

Logging level help to provide how much data is captured. You can establish the level of logging by setting the Run Job characteristics for each Job. Find this setting as part of each Job’s Run Job view.

 

Logging level

·       None

o   Only a total record count will be returned at the end of the Job.

·       Low

o   Returns an overview of each row/stage processed.

·       Medium

o   Returns Low Level messages and displays all SQL statements that are executed.

·       High

o   Returns Low and Medium Level messages and breaks down each field during evaluation. This shows the actual VBScript executed and return values from Function Fields.

·       Notify on complete

·       Argument

 

Reference the Run Options wiki page for more information. 

 

Scripting and workflow event level

 

Custom logging can be captured as part of each script. This is often used to help verify code and to verify handling of poorly formatted data. 

 

·       Starfish.LogMessage()

·       Appends a custom message to the Log which will always be displayed (even if Logging Level is set to None).

·       Message is the passed in String parameter. Second parameter is a valid MsgBoxStyle (vbInformation, vbQuestion, vbExclamation, vbCritical)

 

Additional scripting can be added as part of a triggered workflow event due to an error. Enable this feature for each Job by creating a post-process and selecting the On Row Error Event process type. Add script details as required to custom handle the error.

 

·       On Row Error Event

 

The ‘on row error’ scripting event is a Cloud Designer supported feature. The on-premises Designer does not support ‘on row error’ scripting events.

 

Logging notifications

 

Notifications can be set up to notify a specific person or a group of people. Notifications are sent via email. Notifications embed the error details in the body of the email. 

 

This setting can be set in each Job’s Job Settings view. 

 

Parameters for configuration notifications include:

 

·       Send notifications to:

o   List multiple emails separated by commas

·       Notify On

o   Information

o   Error

·       Send

o   Always

o   Only on activity

o   Only on row error

 

Starfish Monitor Service - Keep-alive services for locally deployed systems

 

·       The Starfish Monitor is intended to help ensure uptime for mission-critical systems integrations developed with our ETL tool. There are two components to this service, Monitor and Watchdog.

·       Monitor is intended to run on the same server as the Starfish Engine. Starfish maintains a log of each time a successful job is run. If your job is supposed to run once every night, then every 24 hours or so this log should be updated. Monitor reviews this log, and if a configured amount of time has passed since it has seen an update (in our example say 25 hours) then it will send a notification email to the administrator you've configured. This is useful in cases where perhaps a Windows update has left IIS in a state of needing to be restarted.

·       Watchdog can be considered an extension of this by ensuring the system is constantly running. Its purpose is to occasionally ping a central server at Starfish ETL, for example every 5 minutes. Our server checks to ensure we've heard from your server successfully in the past 20 minutes (configurable). If monitoring service has not heard from the target sever, Starfish ETL will send a notification to your administrator. This is useful in the case where a server has been unintentionally shut down, loses internet connectivity, or other unforeseen circumstances. If you intend to enable Watchdog, you must inform Starfish ETL Support before so we can enable this feature for your company. 

·       When contacting us to enable Watchdog, please provide the following pieces of information:

o   Your Starfish ETL license code.

o   The kick interval that you configure below.

o   Your desired fault span (how much time we should allow to pass before notifying you).

o   Email address(es) where we should send fault notices to.

 

Monitor Service Configuration

 

  • The settings below must be configured before enabling the Starfish Monitor service. You will edit the StarfishMonitor.exe.config file which should be in the same folder where Starfish Scheduler was installed.
  • Access the Starfish Monitor wiki page for more information. 

 

EnginePath

 

Location of the 'engine.db' file for the StarfishEngine instance you want to monitor.

 

ProblemInterval

 

In seconds, how much time should be allowed to pass since the last successful job was run before sending notifications.

 

CheckInterval

 

In seconds, how often the monitor's internal timer should check to see if there are any problems.

 

SMTPServer

 

SMTP hostname to use when sending emails. Leave defaults to use the provided Starfish account.

 

SMTPPort

 

SMTP server port to use when sending emails. Typically, 25 or 587.

 

SMTPUsername

 

Username to use for the SMTP connection if authentication is required.

 

SMTPPassword

 

Password to use for the SMTP connection if authentication is required.

 

EnableSSL

 

Whether the SMTP server's connections (configured above) require an SSL connection.

 

MailFromAddress

 

Address the email should appear to have been sent from.

 

MailFromDisplay

 

Display name for the address the email should appear to have been sent from.

 

EmailRecipients

 

List of recipients to notify by email. Separate multiple email addresses using a comma.

 

CompanyName

 

Name of your company - will be used in the notification email that gets sent. If you're using Watchdog (below), this MUST match the Company Name for your Starfish license.

 

Debug

 

Logs additional entries to the event log for use when debugging configuration issued. Ensure this is false for production environments.

 

Watchdog Configuration (on-premises only)

 

  • The settings below are also configured in the same StarfishMonitor.exe.config file.
  • The watchdog feature is an on-premises only feature.

 

WatchdogEnabled

 

Boolean to enable watchdog. Inform Starfish ETL support before enabling, we must also enable on our side.

 

LicenseCode

The license code that has been assigned to your company.

 

WatchdogKickInterval

 

In seconds, how often the watchdog should update the last time we heard from you. Please keep this at something reasonable like every 5 minutes (300 seconds).

 

 

Working with the Log file

 

You can access log files from different areas within the Starfish Designer. Each log will list detail about the execution of the job. Some of the data in in the log is system generated and the level of detail is controlled by the logging level. Custom logging is also part of the log. Custom logging is created by the Starfish.LogMessage() called by custom scripts. 

 

Access logs

  • Job’s Preview Mode
  • Job’s Run Job Mode
  • Job History

 

Actions (Cloud only)

  • Export to XLSX
  • Export to CSV
  • Search

 

Log report

  • Timestamp (Cloud only)
  • Type

o   Info

o   Message

o   Result

  • Row
  • Entry

 

Accessing the local endpoint’s event logs

·       Sometimes it is necessary to access a locally deployed system’s event logs to gain deeper access to the data logs if the information is not reported back to the Starfish ETL iPaaS. Sometimes it is required to view the Window’s event viewer to identify the source of the issue.

 

Logging using the Ray

 

Additional logging for the Starfish Ray can be set using the Ray’s configuration file or by using the Cloud Designer’s Edit Ray dialog.

 

Actions

·       Download log

·       Clear log

·       Turn logging on

Additional information can be found on the Starfish Wiki about the Starfish Ray.

 

Log history

 

Lists past job execution with associated snapshot which includes rows, inserts, updates, deletes, skips, ignores and errors.

 

Includes error log if row error encountered.

 

Includes error in result if non-row error encountered including connectivity issues.

 

Grid actions

·       Create filter

·       Sorting

·       Grouping

 

Job history view

·       Error count

·       Error log

·       Result

Run time logs are saved as csv files. Contains row data and the error. View using spreadsheet application.

 


    • Related Articles

    • Run Job Options

      Definitions/explanations of the different options for when you wish to run a job. Option Name Description Begin at Row Begin at Row/Leave blank to start at the first row. End at Row End at Row/Leave blank to process through to the end of all rows. ...
    • Starfish Monitor

      Installation Beginning with Starfish ETL 3.8, the Starfish Monitor service is included and created when you install the Starfish Scheduler. The Monitor service's Startup Type is set to 'Manual' by default. If you wish to use this service, you should ...
    • File System Connector

      File System Origin The File System Connector reads all files from a folder specified in the Directory field. The Username and Password fields are not used. Filter can be used to filter for certain file names/extensions. To search for all files with ...
    • Engine Web.Config Settings

      This file is generally located at: ..\inetpub\wwwroot\StarfishEngine\ Setting Default Description LogPath C:\Inetpub\WWWRoot\StarfishEngine\ Path to save log files in when LogToFile is enabled. LogToFile False Boolean, Default False. If enabled, a ...
    • Sage 100 Web Service Setup and Configuration

      Sage 100 Web Service installation and setup instructions Get ready - Prep Sage 100 credentials Sage 100SQL database Company ? Username and password System Prereqs .NET Framework 4.7.1+ IIS with ASP.NET enabled. Installed somewhere where the Sage 100 ...