StarfishETL iPaaS Logging Features and Settings StarfishETL iPaaS has a number of options for logging and monitoring integration activity. Logging is required at various stages of the integration mapping process, during the testing phase prior to deployment, and during post deployment of the integration.
The logging and monitoring capabilities of StarfishETL iPaaS allow administrators to monitor and track system availability, health of the connection between each end point, errors encountered due to poorly formatted data or errors due to mapping logic. StarfishETL iPaaS will notify administrators as errors are encountered using email notifications.
| System level settings define execution behavior, connection timeout threshold and the expected behavior once the system encounters an error.
· Halt on Error o Job execution will stop if any error occurs including row errors. · Log Error Rows to CSV o Errors will be logged to the Exception CSV files o Creates the Exception CSV files when any errors occur, logging the entire row's data, the row # and error description. · Command Timeout: o Default timeout, in seconds, connectors use when connecting. System parameters are modified using the engine’s web.config settings file.
Cloud deployed systems configure the StarfishETL Cartographer using the system settings dialog found at the top-level Cartographer menu under Settings | Options.
Reference the web.config settings wiki page for additional information.
|
Establish retry parameters. Parameters establish the behavior encountered an issue with the endpoint connection.
· RetryRowOnErrorAttempts o Options: 1 or 0. Retry Rows when an error is encountered. · RetryRowOnErrorAttemptWaitMS o Options: 1 or greater. The number of milliseconds to wait between retry attempts. · RetryRowReconnectAfter o Options: 0 or greater. Attempt reconnect after x number of failures.
Changes to the retry options must be made directly to the iPaaS engine’s web.config settings.
Inform StarfishETL Cloud ops team if your cloud deployed system requires system level changes to the Retry Row options. |
Each connection may handle logging uniquely. It may be required to request each endpoint to provide additional logging that is unique to the connection. Not all connections can provide additional logging information.
If the connection can provide additional logging, you can add string parameters to the connection. Set this parameter for each connection.
Recommended for temporary use. Highest level logging. Very verbose and potentially large files are created.
This information is recorded to a log file local to the iPaaS server. Access to this information will require assistance from the StarfishETL Cloud Ops team.
Reference the web.cofig settings wiki page for more information about connection level settings.
If the connection can provide additional logging, you can add string parameters to the connection. Set this parameter for each connection.
· LogPath
· LogToFile
| |
Not all connections are support additional logging.
If additional logging is required, add the following to the Additional Connection Sting Parameters in the Edit Connection dialog.
logfile=d:\logs\xxx\xxx.txt;verbosity=3
|
Logging level help to provide how much data is captured. You can establish the level of logging by setting the Run Job characteristics for each Job. Find this setting as part of each Job’s Run Job view.
Logging level · None o Only a total record count will be returned at the end of the Job. · Low o Returns an overview of each row/stage processed. · Medium o Returns Low Level messages and displays all SQL statements that are executed. · High o Returns Low and Medium Level messages and breaks down each field during evaluation. This shows the actual VBScript executed and return values from Function Fields. · Notify on complete · Argument
Reference the Run Options wiki page for more information. |
Custom logging can be captured as part of each script. This is often used to help verify code and to verify handling of poorly formatted data. · Starfish.LogMessage() · Appends a custom message to the Log which will always be displayed (even if Logging Level is set to None). · Message is the passed in String parameter. Second parameter is a valid MsgBoxStyle (vbInformation, vbQuestion, vbExclamation, vbCritical)
| |
Additional scripting can be added as part of a triggered workflow event due to an error. Enable this feature for each Job by creating a post-process and selecting the On Row Error Event process type. Add script details as required to custom handle the error.
· On Row Error Event
The ‘on row error’ scripting event is a Cloud Cartographer supported feature. The on-premises Cartographer does not support ‘on row error’ scripting events. |
Notifications can be set up to notify a specific person or a group of people. Notifications are sent via email. Notifications embed the error details in the body of the email.
This setting can be set in each Job’s Job Settings view.
Parameters for configuration notifications include:
· Send notifications to: o List multiple emails separated by commas · Notify On o Information o Error · Send o Always o Only on activity o Only on row error |
· The StarfishETL Monitor is intended to help ensure uptime for mission-critical systems integrations developed with our ETL tool. There are two components to this service, Monitor and Watchdog.
· Monitor is intended to run on the same server as the StarfishETL Engine. Starfish maintains a log of each time a successful job is run. If your job is supposed to run once every night, then every 24 hours or so this log should be updated. Monitor reviews this log, and if a configured amount of time has passed since it has seen an update (in our example say 25 hours) then it will send a notification email to the administrator you've configured. This is useful in cases where perhaps a Windows update has left IIS in a state of needing to be restarted.
· Watchdog can be considered an extension of this by ensuring the system is constantly running. Its purpose is to occasionally ping a central server at StarfishETL, for example every 5 minutes. Our server checks to ensure we've heard from your server successfully in the past 20 minutes (configurable). If monitoring service has not heard from the target sever, StarfishETL will send a notification to your administrator. This is useful in the case where a server has been unintentionally shut down, loses internet connectivity, or other unforeseen circumstances. If you intend to enable Watchdog, you must inform StarfishETL Support before so we can enable this feature for your company.
· When contacting us to enable Watchdog, please provide the following pieces of information:
o Your StarfishETL license code.
o The kick interval that you configure below.
o Your desired fault span (how much time we should allow to pass before notifying you).
o Email address(es) where we should send fault notices to.
EnginePath
| Location of the 'engine.db' file for the StarfishEngine instance you want to monitor.
|
ProblemInterval
| In seconds, how much time should be allowed to pass since the last successful job was run before sending notifications.
|
CheckInterval
| In seconds, how often the monitor's internal timer should check to see if there are any problems.
|
SMTPServer
| SMTP hostname to use when sending emails. Leave defaults to use the provided Starfish account.
|
SMTPPort
| SMTP server port to use when sending emails. Typically, 25 or 587.
|
SMTPUsername
| Username to use for the SMTP connection if authentication is required.
|
SMTPPassword
| Password to use for the SMTP connection if authentication is required.
|
EnableSSL
| Whether the SMTP server's connections (configured above) require an SSL connection.
|
MailFromAddress
| Address the email should appear to have been sent from.
|
MailFromDisplay
| Display name for the address the email should appear to have been sent from.
|
EmailRecipients
| List of recipients to notify by email. Separate multiple email addresses using a comma.
|
CompanyName
| Name of your company - will be used in the notification email that gets sent. If you're using Watchdog (below), this MUST match the Company Name for your Starfish license.
|
Debug
| Logs additional entries to the event log for use when debugging configuration issued. Ensure this is false for production environments. |
WatchdogEnabled
| Boolean to enable watchdog. Inform StarfishETL support before enabling, we must also enable on our side.
|
LicenseCode | The license code that has been assigned to your company.
|
WatchdogKickInterval
| In seconds, how often the watchdog should update the last time we heard from you. Please keep this at something reasonable like every 5 minutes (300 seconds).
|
You can access log files from different areas within the StarfishETL Cartographer. Each log will list detail about the execution of the job. Some of the data in in the log is system generated and the level of detail is controlled by the logging level. Custom logging is also part of the log. Custom logging is created by the Starfish.LogMessage() called by custom scripts.
Access logs
Actions (Cloud only)
Log report
o Info o Message o Result
|
· Sometimes it is necessary to access a locally deployed system’s event logs to gain deeper access to the data logs if the information is not reported back to the StarfishETL iPaaS. Sometimes it is required to view the Window’s event viewer to identify the source of the issue.
Additional logging for the StarfishETL Ray can be set using the Ray’s configuration file or by using the Cloud Cartographer’s Edit Ray dialog.
Actions · Download log · Clear log · Turn logging on Additional information can be found on the Starfish Wiki about the Starfish Ray. |
Lists past job execution with associated snapshot which includes rows, inserts, updates, deletes, skips, ignores and errors.
Includes error log if row error encountered.
Includes error in result if non-row error encountered including connectivity issues.
Grid actions · Create filter · Sorting · Grouping
Job history view · Error count · Error log · Result | |
Run time logs are saved as csv files. Contains row data and the error. View using spreadsheet application. |