Check for duplicates using xref

Check for duplicates using xref

I needed to check for duplicate attachments in a system and remove the duplicates. I created a Job that used the same origin and destination and pulled all records from the Attachments table. I had a before operation that would read and write to the same xref list. In this code, I check to see if the record I'm reading already exists in the XREF. If it exists, then I know it is a duplicate and can proceed to the Delete Stage. If it does not exist, then it is not a duplicate and I write the data to the XREF and skip the Delete Stage.

Sub VBScriptProcedure
	dim strExists
	strExists = xrefread("sf-attachments","@@ORG:ParentId@@@@ORG:NAME@@")
	If strExists = "Exists" Then
		LogMessage "Duplicate of: @@ORG:NAME@@"
	Else
		XrefWrite "sf-attachments","@@ORG:ParentId@@@@ORG:NAME@@","Exists"
		GoToNextRow
	End if
End Sub

    • Related Articles

    • Working with Cross References (Xref files)

      Working with Cross References (Xref files) Xref files servermany purposes. The most common case is as foreign key lookups. Another common case is to use Xrefs as pick list lookups. Tables can be created manually. They can be created by importing the ...
    • Using Row Hashing for Incremental Integrations

      Hashing The Whole Record Typically when setting up an ongoing integration, we only want to pull data from your origin which has changed since the last time Starfish ran. This is accomplished using LastRunDate and applying it as a filter against your ...
    • Cross-reference (Xref)

      StarfishETL has built-in methods for maintaining relationships between different systems. Put simply, you will be storing an ID from your Origin, and the new corresponding ID from your destination for the record. For example, if you are migrating ...
    • Using StarfishETL Scripting Class Properties & Methods in C#

      See available StarfishETL Scripting Class Variables, Properties and Methods. To use these variables, properties and methods, you must append "Starfish." to the beginning of variable, property or method AND you must use the exact capitalization as ...
    • Repeating a Stage using Queued Datasets

      Sometimes when building maps, it becomes necessary to call a stage multiple times to process a subset of nested data. Take for example an Origin which may return an Account, but then also a list of Contacts in a structured format (such as JSON). To ...