Comments on: Deleting Duplicate Records in FileMaker

By: Goetch

Goetch — Fri, 14 Oct 2011 16:04:21 +0000

One tip for large databases is to insert a freeze window before you loop. This drastically reduced the time it took for this simple solution to work. My database of 25000 records originally took about 20 minutes to run, after the freeze window it only took 1 to 2 minutes to run.

By: Fay

Fay — Wed, 10 Aug 2011 09:59:39 +0000

worked a CHARM, thanks!

By: Gerardo Verrone

Gerardo Verrone — Fri, 21 Jan 2011 06:40:20 +0000

Finally Guys I’ve tested the Filemaker Suggested Method of two related tables, counter, calculation fields, etc and the result shoked me.

Import Time From Excel 18 Sec.
Amount of Start Records to Clean: 16592
Cleaning of Duplicated Records 6 Sec.
Amount of records Unique: 2387

With a couple of test, imagination and the amazing suggested FM instructions I have decreased a 27 min duplicated search to just 6 sec.

All the instructions are posted here: by the user: “bodhibrian”

http://forums.filemaker.com/posts/690cd1e356

Cheers bodhibrian to give all of us a great and fresh idea to speed up the process.

Gerardo Verrone.
animatorstudio@hotmail.com

By: Gerardo Verrone

Gerardo Verrone — Fri, 21 Jan 2011 05:54:08 +0000

It’s Me again and with good news !!!
Mi first idea works pretty quick with a big table.

I will explain my fields and type including the full script as with this I can handle a clean process in just .17 seconds compared to 27 minutes using the record by record way.

Table name: People

Record_ID (Serial)
Full Name (Text)
Soc_Security_Number (Number)
FCount (calculation with the formula Get ( FoundCount ) and the tick Do not evaluate if all… at the bottom uncheked)

Start_Time (Time) Global
End_Time (Time) Global
Total_Time (Time) Global

The table has 16.000 records and most of them have the social number and the Full name repeated several times but.

Inside the table is arranged with all the repetitive Full Names twogether one after the other like This:

Id F. Name Social Number

1 Jhon Smith 232323
2 Jhon Smith 232323
3 Maxwell Hunt 122343
4 Maxwell Hunt 122343
5 Maxwell Hunt 122343
6 Maxwell Hunt 122343
7 Karen Wright 698956
8 Karen Wright 698956

and so on until 16.000

The Script

Set Error Capture [on]

#Set the Time when the script start in a global (Time type) field. Use time not time stamp as with time stamp is not possible to substract time values.

Set Field [Start_Time; Get(CurrentHostTimeStamp)]
Show all records
Sort Records [by Record_ID] Just in case when you come to this layout or table it is unsorted.
Go to record/Request/Page [First]

Loop
Set Variable [$Social; Value:Soc_Security_Number]
Set Variable [$Record_Num; Value:Record_ID]
Perform Find [looking for the equivalence Table:Soc_Security_Number = $Social. With this all the other records are omited]
#If More than one record found, delete but omiting the first one.
If [People:FCount > 1] #Meaning you have more than one record in the last found set
Go to Record/Request/Page[First]
Omit Record #This will keep the first one
Delete all Records [No Dialog] # This delete all duplicated
End if

# Return to the position of the Record omited to jump to the next one and continue cleaning.

Perform Find[$Record_Num]
Show All Records
Go to Record/Request/Page [Next; Exit after last]
End Loop

Set Field [End_Time; Get(CurrentHostTimeStamp)]
Set Field [Total_Time ; Start_Time - End_Time]

# Show a Message stating when the scipt has finished comparing actual time and start time from a global fields.

Show Custom Dialog [”Process Finished” & “Start Time: ” & Start_Time & “End Time: ” & End_Time & “Total Time: ” & Total_Time

By: Gerardo Verrone

Gerardo Verrone — Fri, 21 Jan 2011 04:54:38 +0000

Hi Atif Khan,
Yesterday and without looking here I’ve performed the same script as you. The reasson why I’ve started to research for other way was because if we do it like you mentioned in a big table it takes so long.
For example I want to filter duplicated records in a table with more than 16.000 records and the result was 27 minutes to clean it in a Dual Xeon brand new computer.
I imagine two possible ways to sort this but I did not tried yet any of this scenarios.
First Case
1) Sort the table by the desired field I want to filter duplicates.
2) Introduce later a Field called something like Id_Number as serial non repetitive value so each record has a different number.
3) Inside a Loop structure grab the first record and store his Id_Number in one variable and the field you want to clean in other variable.
4) Perform a find looking for records equal to the one you want to delete and loop while you delete but bypassing the first one. This can be achieved by ommiting the first from the found set or simple jumping two records in this internal loop only the first time.
5) Delete all the found set.
6) Show all records
6) Perform a search to return to your record_id stored at 3, so the pointer will jump to that record
7) No you only need to go to next record (until last) and repeat alll the steps from 3 in advance.

Basically this structure ios the same as yours but istead of looking with each record for a repetition along all your table in a looping way you are grabing apart all the repetitive stuffs to delete it in just one step. I think this will be quicker.

The second method I have found is the one ilustrated in the help of filemaker using twice the same table in a relationship. Look in the help of FM11 for something like “duplicate” or look at here http://forums.filemaker.com/posts/690cd1e356

Good Luck
G.

By: Emma

Emma — Wed, 29 Dec 2010 21:05:09 +0000

Many thanks indeed, that saved a good few hours for me!

By: Nunz

Nunz — Wed, 01 Dec 2010 03:20:36 +0000

thnx Jesse and Geoff

As always, invaluable tips and tricks. FYI, i am using this technique in place of the ‘validation’ rules in the field definitions. So I am allowing a user to create a record and enter all the details (in the case a ‘company record’). Then, before the record becomes ‘active’, i use this tip as part of the ‘error check routine’ to ensure they haven’t created a duplicate name or company code.

The reason i’m doing this is so that they can fill in a record (and leave it in complete if they need to…while they hunt down phone numbers and stuff we REQUIRE to setup an account). Once they have all the info in it, they ‘activate’ the record (meaning work can be done with that record) and then i use this as part of the error checking step…to ensure we don’t GET duplicates!

So with a bit of luck, using this tip HERE, will stop the duplicates occurring…which was the reason for the article in the first place!

Once again. Thnx

By: Don Walker

Don Walker — Wed, 17 Nov 2010 23:34:25 +0000

I’m trying to work a variation on this theme. I’m working with a sports league which requires each player to be individually registered. I can’t delete the “duplicate” records but want to create a mailing list that has each address once with the first names concatenated into a “names” field to go along with the last name and address fields.
thanks,
Don

By: Atif Khan

Atif Khan — Fri, 19 Mar 2010 19:37:54 +0000

I created this script and it worked fine. $NAME will copy the value for which I am looking a duplicate record. Sort the records first and run this script.

I don’t know how to copy the script form FileMaker to here so this is just my typing here.

Show all records
Sort Records
Go to First Record
Set Variable $Name=cFirstNameLastName (assuming I am checking on this field)
Go to Next Record– to match the $name
Loop
IF( cFirstNameLastName=$NAME
DELETE RECORD — If it is same as previous record delete it
ELSE
Set Variable $Name=cFirstNameLastName –if not reset the variable to current records value
GO to Next Recrd (next, exit after last)
End if

END LOOP

By: Atif Khan

Atif Khan — Fri, 19 Mar 2010 19:35:54 +0000

I created this script and it worked fine. $ID will copy the value for which I am looking a duplicate record. Sort the records first and run this script.

I don’t know how to copy the script form FileMaker to here so this is just my typing here.

Show all records
Sort Records
Go to First Record
Set Variable $Name=cFirstNameLastName (assuming I am checking on this field)
Go to Next Record– to match the $name
Loop
IF( cFirstNameLastName=$NAME
DELETE RECORD — it is same as previous record delete it
ELSE
Set Variable $Name=$NAME –if not reset the variable to current records value
GO to Next Recrd (next, exit after last)
End if

END LOOP