This is a fun project I put out there a while back. I recently went through and optimized the performance a bit so I could officially blog it. It is an implementation of the Levenshtein Distance Algorithm in CFScript that I based off of a C# version written by Siderite Zackwehdex. Finding the "distance" between two strings is a means of comparing two strings to see how similar they both are. This can be done by finding the Longest Common String or LCS. It is as much a brain bender as it can be occasionally useful.The basic gist of the concept is this: Iterate over two strings making a note of how many characters were inserted, deleted, or transposed from one string to the other. When a difference is found, "bookmark" where you are and start looking ahead in each string to see if the strings are going to start matching up again down the line. How far down the line you look is controlled by in an input called maxOffset. The LCS is the number of characters between the two strings which were identical. The "distance" of the two strings is simply the average string length minus the longest common string. The similarity of the two strings can be expressed as a percentage given 1 minus the strings' distance divided the length of the longest string. Enough theory-- let's look at an example: I love to go ride my go-kart. (29 chars)
I love to ride my go-cart outside. (34 chars)
  • There is a deletion of the word "go "
  • There is a substitution of "c" for "k" in go-cart
  • There is an addition of the word "outside"
Overall, the average length of the strings is 31.5 characters.
There are 25 characters in the two strings that are identical. This is our LCS.
String 1 is 4 characters different than string 2, and string 2 has 9 characters different than string 1.
So on average, we will say 6.5 characters would need to be changed to make the strings identical. This is our distance. (4 + 9 = 13 / 2 = 6.5)
Divide that distance by the length of our longest string and we can come to the conclusion that the strings are an 81% match. (1 - (6.5 / 34) = .8088 = 81%) Here's another example:
The rain in Spain stays mainly on the plains
The rain in Spain stays mainly on the plains
The rain in Spain stays mainly on the plains
Lorum Ipsum, yadda yadda.
Lorum Ipsum, yadda yadda.
La La La La Luke, I am your father.
The rain in Madrid stays totally on the plains
The rain in Spain stays mainly on the plains
The rain in Barcelona stays entirely in the air
Lorum Ipsum, Yabba dabba doo.
Whatcha eatin? Nutin' Honey.
Da Da Da Duke, I am your father.
  • Roughly 67 characters are different between the two strings.
  • The Longest Common String (LCS) is 180.
  • The strings are a 73% match.
In addition to porting the logic over to CFScript, there are a few things I added in to my function:
  • If both strings are empty, I short circuit and return a 0 for distance and LCS. Similarity is 100%.
  • If either string is empty, I short circuit and return the length of the non-empty string as the distance. The LCS and similarity is 0.
  • To better detect differences at the start of the string I check the first three characters. That way a matching first letter wouldn't be confused in two strings like "top hat" and "the hat"
  • When looking ahead in the strings trying to reconcile a difference I search for the distance of the maxOffset until I find THREE contiguous matching characters. This is to try and eliminate false positives.
  • My function will highlight the differences between the strings by wrapping the deviations in the HTML tag of your choice. Default is <span style="background: yellow;"></span>
The function is fairly effective for finding basic insertions, deletions, and transpositions between to strings ranging form a few words, to a few paragraphs. Since the algorithm iterates through two strings without looking back, it WON'T find sentences that had their order rearranged. The maxOffset controls how hard the code tries to look and ahead and reconcile an insert or deletion. If you expect entire sentences to be inserted, your maxOffset needs to be at least as big as the largest insertion. Of course, the larger offset you allow for, the more performance will be impacted and the more likely you are to get a false positive when looking ahead. Have fun with it. If you can think of a way to improve the code I would love to hear about it. I have included the code for the function, an example way to call it, and a zipped version of both below.
[code]
<cfscript>

	/*

		StringSimilarity
		Brad Wood
		[email protected]
		May 2007
		Code adopted from Siderite Zackwehdex's Blog
			http://siderite.blogspot.com/2007/04/super-fast-and-accurate-string-distance.html

		Parameters:
			s1:			First string to be compared
			s2:			Second string to be compared
			maxOffset:	Average number of characters that s1 will deviate from s2 at any given point.
						This is used to control how far ahead the function looks to try and find the 
						end of a peice of inserted text.  Play with it to suit.

	*/

    function stringSimilarity(s1,s2,maxOffset)
        {
            var c = 0;
            var offset1 = 0;
            var offset2 = 0;
            var lcs = 0;
			// These two strings will contain the "highlighted" version
			var _s1 = createObject("java","java.lang.StringBuffer").init(javacast("int",len(s1)*3));
			var _s2 = createObject("java","java.lang.StringBuffer").init(javacast("int",len(s2)*3));
			// These chaactes will surround differences in the strings 
			// (Inserted into _s1 and _s2)
			var h1 = "<span style=""background: yellow;"">";
			var h2 = "</span>";
			var return_struct = structNew();
			// If both strings are empty 
            if (not len(trim(s1)) and not len(trim(s2)))
				{	
					return_struct.lcs = 0;
					return_struct.similarity = 1;
					return_struct.distance = 0;
					return_struct.s1 = "";
					return_struct.s2 = "";
		            return return_struct;
				}
			// If s2 is empty, but s1 isn't
            if (len(trim(s1)) and not len(trim(s2)))
				{
					return_struct.lcs = 0;
					return_struct.similarity = 0;
					return_struct.distance = len(s1);
					return_struct.s1 = h1 & s1 & h2;
					return_struct.s2 = "";
		            return return_struct;
				}
			// If s1 is empty, but s2 isn't
			else if (len(trim(s2)) and not len(trim(s1)))
				{
					return_struct.lcs = 0;
					return_struct.similarity = 0;
					return_struct.distance = len(s2);
					return_struct.s1 = "";
					return_struct.s2 = h1 & s2 & h2;
		            return return_struct;
				}
				
			// Examine the strings, one character at a time, anding at the shortest string
			// The offset adjusts for extra characters in either string.
            while ((c + offset1 lt len(s1))
                   and (c + offset2 lt len(s2)))
            {
				// Pull the next charactes out of s1 anbd s2
				next_s1 = mid(s1,c + offset1+1,iif(not c,3,1)); // First time through check the first three
				next_s2 = mid(s2,c + offset2+1,iif(not c,3,1)); // First time through check the first three
				// If they are equal
                if (compare(next_s1,next_s2) eq 0)
					{
						// Our longeset Common String just got one bigger
						lcs = lcs + 1;
						// Append the characters onto the "highlighted" version
						_s1.append(left(next_s1,1));
						_s2.append(left(next_s2,1));
					}
				// The next two charactes did not match
				// Now we will go into a sub-loop while we attempt to 
				// find our place again.  We will only search as long as
				// our maxOffset allows us to.
                else
	                {
						// Don't reset the offsets, just back them up so you 
						// have a point of reference
	                    old_offset1 = offset1;
	                    old_offset2 = offset2;
						_s1_deviation = "";
						_s2_deviation = "";
						// Loop for as long as allowed by our offset 
						// to see if we can match up again
	                    for (i = 0; i lt maxOffset; i=i+1)
	                    {
							next_s1 = mid(s1,c + offset1 + i+1,3); // Increments each time through.
							len_next_s1 = len(next_s1);
							bookmarked_s1 = mid(s1,c + offset1+1,3); // stays the same
							next_s2 = mid(s2,c + offset2 + i+1,3); // Increments each time through.
							len_next_s2 = len(next_s2);
							bookmarked_s2 = mid(s2,c + offset2+1,3); // stays the same
							
							// If we reached the end of both of the strings
							if(not len_next_s1 and not len_next_s2)
								{
									// Quit
									break;
								}
							// These variables keep track of how far we have deviated in the
							// string while trying to find our match again.
							_s1_deviation = _s1_deviation & left(next_s1,1);
							_s2_deviation = _s2_deviation & left(next_s2,1);
							// It looks like s1 has a match down the line which fits
							// where we left off in s2
	                        if (compare(next_s1,bookmarked_s2) eq 0)
		                        {
									// s1 is now offset THIS far from s2
		                            offset1 =  offset1+i;
									// Our longeset Common String just got bigger
									lcs = lcs + 1;
									// Now that we match again, break to the main loop
		                            break;
		                        }
								
							// It looks like s2 has a match down the line which fits
							// where we left off in s1
	                        if (compare(next_s2,bookmarked_s1) eq 0)
		                        {
									// s2 is now offset THIS far from s1
		                            offset2 = offset2+i;
									// Our longeset Common String just got bigger
									lcs = lcs + 1;
									// Now that we match again, break to the main loop
		                            break;
		                        }
	                    }
						//This is the number of inserted characters were found
						added_offset1 = offset1 - old_offset1;
						added_offset2 = offset2 - old_offset2;
						
						// We reached our maxoffset and couldn't match up the strings
						if(added_offset1 eq 0 and added_offset2 eq 0)
							{
								_s1.append(h1 & left(_s1_deviation,added_offset1+1) & h2);
								_s2.append(h1 & left(_s2_deviation,added_offset2+1) & h2);
							}
						// s2 had extra characters
						else if(added_offset1 eq 0 and added_offset2 gt 0)
							{
								_s1.append(left(_s1_deviation,1));
								_s2.append(h1 & left(_s2_deviation,added_offset2) & h2 & right(_s2_deviation,1));
							}
						// s1 had extra characters
						else if(added_offset1 gt 0 and added_offset2 eq 0)
							{
								_s1.append(h1 & left(_s1_deviation,added_offset1) & h2 & right(_s1_deviation,1));
								_s2.append(left(_s2_deviation,1));
							}
	                }
                c=c+1;	
            }
			// Anything left at the end of s1 is extra
			if(c + offset1 lt len(s1))
				{
					_s1.append(h1 & right(s1,len(s1)-(c + offset1)) & h2);
				}
			// Anything left at the end of s2 is extra
			if(c + offset2 lt len(s2))
				{
					_s2.append(h1 & right(s2,len(s2)-(c + offset2)) & h2);
				}
				
			// Distance is the average string length minus the longest common string
			distance = (len(s1) + len(s2))/2 - lcs;
			// Whcih string was longest?
			maxLen = iif(len(s1) gt len(s2),de(len(s1)),de(len(s2)));
			// Similarity is the distance divided by the max length
			similarity = iif(maxLen eq 0,1,1-(distance/maxLen));
			// Return what we found.
			return_struct.lcs = lcs;
			return_struct.similarity = similarity;
			return_struct.distance = distance;
			return_struct.s1 = _s1.toString(); // "highlighted" version
			return_struct.s2 = _s2.toString(); // "highlighted" version
            return return_struct;
        }


</cfscript>
[/code]
[code]<cfset string1 = "The rain in Spain stays mainly on the plains
				The rain in Spain stays mainly on the plains
				The rain in Spain stays mainly on the plains
				Lorum Ipsum, yadda yadda.
				Lorum Ipsum, yadda yadda.
				La La La La Luke, I am your father.">

<cfset string2 = "The rain in Madrid stays totally on the plains
				The rain in Spain stays mainly on the plains
				The rain in Barcelona stays entirely in the air
				Lorum Ipsum, Yabba dabba doo.
				Whatcha eatin?  Nutin' Honey.
				Da Da Da Duke, I am your father.">

<cfset comparison_result = stringSimilarity(string1,string2,10)>

<cfoutput>
Roughly #comparison_result.distance# characters are different between the two strings.<br>
The strings are a #numberformat(comparison_result.similarity*100)#% match.<br>
The Longest Common String is #comparison_result.lcs#.<br>
<br>
<table border="1" cellpadding="10" cellspacing="0">
	<tr>
		<td>
			#replacenocase(comparison_result.s1,chr(10),"<br>","all")#
		</td>
		<td>
			#replacenocase(comparison_result.s2,chr(10),"<br>","all")#
		</td>
	<tr>
</table>
</cfoutput>[/code]