VB6: Convert HTML into Searchable Text using Regular Expressions

OK, So what I need to do is remove HTML mark up tags and unnecessary whitespace from HTML so that it can be indexed for a searching algorithm

To do this, I will mostly be using VB's own Replace function, and the Scripting Regular Expression Object available by adding a Visual Basic project reference to: Microsoft VBScript Regular Expressions 5.5

The Plan

• Swap <BR> for a new-line so that occurances of "hello<br>goodbye" don't become "hellogoodbye"
• Swap &nbsp; for a space 
• Remove any HTML tags and replace with nothing
• Remove multiple whitespace characters and swap for a single space
• Handle translating common named entities such as &amp; &gt; &lt; etc.

Further Reading

For more information, see Microsoft Beefs Up VBScript with Regular Expressions

Code

Private Function HTMLClean(ByVal strText As String) As String

 

    Dim objRegEx As VBScript_RegExp_55.RegExp
   
    ' replace <br>'s for a newline
    strText = Replace$(strText, "<br>", Chr$(10), 1, -1, vbTextCompare)
   
    ' replace non-breaking spaces for a space
    strText = Replace$(strText, "&nbsp;", Chr$(32), 1, -1, vbTextCompare)
   
    ' create new regex object
    Set objRegEx = New VBScript_RegExp_55.RegExp
    objRegEx.Global = True ' don't just operate on first find.
   
    ' remove HTML tags
    objRegEx.Pattern = "<[^>]*>"
    strText = objRegEx.Replace(strText, "")
   
    ' ditch excessive white space
    objRegEx.Pattern = "\s+"
    strText = objRegEx.Replace(strText, " ")

 

    ' thanks, bye
    Set objRegEx = Nothing

 

    ' named-entities
    strText = Replace$(strText, "&gt;", ">", 1, -1, vbTextCompare)
    strText = Replace$(strText, "&lt;", "<", 1, -1, vbTextCompare)

 

    ' insert your favourite named-entities here.
    strText = Replace$(strText, "&amp;", "&", 1, -1, vbTextCompare)  ' must do last
   
    ' return
    HTMLClean = strText
   
End Function

Debug.Print HTMLClean("<greets>Hello &lt;oh&gt;&nbsp;&amp;" & vbNewLine & _

                      "<b>I</b> <a href=""#"">wonder</a>  " & vbNewLine & _

                      "   what<br>will become of all this?</greets>")

Hello <oh> &I wonder what will become of all this?