XTreePath: A generalization of XPath to handle real world structural variation

We discuss a key problem in information extraction which deals with wrapper failures due to changing content templates. A good proportion of wrapper failures are due to HTML templates changing to cause wrappers to become incompatible after element inclusion or removal in a DOM (Tree representation of HTML). We perform a large-scale empirical analyses of the causes of shift and mathematically quantify the levels of domain difficulty based on entropy. We propose the XTreePath annotation method to captures contextual node information from the training DOM. We then utilize this annotation in a supervised manner at test time with our proposed Recursive Tree Matching method which locates nodes most similar in context recursively using the tree edit distance. The search is based on a heuristic function that takes into account the similarity of a tree compared to the structure that was present in the training data. We evaluate XTreePath using 117,422 pages from 75 diverse websites in 8 vertical markets. Our XTreePath method consistently outperforms XPath and a current commercial system in terms of successful extractions in a blackbox test. We make our code and datasets publicly available online.