Statistical Identification of Domain-Specific Keyterms for Text Summarisation

Abstract

We believe that in order to be useful a text summarisation technique must be domain dependent, in that the resulting summary must cover the important aspects and concepts specific to the subject matter’s domain. The main problem with a typical domain-dependent text summarisation technique is the cost of acquiring and hand-coding the required domain-specific knowledge into the system, e.g., in the form of phrase-structure templates. To solve this problem, we propose a solution which uses automatically retrieved sample documents as the source of the domain-specific knowledge, and extracts the knowledge in the form of keyterms. These keyterms represent the key aspects and concepts (terminology) relevant to the input document. The sample documents are retrieved from a collection, called base collection-containing documents of various topics, based on their similarity with the input document. The input document is then summarised by extracting a number of sentences containing the keyterms.

Our text summarisation technique is based on the statistical distribution of words among documents in the base collection, within individual documents, and among sentences in the input document. In particular, statistically-based formula are employed for scoring each of the candidate sample documents, keyterms, and key sentences. Our technique makes use of standard word or term distribution parameters that are commonly provided or can be easily obtained through the use of modern text retrieval systems.