An Arabic auto-indexing system for information retrieval system. (c2002)

Abstract

In this report, a model is proposed for performing auto-indexing text documents. The model consists of four layers that are interdependent. This work tackles the problem of auto-indexing Arabic documents. However, due to this interdependency of the proposed model, the algorithm in this report may be applied to documents in any language. The only thing to change is the layer (or programmatically speaking: the module) that extracts the words to their original stem word. Obviously, that would require the Arabic grammar to be taken into consideration. In addition, this report introduces a new concept to calculate the weight of a term relevant to its container document. Traditionally, the weight of a term used to rely totally on the rate of repeal (or the count) of that term. The new innovation is to take into consideration the rate of "spreading" within the document. In other words, if a certain word is concentrated at a specific part of a document, then it is less likely that this word reflects its document had it been more spread in the document. This assumption is mathematically proven, and is illustrated by real examples.