When considered together, the features of a document describe a multidimensional feature space on which machine learning methods can be applied.įor this reason, we must now make a critical shift in how we think about language-from a sequence of words to points that occupy a high-dimensional semantic space. For text, features represent attributes and properties of documents-including its content as well as meta attributes, such as document length, author, source, and publication date. Each property of the vector representation is a feature. In text analysis, instances are entire documents or utterances, which can vary in length from quotes or tweets to entire books, but whose vectors are always of a uniform length. Representing documents numerically gives us the ability to perform meaningful analytics and also creates the instances on which machine learning algorithms operate. This process is called feature extraction or more simply, vectorization, and is an essential first step toward language-aware analysis. In order to perform machine learning on text, we need to transform our documents into vector representations such that we can apply numeric machine learning.

Machine learning algorithms operate on a numeric feature space, expecting input as a two-dimensional array where rows are instances and columns are features.

Text Vectorization and Transformation Pipelines

Free vectorize