Common words like "the" or "that" will have high term frequencies, but when you weigh them by the inverse of the document frequency, that would be 1 (because they appear in every document), and since TfIdf uses log values, that weight will actually be 0 since log 1 = 0. You can read ton of information on text pre-processing and analysis, and there are many ways of classifying it, but in this case we use one of the most popular text transformers, the TfidfVectorizer.Ĭompared to a Count Vectorizer, which just counts the number of occurrences of each word, Tf-Idf takes into account the frequency of a word in a document, weighted by how frequently it appears in the entire corpus. The text processing is the more complex task, since that's where most of the data we're interested in resides. Vectorizing text with the Tfidf-Vectorizer Most of them wouldn't behave as expected if the individual features do not more or less look like standard normally distributed data. This is a common requirement of machine learning classifiers. We process the numeric columns with the StandardScaler, which standardizes the data by removing the mean and scaling to unit variance. Here are the ones I use to extract columns of data (note that they're different for text and numeric data): from sklearn.base import BaseEstimator, TransformerMixinĬlass TextSelector (BaseEstimator, TransformerMixin): def _init_ (self, field):ĭef transform (self, X): return XĬlass NumberSelector (BaseEstimator, TransformerMixin): def _init_ (self, field):ĭef transform (self, X): return X] Transformers must only implement Transform and Fit methods.
You can build quite complex transformers, but in this case we only need to select a feature. Incorporating it into the main pipeline can be a bit finicky, but once you build your first one you'll get the hang of it.Įach feature pipeline starts with a transformer which selects that specific feature. The reason we use a FeatureUnion is to allow us to combine different Pipelines that run on different features of the training data. ( 'clf', XGBClassifier(max_depth= 3, n_estimators= 300, learning_rate= 0.1)), ( 'wordext', NumberSelector( 'TotalWords')), ( 'svd', TruncatedSVD(algorithm= 'randomized', n_components= 300)), #for XGB ( 'tfidf', TfidfVectorizer(tokenizer=Tokenizer, stop_words=stop_words, I'll post the pipeline definition first, and then I'll go into step-by-step details: from sklearn.pipeline import Pipeline, FeatureUnionįrom sklearn.feature_extraction.text import TfidfVectorizerįrom sklearn.preprocessing import StandardScalerįrom composition import TruncatedSVDįrom sklearn.ensemble import RandomForestClassifier While you can do all the processing sequentially, the more elegant way is to build a pipeline that includes all the transformers and estimators. X_train, X_ test, y_train, y_ test = train_ test_split(X, Y, test_size=0.25) Skipping over loading the data (you can use CSVs, text files, or pickled information), we extract the training and test sets for Pandas data: X = df]įrom sklearn.model_selection import train_ test_split
It's very similar to sentiment analysis, only we have only two classes: Positive and Neutral (which also includes Negative).Īs an additional example, we add a feature to the text which is the number of words, just in case the length of a filing has an impact on our results - but it's more to demonstrate using a FeatureUnion in the Pipeline. For more background, I was working with corporate SEC filings, trying to identify whether a filing would result in a stock price hike or not. The problem is very simple, taking training data represented by paragraphs of text, which are labeled as 1 or 0.
I'll focus mostly on the most challenging parts I faced and give a general framework for building your own classifier. In this first article about text classification in Python, I'll go over the basics of setting up a pipeline for natural language processing and text classification.
0 kommentar(er)
