Text preprocessing pipeline

Text preprocessing pipeline is a command-line tool in Python that cleans a CSV of raw text for natural-language processing:

python client.py raw.csv processed.csv

Each text goes through four steps:

1. cleaning: lower-casing and removing punctuation and noise;
2. tokenising into words;
3. removing stop words, such as "the" and "and";
4. stemming, so "running" and "runs" both become "run".

Helpers save the processed data and plot the most frequent words. The repository includes tests, a notebook and sample data.

See also

• Natural language processing
• Machine learning and data
• Intro to machine learning