Reddit Data Collector is a Python package that allows a user to collect post and comment data from Reddit. It is built on top of the Python module PRAW, which stands for "The Python Reddit API Wrapper". It aims to make it very simple for a user to collect data from Reddit for further analysis (e.g. Natural Language Processing), without having to learn the inner workings of PRAW or the Reddit API.
It is currently maintained by Nico Van den Hooff.
Reddit Data Collector requires Python and:
- pandas (>=1.3.5)
- praw (>=7.5.0)
- tqdm (>=4.62.3)
The recommended way to install Reddit Data Collector is using pip:
pip install reddit-data-collectorPlease see the examples directory for step by step instructions on how to use Reddit Data Collector.
- Official source code repo: https://github.com/nicovandenhooff/reddit-data-collector
- Downloaded releases: https://pypi.org/project/reddit-data-collector/
- Issue tracker: https://github.com/nicovandenhooff/reddit-data-collector/issues
You can check the latest sources with the command:
git clone https://github.com/nicovandenhooff/reddit-data-collector.gitTo learn more about making a contribution to Reddit Data Collector, please see the contributing file.
- Add ability to collect images from Reddit posts that contain them.
- Add author information to post and comment data, currently the Reddit API is inconsistent with suspended and deleted author data, so this functionality has not been built in yet.
- Add plotting module that creates useful visualizations of the data that has been collected
- Add preprocessing module that cleans up the posts and/or comment data collected
After installation, you can launch the test suite, which is contained in the tests/tests.py. Note that you will have to have pytest >= 6.2.5 and pytest-cov >= 3.0.0 installed. You can launch the test suite by following these steps from the projects root directory:
- Open up
tests.pywith the following command:
open tests/tests.pyComment out lines 24 to 30. Change the values in DataCollector() in line 32 to your Reddit credentials.
- Run the following command to run the tests:
pytest tests/tests.py- If desired, run the following command to show test coverage:
pytest --cov=src tests/tests.pyThe project was started in January 2022 by Nico Van den Hooff as a side project while he was completing the UBC Master of Data Science Project. Nico wanted to obtain a sample of posts and comments from Reddit, but noticed that while PRAW existed and provided seamless access to Reddit's API, there was no package available that allowed for a simple method to collect this data.
Certain sections of this README file was inspired by the scikit-learn README.