Scrapy installation on CentOS and Windows

Scrapy installation on CentOS and  Windows

Scrapy is an application framework supporting development of applications in a given environment. I discuss here the steps of installation of Scrapy both CentOS and Windows environments including installation of the dependencies thereof.

Scrapy Installation on Centos 6.5

Scrapy needs python 2.7 and above to run in CentOS . CentOS 6.5 comes with Python 2.6 .So we need to install python 2.7+ to run Scrapy code. Here are the steps to install Python 2.7.11. Firstly install the Scrapy dependencies, preceding installation of Python 2.7.11.

yum -y update  
yum groupinstall -y 'development tools'
yum install -y zlib-devel bzip2-devel openssl-devel xz-libs wget python-devel libxml2-devel libxslt-devel pyOpenSSL gcc libffi libffi-devel sqlite-devel sqlite

Installing Python 2.7.11

tar -xvzf Python-2.7.11.tgz

# Enter the directory:
cd Python-2.7.11

# Run the configure:
./configure --prefix=/usr/local

# compile and install it:
make altinstall

# Checking Python version:
[root@nicetry ~]# python2.7 -V
Python 2.7.11

#Environment variables
Add below line in .bashrc file and run source .bashrc
export PATH="/usr/local/bin:$PATH" 

Install setup tools


wget --no-check-certificate

# Extract the files:
tar -xvf setuptools-1.4.2.tar.gz  
cd setuptools-1.4.2

# Install setuptools using the Python 2.7.8:
python2.7 install

Install PIP (a package management system used to install and manage software packages written in Python.)

curl | python2.7 -

pip -V
pip 8.1.1 from /usr/local/lib/python2.7/site-packages (python 2.7)
[root@hivestore1-srv1 data-ingestion]#

Install Scrapy

CFLAGS="-O0" pip install lxml
pip install scrapy

Once Scrapy installation is done check “scrapy shell <URL> ” from command line

Scrapy Installation on Windows

Python installation

Install Python 2.7.11 from

Update the PATH ; open  Command prompt and run:

c:\python27\python.exe c:\python27\tools\scripts\

Close the command prompt window so that changes become effective. Reopen command prompt and run the following command and to check the Python version:

python --version

pywin32 installation
Install pywin32 from

Be sure you download the architecture (win32 or amd64) that matches your system [even our system is 64 bit also we need to download 32 bit only (example: pywin32-219.win32-py2.7.exe)]

PIP Installation
Install pip from
Now open a Command prompt to check pip is installed correctly:

pip --version

LXML package installation
Install LXML package from below

Install Visual C++ Compiler
Install “Microsoft Visual C++ Compiler for Python 2.7” from
Scrapy installation
Ensure Python 2.7 and pip package manager working, then install Scrapy:
pip install Scrapy
pip install happybase
pip install requests
pip install OrderedDict
pip install html2text

Add PyDev perspective

Install Eclipse Juno or STS then add PyDev perspective to write python code

Once  Scrapy installation is done check “scrapy shell <URL> ” from command line

Leave a Reply



6 April 2020 | 11:00 AM PST

Join the Standav Team & Frank Sohn from Novus CPQ, with Special Guests from Bayer, Box, Extreme Networks, Miro, Mobileiron, and Ross Video as we discuss CPQ, Billing, and Pricing Solutions for the Enterprise.

You have successfully registered! You will receive the details to join the conference.