Understanding India in one map

The aim of this project is to visualize all the villages and cities in India in one map. Here is the data that inspired this project:

 

There are 245mn families in India, out of which 180mn live in villages

source

This figure is from the latest census released by the government in 2011. The first thought that came to my mind when I got to know this figure was, how little do I know about my own country. My best guess would have been 40-60 ratio of people living in villages to people in cities. This figure has changed the entire picture of India that I had in my mind. So I decided to actually paint a new picture of India. I wanted to see what India actually looks like. Hence the project.

Btw this is an example of what you get when you search for such a picture of cities and villages in India, mapped side by side, online now:

so not a good set

Gathering Data

The data that I wanted was basically labelled data of the lat-long of every village and city in the country. I could not find and readymade data source so I decided to build my own. I could get lat-long information from Google’s geolocation api if I know the name of every village and city in the country. No such unified list of names of cities and villages exists but the data from census’ website can be parsed and combined to make that list so I did just that.

Step1: Manually download all the Primary Census Abstract tables from census website

manually downloading 640 files was a pain

each file came like this, I was only interested in extracting level and name information from all the files

Step2: Generate lat-long data for every location and store it in a CSV file

To get the lat-long data I used the geolocation api from google. I created a CSV file like this:

Name, Level, Bounds and Lat-Long data of every village and city in India

Tricky situation: There are more than a million locations (cities/villages) in India. Google lets you query only 2500 in a day.

There are a couple of ways to extend the limit of queries a little bit but none would have suffice my situation of querying more than a million times. I had no option but to split my queries across days. Save a query program in a cloud server and run it to make 2500 queries everyday and update the database. This would also mean that my project will enjoy a gradual completion process. Everyday new data would come in and complete the picture a little further. Even though this came out of constraint but I think it is nice that the true picture of India did not reveal itself in one shot but would gradually get completed on its own over months.

Final Setup:
  • All the code is uploaded on Digital Ocean
  • DataParser is a program that will run periodically every day and add more latlong data to the database by talking to google geolocation api
  • The database is a local JSON file
  • The web app is running forever on the server revealing the map
Here is the link to the map: http://174.138.84.113:8000/
Here is the github: https://github.com/kshivanku/IndiaMap_Complete  and https://github.com/kshivanku/IndiaMap_bkend
Here is what the map looks like at the moment:

Currently drawing Andra Pradesh. Clearly villages will overshoot districts with a lot.

Each village or district is represented by a block

 

This is what the parsed data looks like. Would be a great resource to share once the data for all the villages and cities is added

 

Fin.

One thought on “Understanding India in one map

Leave a Reply