What’s the right approach to building a Data Analytics dashboard?

Recently I have been assigned a project to build a dashboard for displaying records of transactions as analytics sorting out by daily volumes and values. I am already familiar with Python and SQL and currently learning JavaScript. I have been given an API endpoint to receive real time data that gives me JSON format numerical data along with proper fields. I need to to know the right approach to do the below.

  1. Extract data from API (I have already tried to write a Python script for this)
  2. Store the data in an accessible manner for other apps and scripts.
  3. Analyze the data real time with proper ordering methods.
  4. Display the analyzed data on a web app using graphical charts and maybe in text too.

I have thought of using Python to extract data and analyze it and use CSV or something similar that doesnt deal with huge databases to store data and JavaScript to plot the data on the web site.

I have to do this from scratch without using third party analytical services. I need to know how to begin and what other resources I should use to build this.

Hope somebody can give me a few guidelines. Appreciate your help a lot.

1 Like
  1. you can use requests or even standard urllib.request to get data. Usually API doesn’t check web browser hearder User-Agent, it doesn’t display recaptcha, and it doesn’t run JavaScript so code doesn’t need special modules like BeautifulSoup, lxml, Selenium, scrapy.

  2. sometimes storing in database can be much useful then csv. You can use SQL to select/filter data directly in database and it uses code created in C/C++ which is also optimized for some operations so it may work faster then reading from CSV and filtering in Python. Eventually you should use pandas which has many functions (also created in C/C++) which can run faster then pure Python and you don’t have to write them from scratch.

  3. it may need JavaScript (AJAX or websocket) to get data from server in real time and update web page without reloading page.

  4. Every dynamic/interactive plot uses JavaScript. Python can generate plot with plotly, D3, bokeh but they may also need to write JavaScript code to update data in real time.


Maybe check Full Stack Python and pages like

There are also pages about databases, web frameworks, WSGI servers, etc.

© Copyright 2013-2021 Analytics Vidhya