I’ve posted some charts in previous editions of the newsletter - gathering, cleaning and parsing data into readable format so I can chart multiple variable and infer some hypothesis takes more time and effort than actually writing the analysis.
My workflow is usually something like this:
find interesting endpoint ( usually public REST API )
either download historical data or
set up a crontab ( program that runs on say 10 minutes intervals ) to download the current data into the database
perform some operations on data ( usually multiplying or calculating the difference )
plot with matplotlib ( python charting library )
And all of the work above just to create 1 single image/chart.
I’ve decided to automate boring parts: I’ve created small flask ( web framework ) app that: gets url, then it would flatten the json and download the data each 10 minutes. Do you know some interesting endpoints that you would like to add?
Feel free to play with it! Let me know which API to track and which features to add. ( right now I’m working on making the charts more readable )
( Link below )