This week is the last week of our DREAM program, so for the first two days, I helped my teammate to implemented another type of D3.js graphs, heatmap. We successfully created a customized heatmap that could show different colors which indicate different rates. We also displayed the actual number in each grid. After we finished implementing the heatmap, I started to write very detailed comments in my code explaining what does my code do. I explained what is each variable, function, and component, what are they used for, and how can we make change on them. I also wrote a detailed documentation on how to use the components I wrote for the graphs, and how can they add their own graphs. I explained what are the parameters we need to pass in for each graph, and how to insert the components into our web app. I explained my documentation in our group meeting as well, so people who would join our team later will be very easy to pick up my progress and continue building more features if they want.

For last week, we were able to make all the width and height to be stored in the Svelte Store without need to pass the width and height into the component each time when we call the component. We were also able to link the slider with the corresponding graph component, meaning that the width of the slider will match with the component automatically. However, there were still some part we could make it better. One thing is that right now, we still need to manually enter the width and height of each individual graph. We have to manually enter the pixel number for the graph in the Svelte Store, which is not an optimal design. Moreover, it is not responsive. The graph size is fixed. When we resize the screen, the graph and sliders will not change their size. As a result, if people use their phone to open this website, they will have a very large graph that the screen won’t fit. Therefore, for this week, we worked on making the graph responsive without manually entering the width and height. To do so, we first tried to detect the screen size by using Svelte component Window which binds to the innerWidth (finding the browser scrreen size). We then made the width and height in the Svelte Store depend on the innerWidth, so it won’t be a fixed number, and we will not need to manually enter the width and height for each graph. Then we used d3 to pass out the x and y coordinates for each point on the graph to the Svelte Store to better align the sliders with the graph. After that, we now made the graph fully responsive and easy to use.

Even though we are now able to create the Svelte Components for the D3 Graphs and it is easy for other user to write their own graphs, there are still a few issues we need to fix. One problem is that we have to manually enter the height, width, absolute top, absolute left to create the d3 graph. Another problem is that we need to do the same for the sliders, and what Professor Wu want us to do is to link the slider with the graphs the slider could affect (able to change the parameters for that graph), and make them having the same width. Therefore, this week, we tried to fix these issues.

Last week, we were able to create the javascript functions that we can call to generate different graphs. This week, Professor Wu, our advisor, suggested us to change our existing javascript functions to Svelte components to make it more user friendly. For the javascript function we previously wrote, in order to implement and use these functions, we will need to find the proper place in the svelte script to locate the functions and pass the parameters in the correct order. If we switch the order we pass in the parameter, the graph will not be able to generate correctly. Also, for d3 in javascript, we will need to manually clear the SVG plots and redraw it again whenever we change the parameters in the sliders. After changing the functions into Svelte components, we will just need to place the component in the html body by the order we want to display them. Another advantage is that we do not need to pass the parameter in the exact order. For svelte components, we can now pass the parameters in as key value pairs. Also we can use Svelte {#key} blocks to automatically refresh the d3 content without us manually create the clear SVG functions. It is much more convenient now for other users to create their own d3 graphs.

After making the basic app working correctly and able to change the graphs dynamically as users make changes through the dropdown menu or the sliders, we started to organize and clean our current code and tried to make it more concise, easy to read, and reusable. For now, the D3 function is just a block of code in the main svelte file, and the main svelte file is containning thousands of lines of code. It would be very difficult to find it, and it would be difficult for other people to read it as well. I started to create separate javascript functions in different files for each of the graph. We have created 3 different files for the 3 different type of graphs. For this way, the main file is a lot shorter and easy to read, and also it would be very easy for people to add new graphs. Since we are making the d3 components into javascript functions, whenever we want to add another graph, all we need to do is to call the function and pass in some necessary parameters. Originally, if we want to add a new graph, we will need to copy and paste the previous code, find all the places that we need to make changes, and manually make the updates. It would be really easy to make mistakes, and it is not user friendly. But now, after we organized our file, if we want to add a new graph, we just need to type the function name, and type in the parameters such as the title name, x axis name, y axis name, and etc, and it will automatically update everything for us, so that even people don’t know a lot on coding would be able to create their own graphs.

Last week, we were able to create the page that can connect to the server, call and retrieve data from DBT models and postgreSQL database, and display all three graphs. This week, we are focusing on adding sliders, dropdown menus, and input boxes to allow users to choose different regions, use different paarameters for calculation, and show the corresponding plots dynamically. To do so, we first created a dictionary that contains all necessary parameters we need for the sliders and dropdown menus, and created a function to update the dictionary whenever there is a change in the sliders or the dropdown menus. Then we added an updated_variable route to allow us to pass the new parameters to the database and execute the change. We used some Svelte components from the Zambia app and changed some parameters. These components are reusable, and if someone want to use it, all they need to do is to pass some variables in it, and it will create the components automatically. We also created two functions. One called clearSVG, and the other one is called createSVG. For D3, when we change the data and re-draw the graph, it will not automatically clear the previous data. Instead, we need to write a clearSVG function, and call it each time when we make any change. Same for createSVG, we need to call it after we call the clearSVG function.

Last week, we were able to analyze the raw data and generate the final tables we need to send to the server (combined payout, each individual payout, and capped climatology). This week, we are starting to build the actual server and frontend plots. We first wrote our server via Python and Flask. It is similar to the demo app we did two weeks ago. We wrote more routes to retrieve different tables. We also did a little bit data cleaning too to make our data easy to use. For this week, we will not make the website dynamic, which means user would not be able to change parameters through sliders or dropdown menu to see different results. We will just try to display the table we have in backend. Therefore, we did not write the update_variable route as yet.

After we built the sample app that combined database, DBT, Svelte, and D3, we are about the build the actual website to display the index insurance analysis. The team members who joined the project last year already built a website for Zambia, and they are currently trying to add more components and analytical results to it, and reducing the data loading time as it could take a very long time to load and analyze the data. Professor Wu suggested us to build a new app for a different country, Ethiopia, where the Financial Instruments Sector Team used different parameters and calculations to calculate the payout rate, resulting different DBT models and analysis.

The actual website we need to build involves many features. We need to convert the raw csv files into tables and save them to the postgreSQL database. Then we need to use DBT to write models to analyze the data, create sub-tables, and generate the final tables that needed to send to the front end to display. We also need to write servers that connect the backend to the frontend. Moreover, we need to create different plots and graphs for the corresponding tables sent from the back end server, and allow users to change the parameters to see the changes of the plots. Since it has so many features, we decided to start with building a simpler app first. We decided to build a simple app that uses all the necessary tools (postgreSQL, DBT, Svelte, and D3) but with limited functionalities.

For this project, we need to develop an open-source toolkit for index insurance, which we need to deploy a database, analyze the data, and show the visualizations via tables and graphs so that users are able to change certain parameters to see different calculations and results of the index insurance in different circumstances. Before we start the project, Professor Wu suggested us to learn Svelte, a JavaScript library for front end, and DBT, a data transformation tool that lets people to deploy analytics code for SQL database. After we learned more about Svelte and DBT, we then built a simple project in Svelte, which could generate a 24x24 grid system, ask users to choose the width and height of a rectangle, and then creates the corresponding rectangle, display it in the grid system, and allow users to click up, down, left, and right buttons to dynamically move the rectangle. We built this simple project to make ourselves familiar with Svelte and be comfortable using it later.