The actual website we need to build involves many features. We need to convert the raw csv files into tables and save them to the postgreSQL database. Then we need to use DBT to write models to analyze the data, create sub-tables, and generate the final tables that needed to send to the front end to display. We also need to write servers that connect the backend to the frontend. Moreover, we need to create different plots and graphs for the corresponding tables sent from the back end server, and allow users to change the parameters to see the changes of the plots. Since it has so many features, we decided to start with building a simpler app first. We decided to build a simple app that uses all the necessary tools (postgreSQL, DBT, Svelte, and D3) but with limited functionalities.

Building an app in Svelte is not very difficult, but building an app with database, servers, DBT, and front end Svelte all together is harder than we expected. First, we used a demo csv file as our raw data. We then wrote some DBT models to select some rows and columns from this raw data and generate new tables that ready to be sent to the front end. We then used python and Flask to write our routes. In the server, we also connected it to our Docker container that has our postgreSQL and DBT instances. We also set the routes to be able to select the tables we need to display, and make it able to call the DBT to run.

After we finished writing the DBT models and servers, we then created some tables and graphs in Svelte which fetches the data from postgreSQL database and displays them on the website. For the graphs, we used D3.js as the tool to build the plots. We used d3.json to retrieve the data, and create the corresponding graph that represents the data it retrieved. We also created a new post route that allows user to submit their comments to our database, and retrieve and print them in the table as we need this route later to store the customized parameters for each user.

While we started to create this app from scratch, we also wrote a detailed step-by-step guidance on how to create new DBT project and connect it to the Docker container.