Briefing
This project is to develope a system that can colloect, merge, clean and display automatically for user monitoring price changing.
Collecting and Merge
The data is downloaded from Actual Price Registration of Real Estate(abbr. APRRE). It is zip file, so the process should decompress than can merge. Goverment upload new data every 10 days. To avoid the duplicate data, checking mechanism is required also. As the data is reserved and keep for the next check, so it can check top five data from both the new file and the last downloaded file. If it isn't dulplicate than proceed to merge.
Cleaning
Some data is null or incomplete, parts of column is not the right attribute. Thus, following steps are required:
Redefine attribute
Fill in data if null
Complete those incomplete (e.g. data)
Convert measurement (i.e. square meter --> 坪)
Insert new column (e.g. $ per sqaure meter-->$ per坪)
All these are done by Power Query.
Organizing
Means to build relations between datasets.
For user can browse with the map, a dataset with longitude and latetude is loaded and the key is Administrative Region.
Visualizing
The dashboard is generated by Power BI.
The main part is box plot, bar chart and line chart. Upper left are filters. Lower left is table showing detail of transactions.
Summary
This project is composed of Python and Power BI. Since the progress should be as automatic as possible, also the amount of data is huge, Pyhton is suitable it the project.Yet Power Query is easy use to use when cleaning and organizing, so does visualizing, 1/2 of process is done by Power BI.
The code is activated by manual, should Power Automate be adopted in the future so the code can be totally executed automatically by "time".
Comments