Hugofox: System for providing up-to-date information on planning applications for a council and club oriented CMS system
Maintaining near-real-time planning data involves a novel use of Selenium browser control via API
Above: the Hugofox home page.
Background
Hugofox is a SaaS (software as a service) offering that has specialized features for UK parish councils as well as clubs, societies and other not-for-profits.
Brief
The majority of UK planning applications are available online through a variety of county council web sites. Without any public APIs, we needed to devise a way to collect this data from hundreds of council systems nightly via public pages, deposit it into a database and then make this data available on parish council web sites built on the client's CMS SaaS.
The solution
For power and reliability, we opted to build a server side Windows tool that could be run either manually via a graphical interface, or via command line (triggered by scheduled task or script). We used the latest .NET 6 Core version, coding in C#.
We used a Microsoft SQL Server db to store the planning locations that needed to be checked each night, as well as to store the incoming data retrieved from the sources.
Planning authorites deploy a number of different software packages to run their planning web sites, so we support obtaining data from a number of different providers, as well as some custom implementations for planning authorites running "home brew" software.
One of the key difficulties that arose early on was that most planning authority systems were specifically designed to be used by humans, via a browser. Javascript and other browser features were often required, so simply connecting a tool directly to pull data would not work. We therefore coded the Planning Tracker app to use Selenium, a widely supported testing tool for browsers. This allows our Planning Tracker software to control a Chrome browser installed on the same server to collect data from web sites, and to extract data from specific locations on the pages. We also had to overcome a number of other issues the could block or frustrate data collection. We monitor the data coming in so we can focus on any problem sources quickly, before users alert us of an issue. Maintaining the widest coverage as possible across the UK is a constant battle as of the hundreds of source council web sites, a few change systems or make other changes each month.
Above: the graphical user interface (GUI) of the Windows application that runs on the main server.
On the Parish councils' Hugofox web sites, local planning applications can be queried by distance and type.
Above: planning applications listed on a client's parish council web site.
The status and any decisions can be viewed, as well as a map of the location produced via Google Maps.
Above: clicking through on a single application to view more details and location map.
Conclusion
The Planning Tracker software has proven extremely reliable, and the modular nature of the .NET 6 code makes it easy to extend and update when required. Detailed logging and reporting gives us information to resolve issues in any feed without client notification or assistance. The incorporation of Selenium allows us to mimic how a human interacts, thereby accessing systems where machine access has never been a consideration.