To make sense of the enormous quantities of data generated by a city, it is imperative that information is stored properly for subsequent analysis and dissemination. When deciding what kind of storage solutions and process tools to use, there are several factors that local authorities should take into consideration.

Desired output

The starting point is always to consider the policy issues and questions to be addressed. These directly influence the types of data to be collected and their likely storage requirements. Policymakers should also clearly identify the legalities around what information can be stored, how it can be retained and where the trade-off between data retention and destruction should be.

Frequency of data updates

It is also essential to strike a balance between the frequency of data updates and the consumption of resources. Real-time data processing plays a vital role in many smart city applications, but is generally very resource-intensive. Choosing the appropriate frequency of data collection, therefore, involves a clear trade-off: consistency and accuracy of data collection versus storage capacity and cost.

Centralised versus distributed storage

Once the desired output and frequency are clear, cities often have to weigh up the trade-offs between physical and centralised storage solutions or those in the cloud. Physical storage infrastructure is likely to be more suitable to smaller databases that are confidential and highly sensitive in nature. The advantage may be on-hand security, control and customisation, but the disadvantages are typically low scalability, high maintenance costs, and the technical responsibilities of managing such a system.

An alternative is cloud storage, where unlike traditional centralised servers, which can only be accessed on site, cloud systems allow for distributed storage of data supported by a dispersed network of computers across multiple locations. Cloud systems have the key advantage of being easily scalable and flexible, as new server systems can be quickly added as and when an organisation needs. Such systems also make it easier to store data securely, as users are no longer dependent on the functioning of a single server, which, as a singular system, may be more vulnerable to cyber attacks or physical damage.

Lastly, because cities are effectively leasing server space as opposed to investing and constructing the servers themselves, cloud storage can be a much cheaper alternative to traditional methods.

There are three broad categories of cloud systems:

• public cloud,
• private cloud
• hybrid cloud. [i],[ii],[iii]

Cities use the public cloud by renting a slice of distributed data-centre infrastructure from a service provider. The advantage is its scalability (unlimited on-demand cloud resources), lower costs (no need to spend on equipment and IT maintenance) and reliability (as the services are distributed to different data centres). Yet, control over data security may be a concern.

A private cloud is a dedicated cloud infrastructure that can be managed by the city itself or by hiring a third-party service, hosted in the city data centre or off premises. The advantage is its security, control and customisation, as well as the flexibility of being able to move non-sensitive data to a public cloud when needed. The drawback is its higher cost and the responsibility involved in operating and maintaining the data centre.

A hybrid cloud is a mixture of on-premises infrastructure, as well as private and public cloud services. This mixed computing and storage environment allows data and applications to be shared between the various platforms, from private physical storage to public cloud.

Data processing

Data processing follows a structural cycle starting with the input stage – which is where collected data are converted into machine-readable formats.[iv] The next stage of processing is where raw data in the form of images, videos, text, numbers and other alternatives will be classified and analysed to yield meaningful results. This can happen either through a local server or in the cloud, depending on the authority’s needs, and once transformed this makes up the final stage of output presented to the end-user or stored for future use.

The idea of processing raw data near its point of origin (that is, the sensor) before sending it to be stored is an important consideration for municipal governments and one behind recent innovations in Edge computing.[v],[vi]The advantage is that only the processed data and essential information will be transferred over the network, requiring less bandwidth and leading to lower battery consumption by sensor devices.

Ultimately, the most appropriate storage solution for a city will depend on its budget, scale, data sensitivity and desired objectives, among other factors. Cities may look to use proprietary or national, centralised server systems where available, or cooperate with solutions providers to form their own cloud. The main advantage of cloud is its scalability and flexibility, as well as its potentially lower cost, but cities should carefully consider their capacity to uptake such systems when designing their approaches to storage.

As with data collection and transmission, governments are likely to involve the private sector with data storage and process infrastructure to varying degrees. The private sector can support the supply of vital infrastructure through traditional procurement models, and may even take on greater levels of risk sharing – for instance, through contracts for IT operation and modernisation, implementation of cloud-based solutions, and server efficiency improvements, among others.

Large-scale infrastructure such as data centres are also likely to be well placed for typical PPP models, given their high construction and operation risks, and the requirements for heavy investments in space, power, cooling, connectivity and security.[x] In this way, governments could look to lease space for their IT equipment rather than build their own facilities, and this applies equally to physical data centres as well as cloud-based operations. One thing to consider here, however, is that private-sector operators may have greater proximity and access to sensitive public data. For this reason, it is particularly critical that governments work with trusted partners who place a high degree of emphasis on the security of their systems against internal and external breaches.