As far as video surveillance is concerned, data has two aspects of content – massive and unstructured. The amount of video surveillance data is huge, and with the trend of high-definition and ultra-high definition, the scale of video surveillance data will grow at a faster exponential level; unlike the structured data, the video surveillance service generates the most data. Most of them are based on unstructured data, which poses a great challenge to traditional data management and usage mechanisms.
video surveillance data faces three problems
1, Storage
Video surveillance data is characterized by high concurrency and large capacity. Taking 1080P as an example, at a bit rate of 8 Mbps, each camera produces about 84 GB of video data per day. The monitoring scale of a medium-sized city is generally thousands to tens of thousands of cameras, and these data generally require 30 days to be stored in the system. the above. The storage system must also have a high level of fault tolerance. The failure rate of the storage medium is usually high, but the failure should not cause the loss of monitoring data. In addition, as the security project itself continues to evolve, the system may need to be expanded and upgraded online, which requires a high degree of scalability of the storage system to easily and easily add storage devices to the system.
2, Sharing
Big data requires rapid extraction, discovery, and analysis to extract value from large, multi-category data. The most striking feature of the security big data era is the massive and unstructured data sharing to improve data processing capabilities. Compared with scientific computing and the Internet, big data processing for video surveillance is particularly difficult. First, video recording is a more primitive non-text unstructured data. It must undergo complicated and arduous analysis to extract text structured data. The next step is processing; secondly, video recording is several orders of magnitude larger than other forms of data and has a large bandwidth requirement for transmission, storage, and calculation.
3, Security
The construction of safe cities and smart cities has promoted the new application of security storage technology. A major requirement of smart cities is to link and share video storage data with each other. At the same time, video surveillance data has the characteristics of high privacy and strong confidentiality, which is not only the basis for subsequent tracing but also the basis for subsequent data analysis and mining. Therefore, we say that data security refers to the intrusion and illegal acquisition of external data, and on the other hand, the robustness of the huge system and the system fault tolerance mechanism. When the hardware-software fails, the data can still be recovered and saved. In the face of the storage and sharing of massive data, hardware and software devices carry great risks, so how to build large-scale, massive video surveillance storage systems, data analysis systems, and fault-tolerant redundancy mechanisms is a difficult problem.
The big data storage pattern
As a data carrier and driving force, storage systems are the most critical core of the big data infrastructure.
Traditional data centers, whether in terms of performance, efficiency, investment income, and security, are far from meeting the needs of emerging applications. Data center services are in urgent need of new big data processing centers to support them. In addition to traditional high reliability, high redundancy, and green energy conservation, the new big data center needs to have a series of features such as virtualization, modularization, flexible expansion, and automation to meet the needs of applications with big data features. These unprecedented demands have brought unprecedented changes to the architecture and functionality of storage systems. Obviously, the NVR architecture’s storage is mainly for small HD surveillance and cannot cope with big data needs. The following two types of big data storage architectures are described.
Storage system under SAN architecture
The platform SAN architecture is mainly for medium and large-scale high-definition surveillance systems, with hundreds of thousands or even tens of thousands of front-end channels. Generally, FCSAN or IPSAN is used to build a high-definition video storage system. As an important part of the monitoring platform, the front-end monitoring data is stored in the SAN through the video storage management module. The architecture of this type of architecture has a higher elevation of the front-end NVR than the node NVR. It has fast and convenient scalability and mature technology.
FCSAN is widely used in industry users and closed storage systems, such as high-level monitoring projects at the county or prefecture-level cities. The concurrent reading and writing of large data volume pose a big challenge to Gigabit network switching, but the application of FCSAN is relatively independent. The storage subsystem can effectively solve the above problems.
It is worth pointing out that FCSAN adopts a hierarchical centralized storage scheme. Video data is stored in different sub-platforms by region and is usually stored using RAID. However, RAID is difficult to achieve an ideal balance in terms of performance, utilization, and reliability, and the cost is high. Beijing Smart Warehouse Storage Technology Co., Ltd. is committed to providing customers with cost-effective, high-value storage products and solutions in the era of big data. We are the only manufacturer in China that has RAID controller technology such as Fibre Channel technology, hardware RAID technology and embedded technology. It completely solves the price bottleneck, performance bottleneck and stability bottleneck that plagues security monitoring customers.
For IPSAN, the data concurrent read and write transfer rate is consumed in the ISCSI link. Because this hierarchical centralized storage enters the security industry earlier, it is still favored by many customers.
Architecture cloud storage
The core application of big data is for intelligent analysis, just as storage is the core technology of big data. In the monitoring system, storage and transmission problems are the primary difficulties. A large amount of useless video information is stored and transmitted, which wastes storage space and increases bandwidth. The purpose of the intelligent analysis is to reduce the bandwidth required for video storage and relieve bandwidth pressure. Or for some useless video, the low-stream method is used for compression or transmission, which is more convenient for the whole system to investigate or query, and improve the application value of monitoring in big data.
For an intelligent analysis of big data, cloud storage architecture is the best solution. However, this application is currently only used in the Ping An City project, and the maturity of the technology needs to be improved compared to the SAN structure.
The conclusion
Exploring big data security, the core is to grasp the challenges that big data brings to the security industry, and choose which storage methods to solve the problem of data management. However, applications have just begun, monitoring storage patterns have also changed in the era of big data, we can wait and see!