搭建大数据伪分布式环境，将用户手机信令数据作为数据集，数据的处理主要经过数据输入、数据清洗、数据缓冲、数据处理、数据存储和数据可视化六步。首先采用Flume的Taildir Source清洗手机信令数据，通过设计数据处理算法，利用Spark编程处理，之后将处理结果存入redis、mysql、Hbase数据库中，来分析人口热力密度、用户驻留和出行行为。创建SpringBoot后台，引用高德地图API，将处理所得的数据可视化，方便交通管理部门对交通状况的掌控和交通工具的调度。 This paper builds a pseudo distributed big data environment, and takes the user’s mobile phone signaling data as the data set. The data processing mainly includes six steps: data input, data cleaning, data buffering, data processing, data storage and data visualization. Firstly, flume’s taildir source is used to clean the mobile phone signaling data, and the data processing algorithm is designed and processed by spark programming. Then the processing results are stored in redis, MySQL and HBase databases to analyze the population thermal density, user residence and travel behavior. It creates the springboot background, references the Gode map API, and visualizes the processed data, which is convenient for the traffic management department to control the traffic situation and dispatch the vehicles.
展开▼
机译：搭建大数据伪分布式环境，将用户手机信令数据作为数据集，数据的处理主要经过数据输入、数据清洗、数据缓冲、数据处理、数据存储和数据可视化六步。首先采用Flume的Taildir Source清洗手机信令数据，通过设计数据处理算法，利用Spark编程处理，之后将处理结果存入redis、mysql、Hbase数据库中，来分析人口热力密度、用户驻留和出行行为。创建SpringBoot后台，引用高德地图API，将处理所得的数据可视化，方便交通管理部门对交通状况的掌控和交通工具的调度。 This paper builds a pseudo distributed big data environment, and takes the user’s mobile phone signaling data as the data set. The data processing mainly includes six steps: data input, data cleaning, data buffering, data processing, data storage and data visualization. Firstly, flume’s taildir source is used to clean the mobile phone signaling data, and the data processing algorithm is designed and processed by spark programming. Then the processing results are stored in redis, MySQL and HBase databases to analyze the population thermal density, user residence and travel behavior. It creates the springboot background, references the Gode map API, and visualizes the processed data, which is convenient for the traffic management department to control the traffic situation and dispatch the vehicles.
展开▼
