Computer and Modernization

Previous Articles     Next Articles

Big Data Log Collection System Based on Self-controlled Container Platform

  

  1. (Department 4 of Foundation, North China Institute of Computing Technology, Beijing 100083, China)
  • Received:2018-11-11 Online:2019-02-25 Published:2019-02-26

Abstract: With the application of technologies such as cloud computing, virtualization, container cloud, and domestically-controlled and controllable requirements, more and more services will be deployed on domestic servers and self-controlled container cloud. In the process of running the service, we need to obtain log data for a series of monitoring, statistics, analysis and forecasting work. However, due to the particularity of the domestic server and the particularity of self-controlled container cloud, the traditional log collection methods (including real-time acquisition and centralized acquisition) are not well suited for self-controlled container cloud. In order to improve the quality and performance of the service and ensure a certain comprehensiveness of the log data, an adaptive data acquisition algorithm is proposed, which can self-adjust the log data collection and transmission according to the load of the server. When the server load is low, the number of data collection and transmission is increased, even in real-time collection; when the server load is high, the collection and transmission of log data is reduced, thereby reducing the pressure on the server load, and improving the quality and performance of the service itself. Finally, theoretical and experimental analysis proves that the data acquisition algorithm can effectively alleviate the pressure of domestic servers while ensuring a certain comprehensiveness of data.

Key words: self-controllable, container cloud, data collection, big data

CLC Number: