Wenliang Huang;Zhen Chen;Wenyu Dong;Hang Li;Bin Cao;Junwei Cao;

• 1:China Unicom Groups No. 21 Financial Street Xicheng District
• 2:Research Institute of Information Technology (RIIT) and Tsinghua National Lab for Information Science and Technology (TNList) Tsinghua University
• 3:Department of Computer Science and Technology Tsinghua University
• 4:Department of Computer Science and Technology PLA Univ. of Info. & Eng.

摘要(Abstract)：

China Unicom, the largest WCDMA 3G operator in China, meets the requirements of the historical Mobile Internet Explosion, or the surging of Mobile Internet Traffic from mobile terminals. According to the internal statistics of China Unicom, mobile user traffic has increased rapidly with a Compound Annual Growth Rate(CAGR) of 135%. Currently China Unicom monthly stores more than 2 trillion records, data volume is over 525 TB, and the highest data volume has reached a peak of 5 PB. Since October 2009, China Unicom has been developing a home-brewed big data storage and analysis platform based on the open source Hadoop Distributed File System(HDFS) as it has a long-term strategy to make full use of this Big Data. All Mobile Internet Traffic is well served using this big data platform. Currently, the writing speed has reached 1 390 000 records per second, and the record retrieval time in the table that contains trillions of records is less than 100 ms. To take advantage of this opportunity to be a Big Data Operator, China Unicom has developed new functions and has multiple innovations to solve space and time constraint challenges presented in data processing. In this paper, we will introduce our big data platform in detail. Based on this big data platform, China Unicom is building an industry ecosystem based on Mobile Internet Big Data, and considers that a telecom operator centric ecosystem can be formed that is critical to reach prosperity in the modern communications business.

关键词(KeyWords)：

Abstract:

Keywords:

基金项目(Foundation): supported in part by the National Key Basic Research and Development(973)Program of China(Nos.2013CB228206 and 2012CB315801);; the National Natural Science Foundation of China(Nos.61233016 and 61140320);; supported by the Intel Research Council under the title of“Security Vulnerability Analysis Based on Cloud Platform with Intel IA Architecture”

作者(Authors): Wenliang Huang;Zhen Chen;Wenyu Dong;Hang Li;Bin Cao;Junwei Cao;

参考文献(References)：

• [1]J.Manyika,M.Chui,B.Brown,J.Bughin,R.Dobbs,C.Roxburgh,and A.H.Byers,Big data:The next frontier for innovation,competition,and productivity,Technical report,McKinsey Global Institute,2011.
• [2]M.Meeker and L.Wu,Mary Meeker’s 2013 Internet Trends report,http://www.kpcb.com/insights/2013-internet-trends,2013.
• [3]Cisco Inc.,Visual networking index:Forecast and methodology,2010-2015,http://apo.org.au/research/ciscovisual-networking-index-forecast-and-methodology-2010-2015,2011.
• [4]T.Li,F.Y.Han,S.Ding,and Z.Chen,LARX:Largescale anti-phishing by retrospective data-exploring based on a cloud computing platform,in Proc.20th International Conference on Computer Communications and Networks(ICCCN),Hawaii,USA,2011,pp.1-5.
• [5]Z.Chen,F.Y.Han,J.W.Cao,X.Jiang,and S.Chen,Cloud computing-based forensic analysis for collaborative network security management system,Tsinghua Science and Technology,vol.18,no.1,pp.40-50,2013.
• [6]Z.Chen,L.Y.Ruan,J.W.Cao,Y.Yu,and X.Jiang,TIFAflow:Enhancing traffic archiving system with flow granularity for forensic analysis in network security,Tsinghua Science and Technology,vol.18,no.4,pp.406-417,2013.
• [7]S.Kornexl,V.Paxson,H.Dreger,A.Feldmann,and R.Sommer,Building a time machine for efficient recording and retrieval of high-volume network traffic,in Proc.5th ACM SIGCOMM Conference on Internet Measurement,2005.
• [8]G.Maier,R.Sommer,H.Dreger,A.Feldmann,V.Paxson,and F.Schneider,Enriching network security analysis with time travel,in Proc.ACM SIGCOMM 2008 Conference on Data Communication,New York,USA,2008,pp.183-194.
• [9]J.Li,S.Ding,M.Xu,F.Y.Han,X.Guan,and Z.Chen,TIFA:Enabling real-time querying and storage of massive stream data,in Proc.International Conference on Networking and Distributed Computing(ICNDC),Osaka,Japan,2011,pp.61-64.
• [10]K.Wu,S.Ahern,E.W.Bethel,J.Chen,H.Childs,E.Cormier-Michel,C.Geddes,J.Gu,H.Hagen,B.Hamann,W.Koegler,J.Lauret,J.Meredith,P.Messmer,E.Otoo,V.Perevoztchikov,A.Poskanzer,Prabhat,O.Rbel,A.Shoshani,A.Sim,K.Stockinger,G.Weber,and W.M.Zhang,FastBit:Interactively searching massive data,Journal of Physics:Conference Series,vol.180,no.1,2009.
• [11]L.Deri and F.Fusco,MicroCloud-based network traffic monitoring,in IFIP/IEEE International Symposium on Integrated Network Management(IM),2013.
• [12]L.Deri and F.Fusco,Real-time MicroCloud-based flow aggregation for fixed and mobile networks,in 9th IEEE International Wireless Communications and Mobile Computing Conference(IWCMC),2013,pp.96-101.
• [13]Y.Lee and Y.Lee,Toward scalable internet traffic measurement and analysis with Hadoop,ACM SIGCOMM Computer Communication Review,vol.43,no.1,pp.5-13,2012.
• [14]Y.Lee,W.Kang,and H.Son,An internet traffic analysis method with mapreduce,in Network Operations and Management Symposium Workshops(NOMS Wksps),2010,pp.357-361.
• [15]F.Qian,Z.Wang,Y.Gao,J.Huang,A.Gerber,Z.Mao,S.Sen,and O.Spatscheck,Periodic transfers in mobile applications:Network-wide origin,impact,and optimization,in Proc.21st International Conference on World Wide Web,Lyon,France,2012,pp.51-60.
• [16]J.L.Gailly and M.Adler,The ZLIB library,http://www.zlib.org/,2013.
• [17]M.F.Oberhumer,Lzo documentation,http://www.oberhumer.com/opensource/lzo/lzodoc.php,2011.