Big data live broadcast middle station project in the era of 5G high traffic

2020-08-02 21:21:13 0 Comment 1310 views

Course Name: Big Data Live Broadcasting in the 5G Era Application Scenarios: Commercial Live Broadcast Application Fields: Big data quasi-real-time, r

Course name: Big Data Live Broadcasting Project in the 5G Era

Application scenario:Commercial live broadcast

Application field: Development and explanation of big data quasi-real-time, real-time applications

Project description: This project is mainly practiced in a large-scale commercial ToB live broadcast project. The entire project is responsible for the data center of the overall project, and is not responsible for data reporting and data use. The real log data in the enterprise is directly used in the projectAs the backing of data, create a real project environment for each student. The project is to use java language to develop spark streaming program, the core framework is structed-spreaming, the project processes the data reported by each business end in real time, and pushes the result data to the business end for use in time.

Big data live broadcast middle station project in the era of 5G high traffic

Main modules: Acquisition program, data analysis program, data cleaning program, data statistics program, mysql analysis function, data push program.

Application Technology Stack

  • Java Spark
  • Flume
  • Kafka
  • Structured Streaming
  • Flink SQL reconstruction statistics
  • Mysql function
  • Redis
  • quartz

course content:

1. Project overview (one class)

1. Project overview, business Rules combing, data cleaning rules, technical architecture explanation

2, explanation of the components used in the project

Flume / spark-structed-spreaming

/ kafka / redis / mysql function/ quartz

Two , Project explanation and code development (one lesson for each summary)

  1. nginx explanation + flume collecting nginx logs to kafka topic
  2. Real-time program parses nginx logs and writes them to the development of different kafka topics
  3. Development of asynchronous alarm program and monitoring program (the alarm uses a third-party paid SMS interface)
  4. Real-time cleaning program cleans kafka data - Development of cleaning program 1
  5. Real-time cleaning program cleaning kafka data--development of cleaning program 2
  6. Real-time statistics Program processing cleaning result data development
  7. flink explanation + rebuild data statistics program with flink
  8. Kafka data real-time storage program development
  9. mysql function writing and calling execution
  10. data Push program development
  11. Redis-based information warning, offset information storage
  12. Code function test and break Point capacity enhancement, online deployment process

3. Summary and outlook (one class)

1. Real-time program pressure measurement/deployment/monitoring/alarm service/precaution explanation

2. Rapid improvement of coding ability/Technical Q&A and outlook

Students interested in this project Backstage consultation [project] is fine