This paper adopts Hadoop to build and test the storage and retrieval platform for painting resources. This paper adopts Hadoop as the platform and MapReduce as the computing framework and uses Hadoop Distributed Filesystem distributed file system to store massive log data, which solves the storage problem of massive data. According to the business requirements of the system, this paper designs the system according to the process of web text mining, mainly divided into log data preprocessing module, (...) log data storage module, log data analysis module, and log data visualization module. The core part of the system is the log data analysis module. The analysis of search keywords ranking, Uniform Resource Locator, and user click relationship, URL ranking, and other dimensions are realized through data statistical analysis, and Canopy coarse clustering is performed first according to search keywords, and then K-means clustering is used for the results after Canopy clustering, and the calculation of cosine similarity is adopted to realize the grouping of users and build user portrait. The Hadoop development environment is installed and deployed, and functional and performance tests are conducted on the contents implemented in this system. The constructed private cloud platform for remote sensing image data can realize online retrieval of remote sensing image metadata and fast download of remote sensing image data and solve the problems in storage, data sharing, and management of remote sensing image data to a certain extent. (shrink)

The Internet has become a natural communication platform for modern society. Web archives, which began in the 1990s to capture and preserve changing web content, have thus become key sources for research in the recent past. The analysis of their data is complicated by, for example, insufficient competencies of researchers, the need for computing resources or legislation. One way to meet the needs of users is to develop tools and research interfaces that allow to work with data without the need (...) for technological knowledge of advanced extraction and thus open it to researchers. The study addresses the issue of access to archival web data, approaches efforts to formulate a theoretical and methodological framework and proposes a design for access and further data processing, which is applied in a unique research interface for extracting large data from web archives using advanced machine learning to generate and categorization of text outputs. (shrink)

Though full of promise, Big Data research success is often contingent on access to the newest, most advanced, and often expensive hardware systems and the expertise needed to build and implement such systems. As a result, the accessibility of the growing number of Big Data-capable technology solutions has often been the preserve of business analytics. Pay as you store/process services like Amazon Web Services have opened up possibilities for smaller scale Big Data projects. There is high demand for this type (...) of research in the digital humanities and digital sociology, for example. However, scholars are increasingly finding themselves at a disadvantage as available data sets of interest continue to grow in size and complexity. Without a large amount of funding or the ability to form interdisciplinary partnerships, only a select few find themselves in the position to successfully engage Big Data. This article identifies several notable and popular Big Data technologies typically implemented using large and extremely powerful cloud-based systems and investigates the feasibility and utility of development of Big Data analytics systems implemented using low-cost commodity hardware in basic and easily maintainable configurations for use within academic social research. Through our investigation and experimental case study, we found that not only are solutions like Cloudera’s Hadoop feasible, but that they can also enable robust, deep, and fruitful research outcomes in a variety of use-case scenarios across the disciplines. (shrink)

In recent years, the network has become more complex, and the attacker’s ability to attack is gradually increasing. How to properly understand the network security situation and improve network security has become a very important issue. In order to study the method of extracting information about the security situation of the network based on cloud computing, we recommend the technology of knowledge of the network security situation based on the data extraction technology. It converts each received cyber security event into (...) a standard format that can be defined as multiple brochures, creating a general framework for the cyber security situation. According to the large nature of network security situation data, the Hadoop platform is used to extract aggregation rules, and perform model extraction, pattern analysis, and learning on a network security event dataset to complete network security situation rule mining, and establish a framework for assessing the state of network security. According to the results of the federal rule extraction, the level of network node security risk is obtained in combination with signal reliability, signal severity, resource impact, node protection level, and signal recovery factor. A simulation test is performed to obtain the intrusion index according to the source address of the network security alarm. Through the relevant experiments and analysis of the results, the attack characteristics obtained in this study were obtained after manually reducing the network security event in the 295 h window. The results show that after the security event is canceled, the corresponding window attack index decreases to 0, indicating that this method can effectively implement a network security situation awareness. The proposed technique allows you to accurately sense changes in network security conditions. (shrink)

Multiway join queries incur high-cost I/Os operations over large-scale data. Exploiting sharing join opportunities among multiple multiway joins could be beneficial to reduce query execution time and shuffled intermediate data. Although multiway join optimization has been carried out in MapReduce, different design principles are not considered. To bridge the gap of not considering the optimization of Big Data platforms, an end-to-end multiway join over Flink, which is called Join-MOTH system, is proposed to exploit sharing data granularity, sharing join granularity, and (...) sharing implicit sorts within multiple join queries. For sharing data, our previous work, Multiquery Optimization using Tuple Size and Histogram system, has been introduced to consider the granularity of sharing data opportunities among multiple queries. For sharing sort, our previous work, Sort-Based Optimizer for Big Data Multiquery, has been introduced to consider the implicit sorts among join queries. For sharing join, additional modules have been tailored to the J-MOTH optimizer to optimize sharing work by exploiting shared pipelined multiway join among multiple multiway join queries. The experimental evaluation has demonstrated that the J-MOTH system outperforms the naive and the state-of-the-art techniques by 44% for query execution time using TPC-H queries. Also, the proposed J-MOTH system introduces maximal intermediate data size reduction by 30% in average over Hadoop-like infrastructures. (shrink)

Based on cloud computing and statistics theory, this paper proposes a reasonable analysis method for big data of film and television. The method selects Hadoop open source cloud platform as the basis, combines the MapReduce distributed programming model and HDFS distributed file storage system and other key cloud computing technologies. In order to cope with different data processing needs of film and television industry, association analysis, cluster analysis, factor analysis, and K-mean + association analysis algorithm training model were applied (...) to model, process, and analyze the full data of film and TV series. According to the film type, producer, production region, investment, box office, audience rating, network score, audience group, and other factors, the film and television data in recent years are analyzed and studied. Based on the study of the impact of each attribute of film and television drama on film box office and TV audience rating, it is committed to the prediction of film and television industry and constantly verifies and improves the algorithm model. (shrink)

Recently, big data analytics have gained significant attention in healthcare industry due to generation of massive quantities of data in various forms such as electronic health records, sensors, medical imaging, and pharmaceutical details. However, the data gathered from various sources are intrinsically uncertain owing to noise, incompleteness, and inconsistency. The analysis of such huge data necessitates advanced analytical techniques using machine learning and computational intelligence for effective decision making. To handle data uncertainty in healthcare sector, this article presents a novel (...) metaheuristic rough set-based feature selection with rule-based medical data classification technique on MapReduce framework. The proposed MRSFS-RMDC technique designs a butterfly optimization algorithm for minimal rough set selection. In addition, Hadoop MapReduce is applied to process massive quantity of data. Moreover, a rule-based classification approach named Repeated Incremental Pruning for Error Reduction is used with the inclusion of a set of conditional rules. The RIPPER will scale in a linear way with the number of training records utilized and is suitable to build models with data uncertainty. The proposed MRSFS-RMDC technique is validated using benchmark dataset and the results are inspected under varying aspects. The experimental results highlighted the supremacy of the MRSFS-RMDC technique over the recent state of art methods in terms of different performance measures. The proposed methodology has achieved a higher F-score of 96.49%. (shrink)

To improve the understanding of film and television postproduction for college students in the era of intelligent media, a study is conducted on college students’ short video communication education and audience psychology based on the rapid development of virtual image and the Internet of Things. Primarily, the collaborative filtering algorithm is optimized and combined with the principle of Spark and Hadoop platforms as well as the IoT and virtual image technologies. Then, a hybrid computing model is proposed, and the (...) two algorithms are improved and combined, with 90,000 network video records as data samples. Finally, the push accuracy of the hybrid algorithm and the traditional algorithm is calculated and compared, and based on this, a questionnaire survey on the audience psychology of short video production is carried out for college students. The results show that the time user of the combined algorithm is always at least 0.4 s faster than that of a single algorithm and the running speed of the algorithm with five nodes is nearly 80% higher than the algorithm with a single node. The Spark algorithm with multinode has good versatility in image recording and processing of large groups of college students. When processing more than 100,000 image records, the deviation values of Spark and Hadoop with a single node exceeded 1.1, but the deviation value of the hybrid algorithm was still lower than 1.1. With the increase of data volume, the deviation values of the three algorithms are increasing. Compared with the traditional CFA algorithm, the optimized algorithm has a higher speed in processing data and is more accurate in content pushing. From the questionnaire survey of college students, it is found that contemporary college students are not active in learning knowledge of virtual images. Hence, it is concluded that colleges must carry out relevant courses based on short video communication education and strengthen the short video communication education of college students. A reference is provided for the development of college students’ short video communication education in the digital age. (shrink)

Frequent pattern mining is an effective approach for spatiotemporal association analysis of mobile trajectory big data in data-driven intelligent transportation systems. While existing parallel algorithms have been successfully applied to frequent pattern mining of large-scale trajectory data, two major challenges are how to overcome the inherent defects of Hadoop to cope with taxi trajectory big data including massive small files and how to discover the implicitly spatiotemporal frequent patterns with MapReduce. To conquer these challenges, this paper presents a MapReduce-based (...) Parallel Frequent Pattern growth algorithm to analyze the spatiotemporal characteristics of taxi operating using large-scale taxi trajectories with massive small file processing strategies on a Hadoop platform. More specifically, we first implement three methods, that is, Hadoop Archives, CombineFileInputFormat, and Sequence Files, to overcome the existing defects of Hadoop and then propose two strategies based on their performance evaluations. Next, we incorporate SF into Frequent Pattern growth algorithm and then implement the optimized FP-growth algorithm on a MapReduce framework. Finally, we analyze the characteristics of taxi operating in both spatial and temporal dimensions by MR-PFP in parallel. The results demonstrate that MR-PFP is superior to existing Parallel FP-growth algorithm in efficiency and scalability. (shrink)

The efficient processing of large-scale data has very important practical value. In this study, a data mining platform based on Hadoop distributed file system was designed, and then K-means algorithm was improved with the idea of max-min distance. On Hadoop distributed file system platform, the parallelization was realized by MapReduce. Finally, the data processing effect of the algorithm was analyzed with Iris data set. The results showed that the parallel algorithm divided more correct samples than the traditional algorithm; (...) in the single-machine environment, the parallel algorithm ran longer; in the face of large data sets, the traditional algorithm had insufficient memory, but the parallel algorithm completed the calculation task; the acceleration ratio of the parallel algorithm was raised with the expansion of cluster size and data set size, showing a good parallel effect. The experimental results verifies the reliability of parallel algorithm in big data processing, which makes some contributions to further improve the efficiency of data mining. (shrink)

More and more tourists are sharing their travel feelings and posting their real experiences on the Internet, generating tourism big data. Online travel reviews can fully reflect tourists’ emotions, and mining and analyzing them can provide insight into the value of them. In order to analyze the potential value of online travel reviews by using big data technology and machine learning technology, this paper proposes an improved support vector machine algorithm based on travel consumer sentiment analysis and builds an (...) class='Hi'>Hadoop Distributed File System system based on Map-Reduce model. Firstly, Internet travel reviews are pre-processed for sentiment analysis of the review text. Secondly, an improved SVM algorithm is proposed based on the main features of linear classification and kernel functions, so as to improve the accuracy of sentiment word classification. Then, HDFS data nodes are deployed on the basis of Hadoop platform with the actual tourism application context. And based on the Map-Reduce programming model, the map function and reduce function are designed and implemented, which greatly improves the possibility of parallel processing and reduces the time consumption at the same time. Finally, an improved SVM algorithm is implemented under the built Hadoop platform. The test results show that online travel reviews can be an important data source for travel big data recommendation, and the proposed method can quickly and accurately achieve travel sentiment classification. (shrink)

This article starts with the analysis of the existing electronic commerce system, summarizes its characteristics, and analyzes and solves its existing problems. Firstly, the characteristics of the relational database My Structured Query Language and the distributed database HBase are analyzed, their respective advantages and disadvantages are summarized, and the advantages and disadvantages of each are taken into account when storing data. My SQL is used to store structured business data in the system, while HBase is used to store unstructured data (...) such as pictures. These two storage mechanisms together constitute a data storage subsystem. Secondly, considering the large amount of data in the e-commerce system and the complex calculation of the data mining algorithm, this paper uses MapReduce to realize the parallelization of the data mining algorithm and builds a Hadoop-based commodity recommendation subsystem on this basis. We use JavaEE technology to design a full-featured web mall system. Finally, based on the impact of cloud computing, mobile e-commerce is analyzed, including relevant theories, service mode, architecture, core technology, and the application in e-commerce, which can realize e-commerce precision marketing, find the optimal path of logistics, and take effective security measures to avoid transaction risks. This method can avoid the disadvantages of the traditional e-commerce, where large-scale data cannot be processed in a timely manner, realize the value of mining data behind, and realize the precision marketing of e-commerce enterprises. (shrink)