Metabolic analysis based on
fingerprints of Angelica sinensis

Angelica refers to the dry roots of Angelica sinensis, an herb of Chinese traditional medicine. It has a widely curative effect in modern medicine. Because the internal chemistry components of Angelica sinensis are very complex and are lastly affected by different environmental factors and human factors during it growth cycle, it is difficult to control the plucking time and its dose effect. In my bachelor thesis, a systematic research on Angelica sinensis’ metabolomics was conducted at 27 samples in the different growth stages from Minxian, Gansu Province.

The bio-fingerprint information of Angelica sinensis in different growth period was extracted by means of H-NMR and GC-MS. The experimental results of H-NMR and GC-MS are shown in following figures. The statistical test, such as variance analysis and t-test, was implemented to distinguish the chemical characteristic peaks and chemical retention time of Angelica sinensis which varies greatly during the growth period.

The result of H-NMR		The result of GC-MS

In order to discover the dynamic accumulation of angelica metabolites and explore the metabolic components which are sensitive to the changes of years in its growth cycle, orthogonal partial least squares discriminant analysis (OPLS-DA) is used to analyze the integral relative content in spectral data systematically. By comparing with database, the characteristic signal peaks of linoleic acid, Angelica polysaccharide, ligustilide, ferulic acid, Angelica lactone A, butenyl phthalide and 3-n-Butylphthalide were identified on the H-NMR fingerprint. According to the Universal Database of Gas Chromatographic Retention Index, furthermore, we identified about 44 volatile compounds from the volatile oil of Angelica sinensis. Identification results are shown in following figures.

Substance Attribution of spectral data

The experiment results show that: Z-ligustilide, ferulic acid, and other three components are active pharmaceutical ingredients , and the changes of polysaccharide,15 alkanol, n-butylidenephthalide and other nonpharmacodynamic components are also obvious in three years of growth period. Therefore those compounds can be the symbolic chemical compositions used to evaluate the quality of angelica and control the harvest time; In the experimental process, we had found that the medicinal ingredients in angelica sinensis don’t reach the maximum content in the traditional harvest time, but the ratio between the Z-ligustilide and other ingredients, such as n-butylidenephthalide, reaches a reasonable level.

The analysis result of NMR

The analysis result of GC-MS

The applied research can guide the picking period of Angelica sinensis and the extraction of effective medicinal ingredients. Finally, this bachelor paper entitled Metabolic Analysis Based on Fingerprint of Angelica Sinensis stood out among 350 papers of the whole college and later incorporated in the Excellent Graduation Thesis Collection of 2017 Undergraduates of Central South University.

To get more information about this project, please click here to visit its code repository in github.

Community Detection Based on
Heterogeneous Network of Statistical Researches

1. Research Background

Heterogeneous network is different from homogeneous network, which converts the network information extracted from the complex systems into the different types of nodes and links. At present, the majority of community detection methods are worked on homogeneous networks and such a hypothesis that the nodes and the links of a certain network structure are all the identical type is made. That hypothesis provides much convenience for the study on the community detection method. Compared with the community detection in homogeneous networks, the community detection based on heterogeneous networks can find out more association information and more accurate community structure. The research object of this paper is just based on the heterogeneous network which is constructed by using the academic paper data in the statistics research field.

2. Research methods

Data collection of statistical research literatures: by using “beautifulsoup” and “selenium” module in python, I dynamically crawled and collected all literature data about statistics and probability at the database of “web of science” to construct the thesis topic network.

Information page in “Web of Science”

Because Impact Factor of some journals is too small, those journals are unrepresentative. Therefore, the textual data in this thesis are fixed in those papers from 27 famous statistical and probabilistic journals which have high IF and top rank in JCR. Those selected journals include Annals of Statistics, American Statistician, Biometrics, Journal of Statistical Software, etc. And every paper in those journals has many variables, such as “authors”, “ citation count”,” references”,”abstract”,etc. A sample data is shown in the following chart.

A sample data

More detailed data examples and the code of this network data acquisition are stored in this code repository.

Creation and cleaning of data structure. The feature of every literature was saved and then extracted for creating the data nodes——journal node, paper node and writer node respectively. For the data structure of attributes of every node, I saved it by the json tree structure so as to be called conveniently. What’s more, normalization cleaning and disposal are used in textual data to keep the data format identical for modeling.

The construction for heterogeneous network. There are three layers in the heterogeneous network including journal layer, paper layer and writer layer. Among the layers, some adjacent edges with same type exist. The adjacent edges in paper layer represent relation of mutual reference and the edges in writer layer represent cooperative relation. Between the journal layer and paper layer there exist adjacent edge representing relation, and so do the paper layer and writer layer. In such way, a three-dimensional network of statistics research was formed. The detailed network model is illustrated in following figure.

Heterogeneous network in this paper

The community detection of journal layer based on LDA topic model.This process is for the community detection in the journal layer merely. Each journal node contains the abstract information of these papers published in it from 2013 to 2018. The corpus and dictionary of all literatures are established for modeling LDA topic model. According to membership degree of every journal node for each theme, the research field, namely, the community for every journal will be detected. The result of LDA is illustrated in the following figure.

The result of LDA

The community detection based on label propagation model in paper layer.According to the adjacent edge effect existing in interlayers, the paper layer is divided into several paper sub-networks firstly. The community detection will be implemented in these sub-networks by parallel operation in these sub-networks. There is information in these sub-networks, like keyword, abstract and publisher and so on. Referring to these attributes, the similarity among the nodes was defined and the probability of propagation among the nodes was calculated further with those similarity. At last, the community detection should be implemented by propagation probability matrix of label propagation algorithm. The result of community detection in this layer is shown in following picture.

The paper layer has been divided into several paper sub-networks according to the LDA result

the result of community detection in paper layer

The community detection based on label propagation model in author layer.According to the adjacent edge effect existing in interlayers, the author layer is divided into several author sub-networks firstly. The community detection will be implemented in these sub-networks by parallel operation in these subnets. There is information in each node of these sub-networks, including keyword, abstract and university and so on. Referring to these attributes, the similarity among the nodes was defined and the probability of propagation among the nodes was calculated further. At last, the community detection should be implemented by propagation probability matrix of label propagation algorithm.

The result of community detection in author layer

3. Research result

Through the community detection methods, such as LDA topic model and label propagation algorithm, multi-dimensional structure, multimode information, semantic information and link connection in three dimensional networks will be extracted and utilized to seek out the community structures which are relatively close and have the practical significance. In the empirical analysis section, by using the method presented by this paper, multidimensional scientific research communities which have relatively compact structure, higher modularity and explicit practical significance are found out, including 4 journal communities, 20 paper communities under those 4 journal communities, and 92 author communities under those 20 paper communities.

The scientific research communities detected can provide science researchers with the journals, research achievements and research collaborators in the same field and can enable them to grasp the research direction and progress in future. Compared with traditional methods of homogeneous community detection, such as Louvain, the method proposed by the paper can make the most of heterogeneous information and node attribute information contained in the statistical science research network, find out the overlapping communities and cope with complex relationship of multi-dimensional structure well to improve the accuracy of algorithm. Moreover, such method can reduce the complexity of operation time for community detection with the parallel processing.

More information, primitive data, middle results, program code and original paper are stored in the website of ‘github’. Please click here to enter its code repository in github.

Some cases in applied statistics

1. National Statistical Case Contest for Postgraduate

In this contest, our team make music genre classifier based on NLP.Firstly, through web crawler，we collected all English songs on the Billboard Hot 100 from the 1st week of 2012 to the 10th week of 2018. Secondly, by descriptive statistics on the lyrics of different genres, we summarized the lyric features of different genres in terms of the vocabulary.

Word frequency count

At the same time, the most popular singers and albums from 2012 to 2018 were found out, and the song styles of these pop singers and albums were described and analyzed.

song count of top 10 singer

Thirdly, we done the data cleaning and processing on those lyrics of every songs to make each song correspond to a word frequency statistic of its original lyrics. And then we some methods in NLP to deal with those word frequency statistics so as to enable them convert to word vector. After that processing, we tried a variety of machine learning methods in that word vector space, such as Support Vector Machine, Naive Bayesian, Neural Network and so on. Finally, we realize a simple text categorization for song style.

Eventually, with this statistical case, our team won the first prize in the Statistical Model Contest in Central University of Finance and Economics and the third prize in the National Statistical Case Contest for Postgraduate.

More on this case are stored in this repository.

2.Four cases in Exploratory Analysis

Analysis for Mobile Phone Questionnaire

In order to explore consumers' satisfaction and preference for different mobile phone brands, a questionnaire was designed by the Lickett Seven-Point Scale. The responses of 106 consumers to 32 questions about Samsung, Apple, HTC, Huawei and other four mobile phone brands were collected. Firstly, descriptive statistics is used to visualize the statistical characteristics of the four major mobile phone brands. Secondly, I construct a scientific evaluation model of satisfaction and loyalty through these mobile phone questionnaires, and then use Factor Analysis and Linear Regression to extract common factors that can reflect 22 evaluation indicators. By this way, we could reflect the four major mobile phone brands with a small number of common factors that can not be evaluated, such as performance-price ratio, user experience and symbolic performance. Then, I used these common factors to regress mobile phone manufacturers' satisfaction and loyalty and used regression model to analyze how the important factors affect loyalty and satisfaction. Finally, I put forward relevant suggestions for mobile phone companies to formulate correct production and marketing strategies.

Word cloud

Analysis for Players in Premier LeagueⅠ

This case discussed the following issues through analyzing competition performance data of 166 soccer players from 16 clubs:

1. The overall macro-statistical analysis of the Premier League

2. The evaluation of players' ability of scoring

3. The detailed analysis of Manchester United's player system

4. The internal correlation analysis of players' indicators

Analysis for Players in Premier LeagueⅡ

On the foundation of Analysis l, I executed a greater depth of analysis for soccer data from more different perspectives. Independent variables are selected according to football knowledge. Finally, the basic indicators reflecting players’ ability are divided into six potential factors: players' basic condition, offensive ability, defensive ability, control ability, advancing ability, foul situation. And these indicators have an impact on the number of goals in the next season. Finally, I tried to make Principal Component Analysis, Factor Analysis and Regression Analysis on those variables.

Premier League

Evaluation for Cities Investment Environment

In order to evaluate the investment potential of real estate in China's cities, this case collected economic indicators and housing supply and demand indicators of 20 major cities in the Yangtze River Delta, the Pearl River Delta and the Bohai Rim respectively. Firstly, descriptive statistics is used to visualize the urban development level and housing supply and demand of the three economic circles. Secondly, I construct a scientific investment potential evaluation model through these data, and then use principal component analysis method to construct comprehensive evaluation indicators to evaluate the real estate investment potential of the major cities, and rank the real estate development investment potential of each city to assist investors to make rational decisions.

Pearl River Delta

The reports, code and figures of those cases are published in this repository.

3. Some cases in Visualization

This project comprises five assignments in the class of Visualization. Four of them are Individual assignment and the left one is a teamwork. The purpose of those assignments is equipping students with the ability of converting given data into understandable statistical graphs with R, such as line plot, box plot, map, pie chart, Heat Map, etc.

The reports, code and figures of those cases are published in this repository.

Visualization

4. Some cases in Sampling Techniques

This project comprises two assignments in the class of Sampling Techniques. One is the assignment in class and the other is an answer sheet for exam. Through training, this class equipped student with the ability of using the sampling methods and estimating the total quantity in the sampling survey correctly. The main content in this class consist largely of Simple Estimation, Ratio Estimation, Regression Estimation.

The reports, code and figures of those cases are published in this repository.

5. Four cases in Data preprocessing

This project comprises three cases in the class of Data preprocessing. Three of them are the assignment in class and another is an answer sheet for exam. Through training in this class, I have had a good command of converting original data to clean data with some r packages. The main content in this class includes structured data processing, unstructured data processing, regular expression, missing value processing, etc.

The reports, code and figures of those cases are published in this repository.

Distributed Processing & Parallel Processing

In the class of Distributed processing Foundation of Big Data，We have learned how to deploy and employ Hadoop on servers and use it in big data analysis. At the same time, we put those knowledges into practice on Alibaba Cloud in person. To be specific, we learn how to do a standard distributed processing with MapReduce, how to deal with machine learning with Spark and how to make database management with Hive. Therefore, this project comprises four assignments in the class. Three of them are school assignments and another is a final project.

Lab1：Distributed Computing by MapReduce

The data set used for distributed computing experiments is collected from the information of people who successfully applied for loans in Lending Club from 2007 to 2015. It contains 235629 records. The variables loan and income are selected to calculate their mean, variance and count on Hadoop with MapReduce.

Hadoop

Lab2：Linear regression on high dimensional matrix by MapReduce

A data set without intercept( linear_random.csv) is generated by R. The data set contains 10 variables and 1 dependent variable and 10,000 records.

According to the principle of multiple regression, the formula for calculating regression coefficient can be divided into two parts——xy part and x'x part, and two parts can be calculated on Mapper. To be specific, firstly, xy calculation is done after line segmentation on each computational node (for example x11y, x12y,..., x110y) as the first column of data, and so on…… Now each node adds all the vectors. The calculation part of x'x is the same, except that x needs to be divided into 10 column vectors and the product part of X'is xy. After that, the two parts of each node's calculation result are merged into one line, and the flow is output to reducer. And then the sum is merged. Finally the matrix is restored to calculate deta.

Linear regression by parallel acceleration

Lab3：Database Management by Hive

Firstly, the database of cufe_wangsiyu is established under the path of’/ user/lifeng/cufe_wangsiyu/hive’. In this database, stocks tables are established using stocks.csv, and then three query hql are created to make selection about stock in Hive. The query results are saved in three folders: cufe_wangsiyu/lab3/res1, result2 and result3.

Hive

Final project：Lyric-based emotional classifier for songs (supported by Spark)

The crawler of Python was used to collect the lyric information of songs which have two opposite emotions (sad and happy) on the Internet music platform. Some lyrics in the sample were used to construct corpus and transformed into lyric vectors containing latent semantics by LSI and TF-idf model.

Firstly, Lyric vectors with latent semantics was considered as input and the unknown-style songs were classified by the neural network built through Keras of python. Then, the music styles for large quantities of Lyric texts were classified by machine learning methods such as Naive Bayesian, Support Vector Machine and Random Forest in Mllib of Spark. The cross-validation accuracy of each model was more than 95%.

Hadoop armed spark

To learn more about this project. Please click here to visit its repository in which code, report and result are stored.

Fintech Cases

1. Credit Evaluation System Based on Credit-card data

Based on the bank credit card data, this case uses a large number of user information data to carry out descriptive data mining and inferential data modeling respectively to divide each user to a certain group and to analyze the credit status, risk status and loss of each group. At the same time, this case uses information gain rate to measure the weight of each statistical indicator as well.

More about this case are stored in this repository.

You can also click here to browse markdown of this project for convenience.

Credit-card

2. Personal Credit Evaluation Based on User Behavior

This project originates from the FengYun Cup Machine Learning Modeling Competition. With the User behavior data provided by sponsors, we extracted data features from structural data and unstructural data and converted them to vector which can be processed by machine learning method. After several unsuccessful experiments, we decided to use Xgboost to predict whether one will default given his user behavior data finally.

Eventually, with a high default discrimination ratio, our team enter the top 10 of 400 teams and won the third prize in this Competition. More about this case are stored in this repository.

Fengyun Cup

3. Loan Credit Audit Model

This project derives from National Statistical Modeling Contest for Postgraduate. With the borrower's information and default data provided by sponsors, a variety of machine learning models, including SVM and various Ensemble learning algorithms, are established. All these machine learning can achieve high prediction accuracy. Among them, the Xgboost has the highest prediction accuracy of 93.61% and a high recall rate of 90%. At the same time it also has a short running time fortunately. The Xgboost model can help credit institutions to select potential loan applicants from a large number of loan applicants, effectively avoid losses caused by overdue users, and formulate lending plans and management methods, so as to improve the operational efficiency of financial institutions.

More explanations and code about this case are stored in this repository.

Certificate

4. Credit Rating Evaluation Based on Several Clustering Method

Based on Taiwan credit data, this research established various clustering models, including Hierarchical Clustering (AGNES), Kmeans Clustering, Density-Based Spatial Clustering (DBSCAN), Hierarchical Sparse Clustering, Kmeans Sparse Clustering and Bidirectional Clustering. According to the Adjusted Rand index, I judged the different clustering methods’ performance on credit data. This research can provide suggestions for credit institutions to classify credit rank grades. Finally, for clustering methodology, the characteristics and advantages of each clustering method are discussed and compared.

More explanations and code about this case are stored in this repository. You can also click here to browse markdown of this project for convenience.

Statistical learning

1. Ensemble learning

In the final project of Statistical computing, basing on Credit Card Clients Data Set, I realized a variety of Ensemble Algorithms and compared them from their advantages and disadvantages. This case includes four kinds of Ensemble learnings, Decision Tree, Random Forest, Adaboost，XGboost. Finally, for this data, we discussed how to prune, adjust depth of each tree and adjust other parameters on those methods.

More details and complete report are displayed in this repository.

Random Forest

2. Statistical Methods

This project was finished in the class of Statistical Modeling of Big Data. And it consists of two classical statistical regression methods—— Ridge Regression and Stepwise Regression——and two verifications for Law of Large Numbers and Central Limit Theorem. Besides, I also accelerated the regression speed with Cholesky Principle and Parallel Computing.

More details about this project and complete report are displayed in this repository.

3. SVM & NN

The Support Vector Machine and Neural Network are realized in this project with the Abalone dataset. In the class of Machine Learning of Big Data, we learn theories and algorithms and put them into practice in our school assignments. Exactly, I only made those algorithms operate with certain modules.

More details about this project and complete report are displayed in this repository. You can also click here and here to browse markdown of this project for convenience.

Neural Network

Community Detection in Citation Network

At present, I am working at this paper under the guidance of my master supervisor.

The research field of this paper is statistical modeling in complex network. More specifically, we want to achieve more accurate community detection on publication citation network. Depending on this work, we hope to divide the mainstream statistical research network to accurate and meaningful communities in which the density of links among nodes is high. With the result of this project, paper recommendation system can be established, and it can help researchers to find more valuable statistical fields and central papers.

The range of this research is fixed on the papers published on the top4 journal of statistical field——Annals of Statistical, American Statistician, Biometrika and Journal of The Royal Statistical Society Series B. The time span of those paper is from 1998 to 2018 and the source of them are crawled from Web of Science. With those data and citation relationship among the papers, I constructed citation network of top4 journals which consist of about 6303 nodes(papers). The global network is shown as following figure.

Citation Network

For research methods, firstly, we do some descriptive statistics on this citation network, such as the density of this network and several centrality degree indices. And then I try some direct network community detection methods on this network. Finally, we chose the D_score as the main method in our paper. Compared with other method, the Dscore compute fast so that it is suitable for relatively large network. On the base of that algorithm, we also improved Dscore with auxiliary attributes, such as information extracted from abstract by NLP, university of author, keywords …… so as to enable those divided paper citation communities to have practical significance. At the same time, in order to make the count of communities more convincing, we improve the index to supervise community detection on the foundation of Calinski-Harabaz index and silhouette coefficient. Through our method, we found several practical communities in this citation network such as “Large-Scale Multiple Testing” community, “Bayes” community, “Variable Selection” community, etc. More information and some codes of this project are stored in this repository.

Sihouette Coefficient		Scree Plot

Recently, the majority of this paper has been finished, and this work is at the phase of writing. I’m confident that it will be accepted or published before I became a formal doctoral student.

One of result

Natural Language Processing

This project comprises 12 assignments in the class of Statistical Cases Studies. Ten of them are Individual assignment, one of them is a class presentation and the left one is a final project. I have had a basic grasp of processing flow and modeling method about NLP through several classes and those practices. More specifically, the skills I learned in this class are as following:

Collection for Text Data: Crawl data with python modules and frames, such as selenium，beautifulsoup，scrapy and so on.

Preprocessing Method for Text Data: Regular expression and some python modules, such as re, nltk, jieba and so on.

Model: Nonnegative Matrix Factor, Word2vector, N-gram, LDA and some methods derived from LDA.

Text2Vector Method: tf-idf, doc2bow, doc2vec, lsi, etc.

The reports, code and figures of those cases are stored in this repository.

Natural Language Processing

Network Evolution

1. Evolution detection in paper citation Network

This work belongs to the field of statistical modeling in complex network. I pay attention to the evolution of the paper citation network and choose the statistical citation network as our research object. To be specific, I use the papers published on the top4 journals of statistical field to construct a paper citation network which has a timeline and can reflect the evolution from 1998 to 2018. The total dynamic network includes about 6303 nodes and it is shown as following gif.

Dynamic Network of Statistical Research

It will discuss the evolution of citation network from two perspectives——community evolution and link prediction. As for community evolution, I intend to discuss dynamic evolution of each community on the basis of different time slice. In the section of community evolution, the events I want to discuss include generation, survival, division, merger, extinction, development of different communities. Because those communities are theme communities, discussing the evolution of community is just to discuss the evolution of statistical topics.

As for the link prediction, what I mainly intend to do is constructing statistical model on paper citation network on the basis of corrected block model (DCBM). To be specific, with the slice of network in different period, I want to simulate the adaptation of DCBM to estimate inter-group parameter and intra-group parameter of link generation. Besides, I also want to add more attributes into improved model so as to make it more accuracy and practical. With those estimated parameters in different periods, we can do some conclusion on the community evolution and even we can establish model on those parameters to predict the evolution trend in future statistical field.

This research is at the phase of statistical modeling by now. Some of trails can be followed in this repository.

2. CluBear

CluBear, an online education platform of data science, is founded by Wang Hansheng, a well-known professor of Statistics at Peking University in China. During postgraduate student period, I published two popular science cases on this platform, one of which entitled Talking about the Characteristics of Data Network Based on Erdos Number is about the complex network, other of which named Reading Big-Data Books from the Text Analysis Perspective is about the natural language processing. Both are highly praised by teachers and students.

Logo of Clubear

3. Bibliometric

Based on the publication data collected from Web of Science , I want to utilize those keywords belonging to those papers to construct a Co-word network. Because the project is in the conceptual stage, I only have a rough idea about this network. The figure as following is a preliminary attempt for co-word network. To be specific, I use the keywords from 34 journals to construct that network. The time span of this research is fixed from 1998 to 2018.

So-called co-word network considers those keywords as nodes in it and if two keywords appear together in a certain network there will be a link between two nodes. Especially, if the count of common appearance of two keywords is more than one, the weight of their link will be more than one. This is, the larger the count is, the larger their weight is. What I want to achieve with this network is to find out if there exists some laws in statistical field merging different research direction into cross-discipline. To follow the progress of this project, please visit this repository.

Coword Network