Hello, dear friend, you can consult us at any time if you have any questions, add WeChat: YSYY66

M ultivariate Data Analysis

2021/2022

— C ou r se w o r k —

E x p l o ri ng k e r ne l r eg r e ss i on

1 F r a m e w o r k and m o t i va t i on

The objective of this coursework is to explore the numerical implementation and the properties of simple kernel regression models. To deepen our understanding of this type of models, we will design our own implementations and not rely on any already existing packages or toolboxes for kernel regression.

You are free to use the programming language of your choice. As we already dis- cussed, R or Python are perfectly adapted (and recommended) languages to complete this project. In order to make your pieces of code and experiments more readable, you may use a Jupyter Notebook environment; in particular, to help you start your study, two notebooks are available on Learning Central, one in R and one in Python, namely 2022_KernelRegX1_R and 2022_KernelRegX1_Py. The PDFs of these notebooks are also available, in case you would like to copy-paste the various batched of commands and use them in a diﬀerent programming environment (like RStudio or Spyder).

To run jupyter notebooks with R or Python on your laptop/desktop, you will need to install Jupyter on your machine; this can be done in diﬀerent ways, and is very well documented online (this can for instance be done with Anaconda, but also directly, i.e. without using Anaconda). Notice that by default, Jupyter will only provide you with Python notebooks; depending on your operating system, you may have to do a few extra operations to enable R notebooks (again, all this is very well documented on the internet).

As a remark, these notebooks use functions simulating the realisations of Gaus-

sian ramdon vectors of the type x ∼ N d ( a , E ) for a given mean vector a and a

given covariance matrix E (in R, this is done with the mvrnorm function, and with random.multivariate_normal in Python). Nevertheless, we could as well only rely only on functions allowing for the simulation of independent realisations of a standard real Gaussian random variable (i.e. independent realisations of a real random variable following a N(0 , 1) distribution). Indeed, from the Standardisation Procedure and the decomposition E = II T , we have

x I ξ + a , with ξ ∼ N d (0 , 6 d ),

so that the component of ξ consist of independent N(0 , 1) random variables (you may if you want try to implement such an approach).

2 Abou t t h e re po r t

You will have to write a short report presenting and discussing the experiments and the analyses you will have carried out; the assessment will be based on this report.

Ideally, you should try to write your report in LATEX; for this purpose, you may use the .tex ﬁle use to generate this document as a base model (it is available on Learning Central; you may also take inspiration from the Section 3.3 of the lecture notes); you may of course use a diﬀerent font size, modify the margin, etc. If you are not

already familiar with LATEX, this could be a good opportunity for you to learn it, but you might otherwise use Word, or R Markdown for instance.

There are no strict page limits for the length of the report: around 10 pages, including ﬁgures, should be ﬁne; please try to keep the length of your report below 15 pages; you may include large ﬁgures or additional illustrations in appendix. Try to be concise, precise and pertinent; pay attention to the quality of your report in both substance and form (i.e. content, organisation, quality of the writing and of the graphics, overall presentation and clarity of the discussion). S ub m it you r r epo rt i n PD F f o r m a t .

You can make this project alone, or in small group of two or three; as you wish. Do not forget to put your na m e ( s ) and s t u den t nu m be r( s ) at the beginning of your report, and submit only one report per group. Submission of the report will be via Learning Central (Assessment folder); the submission deadline will also be indicated on Learning Central. If you encounter any problem or have questions, please ﬁll free to contact me by email at gau t h i e r b @ c a r d i ﬀ . a c. u k .

3 E xpec ted w o r k

You should try to reproduce and deepen the experiments that are performed in the Section 3.3 of the lecture notes, while explaining and commenting your developments. You may in particular focus on the tasks listed below.

W h a t y ou n ee d t o do :

(a) Study the provided pieces of code: try to understand the operations they

perform, how they work and the way they are related to the theoretical devel- opments that we studied in the lecture notes.

(b) Build the conﬁdence region related to a given Gaussian process regression

model (see the notebooks and the lecture notes).

(c) Illustrate the impact of the choice of the kernel and of the kernel parameters; a list of kernels you might use is provided in the Section 4.1 of this document.

(d) Perform the maximum-likehood (ML) estimation of the covariance kernel parameter(s) on an exemple (see the lecture notes and the notebooks).

(e) Apply kernel regression on the dataset DataVolcano50.txt (see the note- books and Section 4.3 ).

W ha t you cou l d do t o go f u rt he r: (pick some items in the list)

(f) For a ﬁxed dataset (i.e. a ﬁxed set of sample locations and a ﬁxed response

vector), try to illustrate what happens when, in the kernel regression formulas,

the kernel matrix K is replaced by ( K + 口 2 6 ) for various values of 口 2 > 0. This

is related to the model described in the Question 2 of the exercise sheet for Chapter 3, i.e. the “ m o de l w it h ob s e r v a ti on no i s e ” . You could also introduce a (Gaussian) observation noise in your model (i.e. add a random perturbation to the true response vector), and try to estimate the corresponding noise parameter 口 2 (using ML estimation for instance) on a given example.

(g) Try to perform kernel regression with a ﬁnite-dimensional kernel. This is

related to the Question 3 of the exercise sheet for Chapter 3. For ' = ® , you may for instance consider the functions g 1 ( x ) = 1, g 2 ( x ) = x , g 3 ( x ) = x 2 ,

etc. (you can actually use the functions you want). You may also introduce an observation noise in your model.

(h) Perform an experiment illustrating the distribution of the ML estimator(s) of

the covariance-kernel parameter(s) (see the notebook).

(i) Illustrate and discuss what happens when the kernel K is replaced by aK , with a > 0. You may then try to estimate the new additional kernel parameter a .

(k) Try to estimate the covariance-kernel parameter(s) with l ea v e - one - ou t cr o ss- validation (see Section 4.4 ). You may also illustrate the distribution of this estimator.

(l) Perform any follow-up or complementary analysis or experiment that you think is interesting.

4 A dd iti ona l i n f o r m a ti on

Here are additional pieces of information regarding the experiments you may carry out and the redaction of your report.

4 . 1 Co var i a n ce ker n e l s

Any symmetric positive-semideﬁnite kernel can be used to perform kernel regression. Here is a list of popular kernels in Machine Learning.

● Gaussian kernel (' S ® d ):

K ( x, x ′ ) = e − 0 Ⅱ x − x Ⅱ ′ 2 (parameter 0 > 0).

● Laplace kernel (' S ® d ):

K ( x, x ′ ) = e − 0 Ⅱ x − x ′ Ⅱ (parameter 0 > 0).

● Matérn-3/2 (' S ® d ):

K ( x, x ′ ) =╱1 + d 、 e − d , with d = 0 ′ 3Ⅱ x − x ′ Ⅱ (parameter 0 > 0).

● Matérn-5/2 (' S ® d ):

K ( x, x ′ ) =╱1 + d + d 2 、 e − d , with d = 0 ′ 5Ⅱ x − x ′ Ⅱ (parameter 0 > 0).

● Modiﬁed-Brownian kernel (' S ® ≥ 0 , i.e. the positive or null reals): K ( x, x ′ ) = 1 + min( x, x ′ ) .

Inﬁnitely many other symmetric positive-semideﬁnite kernels exist; if you ﬁnd a kernel that you would like to use (while searching online for instance), you are of

course strongly encourage to try to implement it. You may in addition for instance

notice that if K ∈ ® ' × ' is a symmetric positive-semideﬁnite kernel, then for all f ∈ ® ' , so is the kernel ( x , x ′ ) f ( x ) K ( x , x ′ ) f ( x ′ ). Symmetric positive-

semideﬁnite kernels may also be summed or multiplied together (and tensor-product of kernel can be used to deﬁned kernel on product space).

4.2 Figures

The ﬁgures you will include in your report should be numbered and captioned; a similar remark also holds for tables if you intend at using some. Here is an example describing how to include a ﬁgure, see Figure 1 . For readability, do not hesitate to use legend and colours (the quality of the ﬁgures is part of the assessment criteria).

Figure 1: Caption of the ﬁgure.

4.3 Vo lca no

The last example provided in the Chapter 3 of the lecture notes is based on a 50-point data set named DataVolcano50.txt; this data set is available on Learning Central (as a remark, this data set was built from the volcano dataset provided by the R

package datasets). It consists in the ( x , y , z ) coordinates of 50 points in [0 , 1]2 × ® ;

more precisely, the data set comes under the form of a 50 × 3 matrix, and each row

of this matrix consists of the ( x, y, z ) coordinates of one point in the data set. We aim at predicting z as function of ( x, y ) ∈ [0 , 1]2 ; more details are provided in the

notebooks.