Information Theory and Machine Learning
Research activity report, 1998 - David MacKay
The fundamental problem of information theory is to communicate
reliably over an unreliable, noisy channel. We wish to do this
while adding the smallest possible redundancy to the transmitted message,
and without requiring astronomical computational resources at the
encoder or decoder.
Our research in this area in collaboration with Neal, Frey and McEliece
in Canada and USA has been outstandingly successful.
DJCM's paper on low density parity check codes is called
`a landmark paper' by one of the IEEE's referees, and
Matthew Davey, extending and developing these
codes, has created the best known error correcting code (as of June
(i.e., it can communicate over a Gaussian noise channel
using a smaller signal to noise ratio than any other code).
Matthew Davey and DJCM have published 3 papers (one in a refereed journal),
and DJCM has coauthored five other papers with other collaborators
in the last 3 years.
I am hoping to secure industrial funding for
further research in this area. Future directions include
applying our record breaking ideas to other error-correcting codes
and exploring the use of our codes in specific applications such
as mobile telephony.
Simon Wilson has worked on related
error correcting codes and is also investigating
the application of the decoding algorithm to other inference
problems such as medical disease diagnosis.
Empirical Modelling of non-linear relationships
Mark Gibbs (who finished his PhD in 1998) worked on
a `Gaussian process' approach to non-linear data modelling.
Significant advances in this work include
Neither of these advances has made it into a refereed journal
because of the hasty departure of Dr. Gibbs to the financial
world. Advice welcome.
a computer-efficient implementation of Gaussian processes
using John Skilling's approximate matrix inversion methods;
a variational technique allowing Gaussian processes to
be applied to classification problems.
- Harry Bhadeshia of Materials Science is an enthusiastic
collaborator who uses our neural network and Gaussian process
software to model complex metallurgical phenomena.
This collaboration has since 1995 produced eleven papers, including
eight in refereed journals.
- In a collaboration with Phil Withers of Materials Science
Coryn Bailer-Jones was hired on an 18 month postdoc
to work on modelling
of forging processes. Cold forging was
successfully modelled using
existing Gaussian process techniques. Modelling of
Hot forging required the development of new
dynamical models. These models have been developed but have
not yet been applied to real data.
Four papers have been written, and a new grant has been
written to continue this work.
Gaussian processes appear to be a viable alternative to feedforward
DJCM has written a review paper on Gaussian processes
and has been invited to give tutorials at several conferences.
Latent variable models
One problem in machine learning is to discover hidden variables
that are presumed to underlie visible data.
Two students in my group have worked on the following problems.
Blind signal separation, or Independent Component Analysis.
James is studying blind separation of mixed sources whose
marginal distributions are assumed to be non-Gaussian.
Applications include blind
deconvolution of audio signals and (in discussion with MPH)
blind deconvolution of microwave background images.
Latent variable models for protein sequences.
In this work we tried to make a model for recognising
sequences corresponding to `hairpin' structures
in proteins. Several interesting results have emerged,
but overall it seems that latent variables do not significantly
improve the pattern recognition capabilities of this model.
A new project aims to combine an adaptive language model
with an adaptive gesture-tracker so as to allow the
user efficiently to convey text or commands to a computer.
DJCM wrote a prototype interface in December 1997 and
industrial support is currently being sought.
Sanjoy Mahajan will join our group in August and will continue
his work on `how to teach so that the student retains
an understanding of physics in ten years time'.
David MacKay <email@example.com>
Last modified: Thu Jun 3 14:36:07 1999