David MacKay

Environment and Energy 
Information Theory 
·Gallager codes & MN codes 
·Theory of Sparse Graph codes 
·Regular Gallager codes 
·Irregular Gallager codes 
·Turbo codes  
·Insertions and deletions 
·Quantum Codes 
·Cryptanalysis and Coding 
·Discussion documents 
·More Message-Passing 
Probabilistic Data Modelling 
·PhD thesis 
·Bayes for neural networks 
·ICA & Latent Variable Models 
·Bayes FAQ 
·Bayesian computations 
·Monte Carlo methods 
·Variational methods 
·Gaussian processes 
·Language, Proteins, Density nets 
·Images, Radar, Neuroscience 
·Materials science 
·Decision theory, Fair Elections 
Human-computer Interfaces 
Computational Neuroscience 
Evolution and Molecular Biology  
Miscellaneous work 
Software and Files 
Any Problems? 

Search :


David's Publications

What you can find here

This website includes Sustainable Energy - without the hot air

Information Theory, Inference, and Learning Algorithms
My papers and software are mirrored in North America.

Master site, UK | Mirror in Canada | Mirror in USA (under construction)

  • Most of these files are ".ps.gz", i.e. postscript files (.ps) compressed with gzip (.gz).
  • Since Mon 15/7/02, I have tried to provide PDF alternatives to all postscript files.
  • Since Sun 20/10/02, DJVU alternatives are also provided.
  • If you don't have gunzip on your machine, you can find gunzip executables for various systems here; the msdos executable is also here. Some users report that gsview32 is useful for viewing the ps.gz files.
    Further links about gzip and gunzip: gzip.org, gzip.html.
    Some users of windows machines find that if they download a ps.gz file by http then it does not unzip and display, but if they download it by ftp, it works fine. To convert an http URL to an ftp URL, please use the following pair as an example:


    djvu files are provided for some papers. [Download djView]

Environment and Energy

Sustainable Energy - without the hot air A popular book by David J.C. MacKay
Published on paper, 2 December 2008. Also available for free online.

See the book's website

[If that doesn't work, there's a local copy here]

You may also enjoy reading The Truth About Phone Chargers
and other thoughts in the Sustainable Energy - without the hot air Blog.

  • On July 2nd 2009 I gave a talk for CityForum describing a possible road-map for UK energy to 2050. There is a paper ("Plan C", pdf) and a small set of slides (html). This road-map includes timescales and build-rates.

For enthusiasts: One hundred pages of further notes on topics cut from the book while finishing it - including carbon, climate change, and strategies used for misleading people.


beagleModel.ps.gz. beagleModel.pdf. abstract.
`Consistency of The Mortality of Chronically-irradiated Beagles with the Linear No-Threshold Model'. [technical report, draft, version 1.0 published online Wed 12/11/14]


RSsolar.ps. RSsolar.pdf. abstract. | <- UK
` Solar energy in the context of energy use, energy transportation, and energy storage '. Technical report, similar to paper published July 2013 as follows: MacKay DJC. 2013 Solar energy in the context of energy use, energy transportation and energy storage. Phil Trans R Soc A 371: 20110431.


Kite.ps. Kite.pdf.
`Kite Power '.
TideEstimate.ps. TideEstimate.pdf. abstract. | <- UK |
` Under-estimation of the UK Tidal Resource '.
Lagoons.ps. Lagoons.pdf. abstract. | <- UK |
` Enhancing Electrical Supply by Pumped Storage in Tidal Lagoons '.


Erg06.ps.gz. Erg06.pdf. abstract. | <- UK | Canada -> | Erg06.ps.gz. Erg06.pdf. abstract.
` Ergodic Pumping: a mechanism to drive biomolecular conformation changes '.
by Robert S MacKay and David J C MacKay

Web-pages on various energy issues

Other useful resources on this website

A rough draft is currently available: all in one 7M pdf file | or in four parts

There is a special high resolution colour version (pdf) (38M)

For printing on a black'n'white printer there is a (not necessarily up to date) special monochrome version (pdf) (38M)

I've also written a 4-page executive summary - (postscript, 4 pages) | (pdf)

Corrections to draft 3.3.3

LOG of major changes

If the above links don't work, there is an older draft here: 8M pdf file

There is a website devoted to this book

Information Theory, Inference, and Learning Algorithms

(Hardback, 640 pages, Published September 2003)

Order your copy

Price: £35.00 / $60.00 from |CUP UK/USA| |amazon.co.uk/.com/.ca/.co.jp| | Barnes & Noble USA | booksprice. | fetchbook.info | allbookstores | biggerbooks | blackwells | directtextbook | kalahari.net (South Africa) | Special Paperback edition for South Asia.|

Download the book too

You can browse and search the book on Google books.

You may download The book in one file (640 pages):

U.K. english Canada canada South Africa south africa
PDF (A4) pdf (9M) (fourth printing, March 2005) pdf pdf
Postscript (A4) postscript (fourth printing, March 2005) (5M) postscript postscript
EPUB - experimental format epub file (fourth printing) (1.4M) ( ebook-convert book.pdf book.epub --isbn 9780521642989 --authors "David J C MacKay" --book-producer "David J C MacKay" --comments "Information theory, inference, and learning algorithms - experimental epub version 31.8.2014" --language "English" --pubdate "2003" --title "Information theory, inference, and learning algorithms" --cover ~/pub/itila/images/Sept2003Cover.jpg ) experimental epub file was created in UK 31 Aug 2014
DJVU djvu file (6M) (2nd printing) djvu file djvu file
(djvu information | Download djView)
Just the words (latex) [provided for convenient searching] (2.4M) (latex) (latex)
Just the figures NEW All in one file (postscript) [provided for use of teachers] (2M)
(pdf) (5M) (ps.gz) (pdf)
In individual eps files
Individual chapters postscript and pdf available from this page mirror mirror


  • Version 6.0 was released Thu 26/6/03; the book is finished. You are welcome to view the book on-screen. Version 6.0 was used for the first printing, published by C.U.P. September 2003.
  • Version 6.6 was released Mon 22/12/03; it will be used for the second printing, to be released January 2004. In this second printing, a small number of typographical errors were corrected, and the design of the book was altered slightly. Page-numbering generally remains unchanged, except in chapters 1, 6, and 28, where a few paragraphs, figures, and equations have moved around. All equation, section, and exercise numbers are unchanged.
  • Version 7.0 is the third printing (November 2004). Its only differences from the 2nd printing are a number of corrections, and the renaming of the chapter `Correlated Random Variables' to `Dependent Random Variables'.

Copyright issues: The book is copyright (c) Cambridge University Press. It has been available in bookstores since September 2003. The cover price in 2003 was 30 pounds (UK) and $50 (USA); in 2006, 35 pounds and $60 (USA).

Now the book is published, these files will remain viewable on this website. The same copyright rules will apply to the online copy of the book as apply to normal books. [e.g., copying the whole book onto paper is not permitted.]

Draft 1.1.1 - March 14 1997.
Draft 1.2.1 - April 4 1997.
Draft 1.2.3 - April 9 1997.
Draft 1.2.4 - April 10 1997. Margins altered so as to print better on Northamerican paper
Draft 1.3.0 - December 23 1997.
Draft 1.9.0 - Feb 1 1999.
Draft 2.0.0 - Jan 6 2000. New page layout.
Draft 2.2.0 - Dec 23 2000. Fresh draft.
Draft 3.1415 - Jan 12 2003. Nearly finished.
Draft 4.0 - April 15 2003. Chapter sequence finalized.
Draft 4.1 - April 18 2003. Adding new frontmatter (Preface etc)to book. Corrected printing of letter version.
Version 6.0 - Thu 26 June 2003. Final version.
Version 6.6 - Mon 22 December 2003. Second printing.

  • Here is my method for converting to two-up under linux:
    pstops '4:0L@.67(20cm,1cm)+1L@.67(20cm,15cm),3R@.67(1cm,15.25cm)\
    +2R@.67(1cm,29.25cm)' $*.ps $*.dps 

Gallager codes, MN codes and review papers on sparse graph codes

mncEL.ps.gz. | PDF. | DJVU. | abstract. | ps mirror, Canada | | PDF | | DJVU |
` Near Shannon Limit Performance of Low Density Parity Check Codes '.
With Radford Neal. (Very short paper, published in Elec. Lett.)
mncN.ps.gz. (55 pages) | PDF | DJVU | abstract | Chinese translation by Dong Xiangyu (rioshering-at-hotmail.com) | | ps mirror, Canada | pdf | DJVU |
` Good Error-Correcting Codes based on Very Sparse Matrices '.
Appeared in IEEE-IT, 1999. Also, with Radford Neal, an early massively shortened version of this paper (published in the 1995 book `Cryptography and Coding' LNCS 1025): mnc4s.ps.gz. | PDF | | DJVU | | ps mirror, Canada | | pdf | DJVU |
mncNcorrns.ps.gz. (1 page) | PDF | DJVU | mncNcorrns.tex. | ps mirror, Canada | | pdf | DJVU |
` Errata for the paper:
Good Error-Correcting Codes based on Very Sparse Matrices
(Contains 4 corrections to the published version)
gfqoptimize.ps.gz. gfqoptimize.pdf. abstract. | <- UK | Canada -> | gfqoptimize.ps.gz. gfqoptimize.pdf. abstract.
`Optimizing Sparse Graph Codes over GF(q)'. (3 pages) (2003)
More on Gallager Codes over GF(q): see Matthew Davey's web pages
dilbert3.ps.gz | PDF | DJVU | Canada -> | dilbert3.ps.gz. pdf | | DJVU |
`Error-correcting codes and probability propagation'. (16 pages of lecture notes for the 2001 meeting on Statistics and Artificial Intelligence) | Also accumulator networks.
sparsecodes0.ps.gz. | PDF | | DJVU | abstract. | <- UK | Canada -> | sparsecodes0.ps.gz. abstract.
` Sparse Graph Codes '. (3 page summary paper)
sparsecodes.ps.gz. | PDF | | DJVU | abstract. | <- UK | Canada -> | sparsecodes.ps.gz. abstract.
` Gallager Codes - Recent Results '. (12 page paper)
encyclopedia.ps.gz. | PDF | | DJVU | abstract. | <- UK | Canada -> | encyclopedia.ps.gz. abstract.
` Encyclopedia of Sparse Graph Codes '.
Encyclopedia of Sparse Graph Codes (hypertext archive, under construction)
rev.ps.gz. | PDF | | DJVU | abstract. | <- UK | Canada -> | rev.ps.gz. abstract.
` A Revolution: Belief Propagation in Graphs with Cycles '.
Brendan J. Frey and David J.C. MacKay. Presented at NIPS 1997.
ibis.ps.gz. | PDF | | DJVU | abstract. | <- UK | Canada -> | ibis.ps.gz. abstract.
` Relationships between Sparse Graph Codes '. This Paper appears in the (unpublished) proceedings of IBIS 2000, Japan.

I have also put a mirror of Gallager's classic book on my website, Gallager, R. G., Low Density Parity Check Codes, Monograph, M.I.T. Press, 1963.

[See also the big paper on Gallager codes and MN codes.]
mceliece.ps.gz | TBSC-Decoding.pdf (old pdf file). | PDF | | DJVU |
BSC Thresholds for Code Ensembles based on `Typical pairs' decoding.
S. Aji, H. Jin, A. Khandekar, R. J. McEliece, and D. J. C. MacKay. In proceedings of IMA meeting 1999.
theorems.ps.gz | djvu | abstract | <- UK | Canada -> | theorems.ps.gz djvu | abstract |
`On Thresholds of Codes'.
fountain.ps.gz. fountain.pdf. abstract. | <- UK | Canada -> | fountain.ps.gz. fountain.pdf. abstract.
` Fountain Codes '. (2004)

Experimental work on regular Gallager codes

[See also earlier papers on Gallager codes and MN codes.]
R3.ps.gz. | PDF | | DJVU | abstract. | <- UK | Canada -> | R3.ps.gz. abstract.
` Two Small Gallager Codes '. with Matthew Davey
sensit.ps.gz. | PDF | | DJVU | abstract. | <- UK | Canada -> | sensit.ps.gz. abstract.
` Sensitivity of Low Density Parity Check Codes to Decoding Assumptions '.
David J.C. MacKay and Christopher P. Hesketh.
margulis.ps.gz. margulis.pdf. | margulis.DJVU. abstract. | <- UK | Canada -> | margulis.ps.gz. margulis.pdf. | margulis.DJVU. abstract.
` Weaknesses of Margulis and Ramanujan-Margulis Low-Density Parity-Check Codes '.
David J.C. MacKay and Michael S. Postol.

High Rate Codes

seagate.ps.gz. | PDF | | DJVU | abstract. | <- UK | Canada -> | seagate.ps.gz. abstract.
` Evaluation of Gallager Codes for Short Block Length and High Rate Applications '. with Matthew Davey.
highrate.ps.gz. highrate.pdf. abstract. | <- UK | Canada -> | highrate.ps.gz. highrate.pdf. abstract.
` Gallager Codes for High Rate Applications '.
Research note written with Ed Ratzer, Jan 2003. Looks in more detail at high rate codes.

Irregular Gallager codes and Repeat-Accumulate Codes

ldpc-irreg.ps.gz. | PDF | | DJVU | abstract. | <- UK | Canada -> | ldpc-irreg.ps.gz. abstract.
` Comparison of Constructions of Irregular Gallager Codes '.
David MacKay, Simon Wilson and Matthew Davey.
Published in IEEE Transactions on Communications (submitted 30 July 1998) and in Proceedings of the 1998 Allerton Conference on Communication, Control, and Computing.
irreg-its.ps.gz. | PDF | | DJVU | abstract. | <- UK | Canada -> | irreg-its.ps.gz. abstract.
` Decoding Times of Irregular Gallager Codes '.
ra-its.ps.gz. | PDF | | DJVU | abstract. | <- UK | Canada -> | ra-its.ps.gz. abstract.
` Decoding Times of Repeat-Accumulate Codes '.
puncture.ps.gz. | PDF | | DJVU | abstract. | <- UK | Canada -> | puncture.ps.gz. abstract.
` Punctured and Irregular High-Rate Gallager Codes '.

Turbo codes

BPTD.ps.zip (old USA site). BPTD.ps (USA master site). BPTD.ps.gz (UK mirror, not nec. up to date). BPTD.ps.gz (Canada mirror).
Turbo Decoding as an Instance of Pearl's `Belief Propagation' Algorithm.
R. J. McEliece, D. J. C. MacKay and J.-F. Cheng. In IEEE Journal on Selected Areas in Communication.
turbo-ldpc.ps.gz. | PDF | | DJVU | abstract. | <- UK | Canada -> | turbo-ldpc.ps.gz. abstract.
` Turbo Codes are Low Density Parity Check Codes '.
This material was included in other papers, such as the big one (mncN.ps.gz).
tcc-al.ps.gz. | PDF | | DJVU | no abstract. | <- UK | Canada -> | tcc-al.ps.gz. no abstract.
` Trellis-constrained codes '.
Brendan J. Frey and David J.C. MacKay. Presented at Allerton 1997.
postscript. | <- UK | Canada -> | abstract.
` Irregular turbocodes '. with Brendan Frey. (in Proceedings of the 37th Allerton Conference on Communication, Control and Computing 1999, Allerton House, Illinois.)
G4096.ps.gz. G4096.pdf. | G4096.DJVU. abstract. | <- UK | Canada -> | G4096.ps.gz. G4096.pdf. | G4096.DJVU. abstract.
` (4096,3249) Gallager Codes compared with Tanner Product Codes '. `

Runlength limiting codes, & codes for correcting insertions and deletions

PDF | | DJVU | abstract. | <- UK |
` "Watermark codes": reliable communication over channels with insertions, deletions, and substitutions '. Matthew Davey and David MacKay. One page paper in ISIT2000.
`Reliable communication over channels with insertions, deletions, and substitutions.' Paper appeared in IEEE-IT Feb 2001. See also Matthew Davey's web pages
ERRATUM: (19th March 2001) In the published version of the above IEEE-IT paper, figure 15 contained a minor error, now corrected on the file on this website. Both the dotted `Zigangirov' curves were a tiny bit higher than they ought to be. You can see the difference between the old curves and the corrected curves in this figure and you can also obtain the corrected curves or the entire corrected page.
The error has no influence on the substance of the paper.
This error arose because our definition of the deletion rate (pd) differs by a factor (1-pi) from that of Zigangirov, where pi is the insertion rate.
Thanks to Dave Leigh.
rll01.ps.gz. rll01.pdf. | rll01.DJVU. abstract. | <- UK | Canada -> | rll.ps.gz. rll01.pdf. | rll01.DJVU. abstract.
` Almost-certainly Runlength-limiting Codes '.
in proceedings of the IMA Cryptography and Coding Conference 2001, copyright Springer
rllt.ps.gz. | PDF | | DJVU | abstract. | <- UK | Canada -> | rllt.ps.gz. abstract.
` An Alternative to Runlength-limiting Codes: Turn Timing Errors into Substitution Errors '.
istc2000.ps.gz. | PDF | | DJVU | abstract. | <- UK | Canada -> | istc2000.ps.gz. abstract.
` Codes for Channels with Insertions, Deletions and Substitutions '. by Ed Ratzer and David MacKay. Presented at ISTC 2000, Brest. [See also

Quantum Error-Correcting Codes

As submitted to journal: qecc.ps.gz. qecc.pdf. abstract. | <- UK | Canada -> | qecc.ps.gz. qecc.pdf. abstract.
`Sparse Graph Codes for Quantum Error-Correction'.
by David MacKay, Graeme Mitchison, and Paul McFadden.
quant-ph/0304161. IEEE Transactions on Information Theory, vol 50, no 10, in press.
Extended version: qeccE.ps.gz. qeccE.pdf. abstract. | <- UK | Canada -> | qeccE.ps.gz. qeccE.pdf. abstract.
`Sparse Graph Codes for Quantum Error-Correction'.
by David MacKay, Graeme Mitchison, and Paul McFadden.
quant-ph/0304161. Extended version of paper appearing in IEEE Transactions on Information Theory.
cayley.ps.gz. cayley.pdf. abstract. | <- UK | Canada -> | cayley.ps.gz. cayley.pdf. abstract.
` More Sparse-Graph Codes for Quantum Error-Correction '.
by David J C MacKay, Amin Shokrollahi, Oliver Stegle, and Graeme Mitchison

Early work on Cryptanalysis and Coding

fe.ps.gz. | PDF | | DJVU | abstract | ps mirror, Canada |
`A Free Energy Minimization Framework for Inference Problems in Modulo 2 Arithmetic'. Also a short version of the same paper: fes.ps.gz, | PDF | | DJVU | | ps mirror, Canada | appeared in Electronics Letters, title `A Free Energy Minimization Algorithm for Decoding and Cryptanalysis'. Or a short version, including pseudocode in appendix: fem.ps.gz. | PDF | | DJVU | | ps mirror, Canada |

Submissions to ISIT97 (all rejected)
muleisit.ps.gz. | PDF | | DJVU | abstract. | <- UK | Canada -> | muleisit.ps.gz . abstract.
` Shortened Turbo Codes '. (A one-page summary)
scc.ps.gz. | PDF | | DJVU | abstract. | <- UK | Canada -> | scc.ps.gz . abstract.
` Decoding Shortened Cyclic Codes by Belief Propagation '. (with Simon T. Wilson) (One-page summary)
gfq.ps.gz. | PDF | | DJVU | abstract. | <- UK | Canada -> | gfq.ps.gz . abstract.
` Low Density Parity Check Codes over GF(q) '. (One-page summary) (with M.C. Davey)

Discussion documents

replica.ps.gz. | PDF | | DJVU | abstract. | <- UK | Canada -> | replica.ps.gz. abstract.
` Thresholds of Low-Density Parity Check Codes (Discussion Document) '.
bethe.ps.gz. | PDF | | DJVU | abstract. | <- UK | Canada -> | bethe.ps.gz. abstract.
` Questions about the Bethe Free Energy and Sum-Product (Discussion Document) '.
With Jonathan Yedidia, William Freeman, and Yair Weiss
xword.ps.gz. xword.pdf. abstract. | <- UK | Canada -> | xword.ps.gz. xword.pdf. abstract.
` Counting Crosswords '. by David J C MacKay and Jeremy Thorpe
Draft 2.5 uploaded Wed 19/4/06
covariance.ps.gz. covariance.pdf. abstract. | <- UK | Canada -> | covariance.ps.gz. covariance.pdf. abstract.
` How to Parameterize a Two--Dimensional Covariance Matrix '.

Other applications of Message-Passing

Papers written by students in my group

Solving the Satisfiability Problem using Message-Passing Techniques
by Sam J Pumphrey (2001)
Solving Combinatorial Problems using Message-Passing
by Ben Young (2001)

PhD Thesis (Caltech 1991) and related papers:

My thesis consisted of four published papers, one unpublished chapter, and `summary' and `postscript' chapters. If you are only interested in one published chapter then you might prefer to download the single `Neural Computation' files listed after each chapter below. The thesis version is to be preferred, however, as it contains slightly more information. The whole thing, postscript | djvu | pdf | Also available from Caltech library |
pp. i-vi (30K); |DJVU|
Chapter 1: Summary
pp. 01-06 (40K); |DJVU|
Chapter 2: Bayesian Interpolation
pp. 07-17 (86K); |DJVU|
pp. 18-25 (68K); |DJVU|
pp. 26-33 (91K); |DJVU|
or as published in Neural Computation 4 3 415-447 (188K). |DJVU|
Chapter 3: A Practical Bayesian Framework for Backprop Networks
pp. 34-43 (94K); |DJVU|
pp. 44-52 (71K); |DJVU|
or as published in Neural Computation 4 3 448-472 (144K).| |DJVU|
Chapter 4: Information-based objective functions for active data selection
pp. 53-64 (122K); |DJVU|
or as published in Neural Computation 4 4 589-603 (125K). |DJVU|
Chapter 5: The evidence framework applied to classification networks
pp. 65-69 (85K); |DJVU|
pp. 70-70 (195K); |DJVU|
pp. 71-78 (191K); |DJVU|
or as published in Neural Computation 4 5 698-714 (456K). |DJVU|
Chapter 6: Inferring input-dependent noise levels
pp. 79-81 (34K); |DJVU|
Chapter 7: Postscript
pp. 82-86 (39K); |DJVU|
pp. 87-92 (30K). |DJVU|

Bayesian methods for neural networks

See also my PhD thesis
network.ps.gz, | PDF | DJVU | abstract | ps mirror, Canada |
`Probable Networks and Plausible Predictions - A Review of Practical Bayesian Methods for Supervised Neural Networks'
This is the final version of a review paper based on a book chapter in a Springer publication entitled "Bayesian methods for backpropagation networks"; it appeared as a commissioned review article in Network (IOPP).
cpi_short.ps.gz | PDF | DJVU | | ps mirror, Canada |
Bayesian Non-Linear Modelling with Neural Networks
A review paper giving a basic introduction to neural networks and then describing Bayesian methods, with two case studies.
nips91.ps.gz.(9 pages) | | PDF | DJVU | | ps mirror, Canada| pdf. | DJVU. |
` Bayesian Model Comparison and Backprop Nets '.
NIPS paper reviewing Bayesian methods, and including discussion of the relationship to the `Effective VC dimension' in Structural Risk Minimization.
cpi4.ps.gz.(791K,43 pages) | PDF | DJVU | abstract. | ps mirror, Canada |
`Bayesian Methods for Neural Networks: Theory and Applications'.
Course notes for Neural Networks Summer School
hme.ps.gz up to date links are here
`Bayesian methods for Mixtures of Experts'
Steve Waterhouse, David MacKay and Tony Robinson
objective.ps.gz. | PDF | DJVU | abstract. | <- UK | Canada -> | objective.ps.gz. abstract.
` Searching for `optimal' inputs with an empirical regression model '.
nnbm.ps.gz. | PDF | DJVU |
`The Nonnegative Boltzmann Machine' with Oliver Downs and Daniel Lee. (NIPS 99)

Applications of Bayesian neural networks

pred.ps.gz, | PDF | DJVU | abstract | ps mirror, Canada |
Bayesian Non-linear Modeling for the Energy Prediction Competition
approx.ps.gz. approx.pdf. abstract. | <- UK | Canada -> | approx.ps.gz. approx.pdf. abstract.
` Bayesian Approximation Theory '. (2004)

(see also the Language, Proteins, Density nets page)

Independent Component Analysis (and other latent variable models)

ica.ps.gz. | PDF | DJVU | abstract. | <- UK | Canada -> | ica.ps.gz. abstract.
` Maximum Likelihood and Covariant Algorithms for Independent Component Analysis '. (see also Barak Pearlmutter's work)
Unpublished paper.
PhD Thesis on Bayesian Independent Component Analysis
by James Miskin
icagenes.ps.gz | pdf | DJVU. abstract. | <- UK | Canada -> | icagenes.ps.gz. | pdf. | djvu. abstract.
` Latent Variable Models for Gene Expression Data '.
With James Miskin

Generalized Boltzmann machines / Contrastive Divergence

rbm.ps.gz. rbm.pdf. abstract. | <- UK | Canada -> | rbm.ps.gz. rbm.pdf. abstract.
` Are Restricted Boltzmann Machines Universal? '. with Marcus Frean and Philip Sterne
gbm.ps.gz | pdf | DJVU | abstract. | <- UK | Canada -> | gbm.ps.gz. pdf. | DJVU. abstract.
`Failures of the One-Step Learning Algorithm'.


boost.ps.gz. boost.pdf. abstract. | <- UK | Canada -> | boost.ps.gz. boost.pdf. abstract.
`Likelihood-based Boosting'. With Radford Neal (Unpublished paper, 1998)

Computational Implementation of Bayesian methods

alpha.ps.gz, | PDF | DJVU | abstract | ps mirror, Canada | pdf | djvu |
Hyperparameters: optimize, or integrate out?
This paper appeared in shortened form in the 1993 Maxent proceedings. The long version underwent a lengthy refereeing process with Neural Computation, where it was published (1999) under the title Comparison of Approximate Methods for Handling Hyperparameters.
laplace.ps.gz. | PDF | DJVU | abstract. | <- UK | -> Canada | laplace.ps.gz. abstract.
`Choice of Basis for Laplace Approximation'.
Published in Machine Learning. Volume 33, No. 1, October 1998
(Submitted October 14th 1996)
delve.ps.gz. | PDF | DJVU | abstract. | <- UK | Canada -> | delve.ps.gz. abstract.
` More sensitive tests in DELVE'.

See also

Monte Carlo methods

erice.ps.gz. | PDF | DJVU | abstract. | <- UK | Canada -> | erice.ps.gz. abstract.
` Introduction to Monte Carlo methods '.
A review paper in the proceedings of an Erice summer school, ed. M.Jordan.
paris.ps.gz. | PDF | DJVU | abstract | ps mirror, Canada. abstract |
` Probabilistic Networks: New Models and New Methods '.
A paper describing an interpolation model with input-dependent noise level, implemented using `BUGS'. This paper is for ICANN95 in Paris. For more information about the BUGS program, see ftp.mrc-bsu.cam.ac.uk.
astro.ps.gz, | PDF | DJVU | astro.tar | ps mirror, Canada |
Inferring the distance to the Virgo cluster from Cepheid data using the BUGS program. (Bayesian inference using Gibbs Sampling). Currently this sketched report uses mock data.
Infrared magnitude-redshift relations for luminous radio galaxies
K. J. Inskip, P. N. Best, M. S. Longair, and D. J. C. MacKay
This is an electronic version of an article published in Monthly Notices of the Royal Astronomical Society: complete citation information for the final version of the paper, as published in the print edition of Monthly Notices of the Royal Astronomical Society, is available on the Blackwell Science Synergy online delivery service, accessible via the journal's Website at: http://www.blacksci.co.uk/MNR.
Exact sampling from antiferromagnetic spin systems
Ryan Patterson, Andrew Childs, Radford Neal and David MacKay. This paper was published in Phys Rev E vol 63, 036113, under the title Exact Sampling from nonattractive distributions using summary states.
slice.ps.gz | pdf | DJVU. abstract. | <- UK | Canada -> | slice.ps.gz | pdf | DJVU | abstract.
` Slice Sampling - a Binary Implementation '.
By John Skilling and David MacKay
To appear in Annals of Statistics
nested.ps | nested.pdf |
` Nested Sampling - a tutorial '.

Variational methods

ensemble.ps.gz. | PDF | DJVU | abstract. | ps mirror, Canada |
` Developments in Probabilistic Modelling with Neural Networks - Ensemble Learning '.
nips.ps.gz. | PDF | DJVU | abstract | ps mirror, Canada |
`Ensemble Learning and Evidence Maximization'.
ensemblePaper.ps.gz. | PDF | | DJVU | abstract. | <- UK | Canada -> | ensemblePaper.ps.gz. abstract.
` Ensemble Learning for Hidden Markov Models '.
vgc.ps.gz. | PDF | | DJVU | abstract. | <- UK | Canada -> | vgc.ps.gz. abstract.
` Variational Gaussian Process Classifiers '.
With M.N. Gibbs.
minima.ps.gz. minima.pdf. | minima.DJVU. abstract. | <- UK | Canada -> | minima.ps.gz. minima.pdf. | minima.DJVU. abstract.
` A Problem with Variational Free Energy Minimization '. (first published Sat 23/6/01)
compact.ps.gz. compact.pdf. abstract. | <- UK | Canada -> | compact.ps.gz. compact.pdf. abstract.
` Compactness of variational approximations '. By David J C MacKay, Richard Turner, and Maneesh Sahani

Gaussian processes

abstract. | <- UK | Canada -> | gpros.ps.gz . abstract.
` Efficient implementation of Gaussian processes. '.
With M.N. Gibbs. Draft paper. was `Efficient methods for Interpolation with Gaussian Processes'
See also Mark's Gaussian process web site
vgc.ps.gz. | PDF | | DJVU | abstract. | <- UK | Canada -> | vgc.ps.gz. abstract.
` Variational Gaussian Process Classifiers '.
With M.N. Gibbs.
See also Mark's Gaussian process web site
gp.ps.gz. | PDF | | DJVU | abstract. | <- UK | Canada -> | gp.ps.gz. abstract.
` Gaussian Processes - A Replacement for Supervised Neural Networks?'.
Lecture notes for a tutorial at NIPS 1997. More about Gaussian processes.
gpB.ps.gz. | PDF | | DJVU | abstract. | <- UK | Canada -> | gpB.ps.gz. abstract.
` Introduction to Gaussian Processes '.
This is a later version of the above lecture notes, to appear in proceedings of a NATO school.

(see also the ICA & Latent Variable Models page)

Language modelling and Hidden Markov models

lang4.ps.gz, | PDF | | DJVU | abstract | ps mirror, Canada |
`A Hierarchical Dirichlet Language Model.' with L.Peto
amino.ps.gz. | PDF | | DJVU | abstract. | <- UK | Canada -> | amino.ps.gz. abstract.
` Models for Dice Factories and Amino Acid Probabilities. '. (Unfinished technical report from 1994; this work led to my work on density networks for protein sequences.)
Also, an unfinished addendum, amino_is.ps.gz. | PDF | | DJVU |

Hidden Markov models

jordan.ps.gz. abstract. | <- UK | Canada -> | jordan.ps.gz. | PDF | | DJVU | abstract.
` Equivalence of Linear Boltzmann Chains and Hidden Markov Models '.
ensemblePaper.ps.gz. abstract. | <- UK | Canada -> | ensemblePaper.ps.gz. | PDF | | DJVU | abstract.
` Ensemble Learning for Hidden Markov Models '.

Protein sequence modelling and Density Networks

ch_learning.ps.gz. abstract. | <- UK | Canada -> | ch_learning.ps.gz. | PDF | | DJVU | abstract.
` Bayesian Neural Networks and Density Networks '.
density.ps.gz, | PDF | | DJVU | abstract | ps mirror, Canada |
`Density Networks and their application to Protein Modelling' (this version appeared in Maximum Entropy Proceedings)
density97.ps.gz. | PDF | | DJVU | abstract. | <- UK | Canada -> | density97.ps.gz. abstract.
D. MacKay and M. Gibbs (1997): ` Density Networks '. (this version in proceedings of Edinburgh meeting, ed. Jim Kay)

Modelling of images and radar

nn_im_decon.ps.gz, | PDF | | DJVU | abstract | ps mirror, Canada |
`Neural Network Image Deconvolution', by John E. Tansley, Martin J. Oldfield and David J.C. MacKay
radar3.ps.gz. | PDF | | DJVU | abstract | ps mirror, Canada |
`Bayesian analysis of linear phased-array radar' by A.G. Green and D.J.C. MacKay.
i3.ps.gz (494K). | PDF | | DJVU | abstract | ps mirror, Canada |
A. Barnett and D. MacKay: `Bayesian Comparison of Models for Images'.

Neuroscience Data Analysis

newint.ps.gz. | PDF | | DJVU | abstract. | <- UK | Canada -> | newint.ps.gz. abstract.
D. MacKay and R. Takeuchi: `Interpolation models with multiple hyperparameters'. (Published in Statistics and Computing)
The spatial arrangement of cones in the fovea: Bayesian analysis (unpublished research note, 25.3.1993, discussing data of Mollon and Bowmaker - Are cones arranged randomly in the retina?)
(7 pages) cones.ps | cones.pdf
shuffle.ps.gz. | PDF | | DJVU | abstract. | <- UK | Canada -> | shuffle.ps.gz. abstract.
` A Comment on Data Shuffling '.
by David J. C. MacKay, Christopher deCharms and Virginia R. de Sa.
cnssumm.ps.gz. | PDF | <- UK | USA -> | | DJVU | <- UK | USA -> | Ginny's site |
Old draft: bridge.ps.gz. abstract. | <- UK | Canada -> | bridge.ps.gz. abstract.
` Model fitting as an Aid to Capacitance Compensation and Bridge Balancing in Neuronal Recording '.
by David J. C. MacKay and Virginia R. de Sa. - Publication details:
de Sa, V.R., & MacKay, D.J.C. (2001). Model fitting as an Aid to Bridge Balancing in Neuronal Recording. Neurocomputing (special issue devoted to Proceedings of the CNS 2000 meeting) Vol38-40, 1651-1656.
  • Time-warp-invariant computation with action potentials: Deductions about the Hopfield-Brody Mouse by David MacKay and Seb Wills
    In December 2000, the Inference Group won Hopfield and Brody's `mouse brain' competition.

  • Dynamical Neural Networks

    dynet.ps.gz. | PDF | | DJVU | | <- UK | Canada -> | dynet.ps.gz. | Abstract (in Germany)
    ` A Recurrent Neural Network for Modelling Dynamical Systems '.
    by Coryn A.L. Bailer-Jones, David J.C. MacKay, Philip J. Withers

    Papers for which postscript files are not available here, but may be available in Materials Science

    H.K.D.H. Bhadeshia, D.J.C. MacKay, and L.E. Svensson.
    Impact toughness of C-MN steel arc welds - Bayesian neural network analysis.
    Materials Science and Technology, 11(10):1046-1051, 1995.

    L.Gavard, H.K. D.H. Bhadeshia, D.J.C. MacKay, and S.Suzuki.
    Bayesian neural network model for austenite formation in steels.
    Materials Science and Technology, pages 453-463, 1996.

    H.Fujii, D.J.C. MacKay, and H.K. D.H. Bhadeshia.
    Bayesian neural network analysis of fatigue crack growth rate in nickel base superalloys.
    ISIJ International, 36(11):1373-1382, 1996.

    T.Cool, H.K.D.H. Bhadeshia, and D.J.C. MacKay.
    The yield and ultimate tensile strength of steel welds.
    Materials Science and Engineering A, A223:186-200, 1997.

    J.Jones, J.King, H.K.D.H. Bhadeshia, and D.J.C. MacKay.
    Modelling the strength of Nickel base superalloys.
    3rd International Parsons Turbine Conference, 1995.

    T.Cool, H.K.D.H. Bhadeshia, and D.J.C. MacKay.
    Modelling the mechanical properties in the HAZ of power plant steels i: Bayesian neural network analysis of proof strength.
    In H.Cerjak, editor, Mathematical Modelling of Weld Phenomena 3, Materials Modelling Series, pages 403-442. Institute of Materials, London, 1997.

    D.J.C. MacKay.
    Bayesian non-linear modelling with neural networks.
    In H.Cerjak, editor, Mathematical Modelling of Weld Phenomena 3, Materials Modelling Series, pages 359-389. Institute of Materials, London, 1997.

    A.Y. Badmos, H.K.D.H. Bhadeshia, and D.J.C. MacKay.
    Neural network models for the tensile properties of mechanically alloyed ODS iron-alloys. Materials Science and Engineering A. Accepted for publication, 1997.

    F. Brun, T. Yoshida, J. D. Robson, V. Narayan, H.K.D.H. Bhadeshia and D. J. C. MacKay.
    Theoretical Design of Creep Resistant Steels. 1997 (Submitted).
    And many more....

    Decision theory

    _decision.ps | pdf | DJVU
    Decision theory - a simple example

    Reinforcement learning

    RLcoin.ps.gz. RLcoin.pdf. abstract. | <- UK | Canada -> | RLcoin.ps.gz. RLcoin.pdf. abstract.
    ` Solution of a Toy Problem by Reinforcement Learning '.

    Fair Elections

    I26P3.pdf | abstract
    Probabilistic electoral methods, representative probability, and maximum entropy.
    Roger Sewell, David MacKay, Iain McLean
    This paper was accepted (Dec 2008) for publication in Voting Matters. The published paper (Issue 26, January 2009) is available as a pdf file. | abstract
    [This paper was formerly titled 'A maximum entropy approach to fair elections' RFSpdf.pdf (sorry, no longer available online)]

    Human-computer Interfaces


    The inference group Dasher site

    My main work in HCI has been on Dasher.

    Dasher - a Data Entry Interface Using Continuous Gestures and Language Models
    by David J Ward, Alan F Blackwell and David J C MacKay
    Abstract | Compressed postscript file. | PDF |

    eyeshortpaper.ps.gz. eyeshortpaper.pdf. | eyeshortpaper.DJVU. abstract. | <- UK | Canada -> | eyeshortpaper.ps.gz. eyeshortpaper.pdf. | eyeshortpaper.DJVU. abstract.
    ` Hands-free Text Entry using Inverse Arithmetic Coding with an Eyetracker '.
    eyeflyer.ps.gz. eyeflyer.pdf. | eyeflyer.DJVU. abstract. | <- UK | Canada -> | eyeflyer.ps.gz. eyeflyer.pdf. | eyeflyer.DJVU. abstract.
    ` Hands-free Text Entry using Dasher with an Eye-tracker '.
    One-page flyer by David J. Ward and David J.C. MacKay.

    dasherButtons.ps.gz. dasherButtons.pdf. abstract. | <- UK | Canada -> | dasherButtons.ps.gz. dasherButtons.pdf. abstract.
    ` Efficient communication through one or two buttons '.
    by David MacKay, Chris Ball, and Mick Donegan.
    dasherBreath.ps.gz. dasherBreath.pdf. abstract. | <- UK | Canada -> | dasherBreath.ps.gz. dasherBreath.pdf. abstract.
    ` Efficient communication by breathing '.
    by Tom H. Shorrock, David J.C. MacKay, and Chris J. Ball
    Published in The Sheffield Machine Learning Workshop, LNCS 3635 (c) Springer Verlag
    OneButton.ps.gz. OneButton.pdf. abstract. | <- UK | Canada -> | OneButton.ps.gz. OneButton.pdf. abstract.
    ` Dasher's One-button Dynamic Mode -- Theory and Preliminary Results '. by David J.C. MacKay and Chris J. Ball
    ` Efficient Communication Through The Timings of One Or Two Buttons '. by JAMES M. G. MEAD and PHILIP J. COWANS and DAVID J. C. MACKAY
    OneButton2.ps.gz. OneButton2.pdf. abstract. | <- UK | Canada -> | OneButton2.ps.gz. OneButton2.pdf. abstract.
    ` Another One-button Dynamic Mode for Dasher: `Two-click mode' (2007) '.

    Korean.pdf. Korean.ps.gz.
    Dasher in Korean - a six-page paper by David MacKay.

    Dasher - an efficient writing system for brain-computer interfaces? (pdf), S. A. Wills and David J. C. MacKay. IEEE Transactions on Neural Systems and Rehabilitation Engineering 14 (2), 244-246 (2006). [see also the brain dasher page]

    David Ward's Dasher site and experimental results

    Tapir - a free disambiguating keyboard

    Disambiguating keyboards

    A note about the iTap / T9 text-to-numbers code
    Can Reduced-keyboard disambiguating text-entry handle spelling errors? - notes by David MacKay and Dale Grover

    Dasher - a Data Entry Interface Using Continuous Gestures and Language Models
    by David J Ward, Alan F Blackwell and David J C MacKay

    Existing devices for communicating information to computers are bulky, slow to use, or unreliable. Dasher is a new interface incorporating language modelling and driven by continuous two-dimensional gestures, e.g. a mouse, touchscreen, or eye-tracker. Tests have shown that this device can be used to enter text at around 25 words per minute, compared with typical ten-finger keyboard typing of 40 - 60 words per minute.

    Although the interface is slower than a conventional keyboard, it is small and simple, and could be used on personal data assistants and by handicapped computer users.

    In UIST 2000: The 13th Annual ACM Symposium on User Interface Software and Technology

    Compressed postscript file. | PDF | Mastercopy of Abstract

    Computational Neuroscience

    Analysis of Linsker's simulations of Hebbian rules. (D. J. C. MacKay and K. D. Miller)
    Network 1, 257-298. [pdf file]
    Here is a text-only version of this paper (postscript). Here is a tar file of gzip-ed tiff or postscript files of the figures. More information.

    The Role of Constraints in Hebbian Learning. (Kenneth D. Miller and David J.C. MacKay)
    More information and pointers to a technical report. postscript (UCSF,USA) | postscript (Cambridge,UK).
    Hyperacuity and Coarse Coding - a short paper that I never got round to publishing
    postscript | PDF | | DJVU | | dvi | Abstract | | ps mirror, Canada | dvi mirror, Canada |
    timing.ps.gz. | <- UK | Canada -> | | PDF | | DJVU | timing.ps.gz.
    ` Associative memory using action potential timing '.
    Roland Muller, David MacKay and Andreas Herz
    advance.ps.gz. | PDF | | DJVU | abstract. | <- UK | Canada -> | advance.ps.gz. abstract.
    ` Time Warp Invariance by Recoding Time Delays into Time Delays '.
    Unpublished research note, March 1996.
    spike_inf.ps.gz. | PDF | | DJVU | abstract. | <- UK | Canada -> | spike_inf.ps.gz. abstract.
    ` Hierarchical Odor or Speech Recognition - Bayesian Inference embodied in Neuronal Spikes '.
    Unpublished research note, March 1996.
    Computation with action potentials: Deductions about the Hopfield-Brody Mouse (html document)
    David MacKay and research group.
    October 2000.

    See also the Neuroscience Data Analysis page.


    gene.ps.gz. | PDF | | DJVU | abstract. | <- UK | Canada -> | gene.ps.gz. abstract.
    ` Rate of Information Acquisition by a Species subjected to Natural Selection '.
    or, `Why sex is good for your bits'

    How does work work?

    Erg06.ps.gz. Erg06.pdf. abstract. | <- UK | Canada -> | Erg06.ps.gz. Erg06.pdf. abstract.
    ` Ergodic Pumping: a mechanism to drive biomolecular conformation changes '.
    by Robert S MacKay and David J C MacKay


    Do deuterated molecules smell different?
    by Luca Turin and David J.C. MacKay
    At this link you can find a reanalysis of the data of Haffenden et al (2003).


    A Short LaTeX tutorial

    Early work

    A Method of Increasing the Contextual Input to Adaptive Pattern Recognition Systems, Technical Report RIPR 1000/14/87, Research Initiative in Pattern Recognition, Royal Signals and Radar Establishment, Malvern, Worcester. WR14 3PS
    (My first paper on neural networks.)
    DJVU files: Readable, but poor quality (130K) high-resolution version (2M) zip file containing huge tif images
    connectionsM.ps.gz | PDF | |DJVU |
    My first paper on statistics and neural networks,
    Maximum Entropy Connections: Neural Networks,
    was published in Maximum Entropy and Bayesian Methods, Laramie, 1990, pages 237-244.
    I don't have the original postscript, but I have slapped together the above postscript document (which includes an unnecessary 7 page bibliography) to preserve the contents of the paper on line.

    Mathematical recreations

    A collection of puzzles for Darwin College magazine.
    And a few more puzzles including physics and probability ideas.
    rectangles.ps.gz. rectangles.pdf. html. abstract. | <- UK | Canada -> | rectangles.ps.gz. rectangles.pdf. html. abstract.
    `Simple Proofs of a Rectangle Tiling Theorem'. (2003)
    [Also, a one-page introduction to this problem is here: (pdf) (postscript)]
    corner.ps.gz. corner.pdf. abstract. | <- UK | Canada -> | corner.ps.gz. corner.pdf. abstract.
    ` Big Things that Squeeze Round Corners '.
    conway.ps.gz. conway.pdf. abstract. | <- UK | Canada -> | conway.ps.gz. conway.pdf. abstract.
    I edited a 15-page book chapter, `The Power of Mathematics', by John Conway. (Another site with a lecture by Conway)

    Other Fun stuff

    A note about the iTap / T9 text-to-numbers code
    Bao: An illustration of what research in theoretical physics is like

    Bayesian methods, quickies

    `The pope is (probably) not an alien' | <- UK | Canada -> | html
    With Sean Eddy
    euro.ps.gz. euro.pdf. | euro.DJVU. abstract. | <- UK | Canada -> | euro.ps.gz. euro.pdf. | euro.DJVU. abstract.
    Belgian euro coins: 140 heads in 250 tosses -- suspicious?

    Teaching methods

    exams.ps.gz. exams.pdf. abstract. | <- UK | Canada -> | exams.ps.gz. exams.pdf. abstract.
    ` Everyone Should Get an A '.
    Number theory (not a highly recommended paper!)
    sumsquares.ps.gz. sumsquares.pdf. | sumsquares.DJVU. abstract. | <- UK | Canada -> | sumsquares.ps.gz. sumsquares.pdf. | sumsquares.DJVU. abstract.
    ` Numbers that are Sums of Squares in Several Ways '.
    by David MacKay and Sanjoy Mahajan
    (Not a highly recommended paper as it was written with no knowledge of number theory)

    Dr Tatiana   Teaching Statistics: A Bag of Tricks, by Andrew Gelman and Deborah Nolan
    Data Analysis - A Bayesian Tutorial - Second Edition - Devinderjit Sivia and John Skilling
    Information Theory, Inference, and Learning Algorithms

    Book recommendations

    Dr. Tatiania's Sex Advice for All Creation, by Olivia Judson
    Teaching Statistics: A Bag of Tricks, by Andrew Gelman and Deborah Nolan
    Data Analysis - A Bayesian Tutorial - Second Edition by Devinderjit Sivia and John Skilling
    Gaussian Processes for Machine Learning by Carl Edward Rasmussen and Christopher K. I. Williams 0-262-18253-X
    published by MIT press. Order from |amazon.co.uk/.com |

    Books written or edited by David MacKay

    Power Edited by Alan Blackwell and David MacKay (ISBN: 0521823773 | ISBN-13:9780521823777) Price: £25.00 from |CUP UK/USA| |amazon.co.uk/.com/.ca/.co.jp| |Barnes and Noble USA |
    Power Contributors include Mary Archer and Tony Benn
    Information Theory, Inference, and Learning Algorithms by David J. C. MacKay
    |amazon.co.uk/.com/.ca/.co.jp| |Barnes & Noble

    Related books

    • Mitzenmacher book Probability and Computing: Randomized Algorithms and Probabilistic Analysis by Michael Mitzenmacher and Eli Upfal amazon.co.uk / .com
    • Jaynes book Probability Theory: The Logic of Science by E T Jaynes (G.Larry Bretthorst, Editor) amazon.co.uk / .com

    Problems using this archive? Some questions and answers.

    Note : If you prefer to get documents by ftp rather than by clicking in a web browser, then you can get to most of these files by anonymous ftp to wol.ra.phy.cam.ac.uk, cd pub/www/mackay.

    Conversely, if you want to get a document by http rather than by ftp, change the address from ftp://www.inference.org.uk/pub/www/mackay/blah to http://www.inference.org.uk/mackay/blah.

    when I logged into your web address as sent, I got a bunch of nondescript characters which at least my Nestscape 1.12 was not deciphering.
    OK - that means that your Netscape doesn't know about uncompressing files ending in .ps.gz and displaying them with ghostview. The thing to do in that situation is to select "Save As file" from the "File" menu, saving to name.ps.gz, then uncompress (gunzip) it by hand and then view the postscript in your favourite manner.
  • Do you have another question about Finding things on our web server? | or any other topic?
  • I mirror my archive in Canada thus

    Not yet available

    Automatic Relevance Determination (ARD) by MacKay and Neal.
    This paper is not available, and will perhaps never be completed. But please see pred.ps and network.ps above, and buy Radford Neal's book (Springer, 1996); all the information is there.

  • The Inference Group is supported by the Gatsby Foundation
    and by a partnership award from IBM Zurich Research Laboratory
    David J.C. MacKay
    Site last modified Wed Nov 12 17:38:44 GMT 2014