Weaving together Astronomy+Statistics+Computer Science+Engineering+Intrumentation, far beyond the growing borders
The problem with data analysis is of course that it is a performing art. It is not something you easily write a paper on; rather, it is something you do. And so it is difficult to publish.
quoted from this conversation
My greatest concern was what to call it. I thought of calling it “information”, but the word was overly used, so I decided to call it “uncertainty”. When I discussed it with John von Neumann, he had a better idea. Von Neumann told me, “You should call it entropy, for two reasons. In the first place your uncertainty function has been used in statistical mechanics under that name, so it already has a name. In the second place, and more important, nobody knows what entropy really is, so in a debate you will always have the advantage.”
The whole story can be found from the page 8 of IMS Bulletin, Vol.37 Issue 7. (click for the pdf file)
A question to astronomers. What do you think WLOG is? It has nothing to do with our BLOG or SLOG. Statisticians, please do not say a word.
Talking about limits in Numerical Recipes in my PyIMSL post, I couldn’t resist checking materials, particularly updates in the new edition of Numerical Recipes by Press, et al. (2007).
PyIMSL is a collection of Python wrappers to the math and statistical algorithms in the IMSL C Numerical Library[1]. I recall the days of digging in IMSL (International Mathematics and Statistics Library) user manuals and learning Fortran and C to use this vast library (Splus was to slow at that time). Upon knowing that Python is very favored among astronomers (click here to see the slog posts about Python) and that limits exist in Numerical Recipes (I didn’t check the latest version published last year, though), probably IMSL is useful for mathematical and statistical analysis for astronomers.
To know more,
I didn’t realize this post was sitting for a month during which I almost neglected the slog. As if great books about probability and information theory for statisticians and engineers exist, I believe there are great statistical physics books for physicists. On the other hand, relatively less exist that introduce one subject to the other kind audience. In this regard, I thought the lecture note can be useful.
[arxiv:physics.data-an:0808.0012]
Lectures on Probability, Entropy, and Statistical Physics by Ariel Caticha
Abstract:
Like spherical cows, true blackbodies do not exist. Not because “black objects are dark, duh”, as I’ve heard many people mistakenly say — black here simply refers to the property of the object where no wavelength is preferentially absorbed or emitted, and all the energy input to it is converted into radiation. There are many famous astrophysical cases which are very good approximations to perfect blackbodies — the 2.73K microwave background radiation left over from the early Universe, for instance. Even the Sun is a good example. So it is often used to model the emission from various objects.
UChicago, my alma mater, is doing alright for itself in the spacecraft naming business.
First there was Edwin Hubble (S.B. 1910, Ph.D. 1917).
Then came Arthur Compton (the “MetLab”).
Followed by Subramanya Chandrasekhar (Morton D. Hull Distinguished Service Professor of Theoretical Astrophysics).
And now, Enrico Fermi.
I still remember my first class as a new grad student. As a cocky Physics graduate, I was quite sure I knew plenty of astronomy. Astro 301, class 1, and it took all of 20 minutes of talk about stellar magnitudes to put that notion to permanent rest. So, for the sake of our stats colleagues, here’s a brief primer on one of the basic building blocks of astronomy.
Differential Emission Measures (DEMs) are a summary of the temperature structure of the outer atmospheres (aka coronae) of stars, and are usually derived from a select subset of line fluxes. They are notoriously difficult to estimate. Very few algorithms even bother to calculate error envelopes on them. They are also subject to numerous systematic uncertainties which can play havoc with proper interpretation. But they are nevertheless extremely useful since they allow changes in coronal structures to be easily discerned, and observations with one instrument can be used to derive these DEMs and these can then be used to predict what is observable with some other instrument.
I grew up in an environment that glamourized mathematical equations. Equations adorned a text like jewelry, set there to dazzle, and often to outshine the text that they were to illuminate. Needless to say, anything I wrote was dense, opaque, and didn’t communicate what it set out to. It was not until I saw a Reference Frame essay by David Mermin on how to write equations (1989, Physics Today, 42, p9) that I realized that equations should be treated as part of the text. You should be able to read them. David Mermin set out 3 rules for writing out equations, which I’ve tried to follow diligently (if not always successfully) since then.
As mentioned before, background subtraction plays a big role in astrophysical analyses. For a variety of reasons, it is not a good idea to subtract out background counts from source counts, especially in the low-counts Poisson regime. What Bayesians recommend instead is to set up a model for the intensity of the source and the background and to infer these intensities given the data.
I have noticed that our statistician collaborators are often confused by our units. (Not a surprise; I, too, am constantly confused by our units.) One of the biggest culprits is the unit of energy, [keV],
Diab Jerius follows up on LOESS techniques with a very nice summary update and finds LOCFIT to be very useful, but there are still questions about how it deals with measurement errors and combining observations from different experiments: