Gary F

eldm

an N

EP

PS

R 23 A

ug

ust 2002 1

Statistics o

f Sm

all Sig

nals

lA

partial d

iscussio

n o

f a pap

er “Un

ified A

pp

roach

to th

e Classical S

tatistical An

alysis of S

mall

Sig

nals,” w

hich

I wro

te with

Bo

b C

ou

sins. [P

hys.

Rev. D

57, 3873 (1988)]

Gary F

eldm

an N

EP

PS

R 23 A

ug

ust 2002 2

A S

imp

le Exam

ple (1)

lS

up

po

se you

are searchin

g fo

r a rare pro

cess and

have a

well-kn

ow

n exp

ected b

ackgro

un

d o

f 3 events, an

d yo

uo

bserve 0 even

ts. Wh

at 90% co

nfid

ence lim

it can yo

u set o

nth

e un

kno

wn

rate for th

is rare pro

cess?l

A classical (o

r frequ

entist) statistician

makes a statem

ent

abo

ut th

e pro

bab

ility of d

ata given

theo

ry. T

hat is, g

iven a

hyp

oth

esis for th

e value o

f an u

nkn

ow

n tru

e value µ, h

e or

she w

ill give yo

u th

e pro

bab

ility of o

btain

ing

a set of d

ata x,P

(x | µ).

lA

classical con

fiden

ce interval (Jerzy N

eyman

, 1937) is a

statemen

t of th

e form

: Th

e un

kno

wn

true valu

e of µ lies in

the reg

ion [µ

1 ,µ2 ]. If th

is statemen

t is mad

e at the 90%

con

fiden

ce level, then

it will b

e true 90%

of th

e time, an

dfalse 10%

of th

e time.

Gary F

eldm

an N

EP

PS

R 23 A

ug

ust 2002 3

A S

imp

le Exam

ple (2)

lP

oisso

n statistics P

(x = 0 | µ

= 2.3) =

0.1. Th

erefore, in

the

“stand

ard” classical ap

pro

ach, µ

< 2.3 at 90%

C.L

. Sin

ce µ =

s + b

, and

b =

3.0, s <

-0.7 at 90% C

.L.

lT

hu

s, we are led

to a statem

ent th

at we kn

ow

is a prio

ri false.

Gary F

eldm

an N

EP

PS

R 23 A

ug

ust 2002 4

Bayesian

Statistics

lA

Bayesian

takes the o

pp

osite

po

sition

from

a classicalstatistician

. He o

r she calcu

lates the p

rob

ability o

f theo

ryg

iven d

ata. Th

at is, given

a set of d

ata x, he o

r she w

illcalcu

late the p

rob

ability th

at the u

nkn

ow

n tru

e value is µ

,P

(µ | x).

lT

his ap

pears attractive b

ecause it is w

hat yo

u really w

ant to

kno

w. H

ow

ever, it com

es at a price:

Gary F

eldm

an N

EP

PS

R 23 A

ug

ust 2002 5

Bayes’s T

heo

rem

lP

(x | µ) an

d P

(µ | x

) are related

by B

ayes’s T

heo

rem, w

hich

inset th

eory is th

e statemen

t that an

elemen

t is in b

oth

A an

d B

isP

(A | B

) P(B

) = P

(B | A

) P(A

)w

hich

for p

rob

abilities b

ecom

esP

(µ | x) =

P(x | µ

) P(µ

)/P(x).

P(x) is ju

st a no

rmalizatio

n term

, bu

t Bayes

’s Th

eorem

transfo

rms P

(µ), th

e prio

r distrib

utio

n o

f “deg

ree of b

elief” inµ

, to P

(µ | x), th

e po

sterior d

istribu

tion

.

A “cred

ible in

terval” or “B

ayesian co

nfid

ence in

terval” isfo

rmed

by

P

(m|x

)dm

m1

m2

Ú=

90

%.

Gary F

eldm

an N

EP

PS

R 23 A

ug

ust 2002 6

An

Exam

ple (1)

lS

up

po

se you

have a larg

e nu

mb

er of m

arbles, w

hich

areeith

er wh

ite or b

lack, and

you

wish

info

rmatio

n o

n th

efractio

n th

at are wh

ite, µ. Y

ou

draw

a sing

le marb

le, and

it isw

hite. W

hat can

you

say at 90% co

nfid

ence?

Classical:

µ ≥ 0.1

Prio

rB

ayesian:

flatµ

≥ 0.316µ

µ ≥ 0.464

1/µµ

≥ 0.1(1-µ)

µ ≥ 0.196

1/(1-µ)

un

no

rmalizab

le

Gary F

eldm

an N

EP

PS

R 23 A

ug

ust 2002 7

An

Exam

ple (2)

lN

otice th

at mo

st of th

e Bayesian

prio

rs do

no

t cover, i.e.,

they are n

ot tru

e statemen

ts the stated

fraction

of th

e time

(90% in

this case). T

here is n

o req

uirem

ent th

at credib

lein

tervals cover. H

ow

ever, Bo

b C

ou

sins w

arns [A

m. J. P

hys.

63, 398 (1995)],

“…if a B

ayesian m

etho

d is kn

ow

n to

yield in

tervals with

frequ

entist co

verage ap

preciab

ly less than

the stated

C.L

. for

som

e po

ssible valu

e of th

e un

kno

wn

param

eters, then

itseem

s to h

ave no

chan

ce of g

ainin

g co

nsen

sus accep

tance

in p

article ph

ysics.”

Gary F

eldm

an N

EP

PS

R 23 A

ug

ust 2002 8

Th

e Ro

le of

Bayesian

Statistics (1)

lH

arrison

Pro

sper [P

hys. R

ev. D 37, 1153 (1988)] arg

ues fo

r a1/µ

prio

r based

on

a scaling

argu

men

t. I fou

nd

itu

nsatisfacto

ry for tw

o reaso

ns:

lIt fails fo

r x =

0. (Un

no

rmalizab

le)l

In g

eneral, it u

nd

ercovers.

lT

o q

uo

te Bo

b’s p

rose fro

m o

ur p

aper:

l“In

ou

r view, th

e attemp

t to fin

d a n

on

-info

rmative p

rior

with

in B

ayesian in

ference is m

isgu

ided

. Th

e real po

wer

of B

ayesian in

ference lies in

its ability to

inco

rpo

rate‘in

form

ative’ p

rior in

form

ation

, no

t ‘ign

oran

ce.’”

Gary F

eldm

an N

EP

PS

R 23 A

ug

ust 2002 9

Th

e Ro

le of

Bayesian

Statistics (2)

lP

rosp

er wro

te that h

e was u

sing

a Bayesian

app

roach

becau

se

l“…

we are m

erely ackno

wled

gin

g th

e fact that a co

heren

tso

lutio

n to

the sm

all-sign

al pro

blem

is mo

re easilyach

ieved w

ithin

a Bayesian

framew

ork th

an o

ne w

hich

uses th

e meth

od

s of ‘classical’ statistics.”

lT

hro

ug

h th

is talk, I ho

pe to

con

vince yo

u th

at this is n

olo

ng

er true.

Gary F

eldm

an N

EP

PS

R 23 A

ug

ust 2002 10

Co

nstru

ction

of

Co

nfid

ence In

tervals

lN

eyman

’s prescrip

tion

: Befo

re do

ing

an exp

erimen

t, for

each p

ossib

le value o

f theo

ry param

eters determ

ine a reg

ion

of d

ata that o

ccurs C

.L. o

f the tim

e, say 90%

. After d

oin

g th

eexp

erimen

t, find

all of

values o

f the th

eory p

ar-am

eters for w

hich

you

rd

ata is in th

eir 90%reg

ion

. Th

is is the

con

fiden

ce interval.

lN

otice th

at there is co

m-

plete freed

om

of ch

oice

of w

hich

90% to

cho

ose.

Th

is will b

e the key to

ou

rso

lutio

n.

Gary F

eldm

an N

EP

PS

R 23 A

ug

ust 2002 11

Exam

ples o

f Po

isson

Co

nfid

ence B

elts

• Fo

r ou

r examp

le: 90% C

.L. lim

its for P

oisso

n µ

with

backg

rou

nd

= 3

Up

per lim

itsC

entral lim

its

Gary F

eldm

an N

EP

PS

R 23 A

ug

ust 2002 12

Th

e So

lutio

n

lF

or b

oth

the u

pp

er limit an

d cen

tral limit, x =

0 exclud

es the

wh

ole p

lane. B

ut co

nsid

er the p

rob

lem fro

m th

e po

int o

fview

of th

e data. If o

ne m

easures n

o even

ts, then

clearly the

mo

st likely value o

f µ is zero

. Wh

y sho

uld

on

e rule o

ut th

em

ost likely scen

ario?

lT

herefo

re, we p

rop

osed

a new

ord

ering

prin

ciple b

ased o

nth

e ratio o

f a given

µ to

the m

ost likely µ

:

wh

ere µ* is th

e mo

st likely value o

f µ g

iven x.

†

R=

P(x

|m)

P(x

|m*)

Gary F

eldm

an N

EP

PS

R 23 A

ug

ust 2002 13

An

Exam

ple (1)

lE

xamp

le for µ

= 0.5 an

d b

= 3:

x0.121

0.1405.0

0.0178

x x

0.2590.149

4.00.039

7

x x

70.480

0.1613.0

0.0776

x x

40.753

0.1752.0

0.1325

x x

10.966

0.1951.0

0.1894

x x

20.963

0.2240.0

0.2163

x x

30.826

0.2240.0

0.1852

x x

50.708

0.1490.0

0.1061

60.607

0.0500.0

0.0300

C.L

.U

.L.

rank

RP

(x|µ

*)µ

*P

(x|µ

)x

Gary F

eldm

an N

EP

PS

R 23 A

ug

ust 2002 14

Un

ified P

oisso

n L

imits

l90%

C.L. unified

limits for P

oisson µ w

ith background = 3

• So

lutio

n to

ou

r orig

inal p

rob

lem: µ

< 1.08 at 90%

C.L

.

Gary F

eldm

an N

EP

PS

R 23 A

ug

ust 2002 15

Exam

ples o

f Gau

ssianC

on

fiden

ce Belts

• 90% C

.L. limits for G

aussian µ ≥ 0 vs. x (total – background) in s

Up

per L

imits

Cen

tral Lim

its

Gary F

eldm

an N

EP

PS

R 23 A

ug

ust 2002 16

Flip

-Flo

pp

ing

(1)

lH

ow

do

es a typical p

hysicist u

se these p

lots?

l“If th

e result x <

3s, I w

ill qu

ote an

up

per lim

it.”

l“If th

e result x >

3s, I w

ill qu

ote a cen

tral con

fiden

cein

terval.”

l“If th

e result x <

0, I will p

retend

I measu

red zero

.”

Gary F

eldm

an N

EP

PS

R 23 A

ug

ust 2002 17

Flip

-Flo

pp

ing

(2)

lT

his resu

lts in th

e follo

win

g:

lIn

the ran

ge 1.36 ≤ µ

≤ 4.28, there is o

nly 85%

coverag

e!l

Du

e to flip

-flop

pin

g (d

ecidin

g w

heth

er to u

se an u

pp

er limit

or a cen

tral con

fiden

ce regio

n b

ased o

n th

e data) th

ese aren

ot valid

con

fiden

ce intervals.

Gary F

eldm

an N

EP

PS

R 23 A

ug

ust 2002 18

Un

ified S

olu

tion

for

the G

aussian

Case (1)

lN

otes:

lT

his ap

pro

aches th

e central lim

its for x >

>1

lT

he u

pp

er limit fo

r x = 0 is 1.64, th

e two

-sided

rather th

anth

e on

e-sided

limit.

Gary F

eldm

an N

EP

PS

R 23 A

ug

ust 2002 19

Un

ified S

olu

tion

for

the G

aussian

Case (2)

lN

otes (co

ntin

ued

):

lF

rom

the d

efinin

g 1937 p

aper o

f Neym

an, th

is is the o

nly

valid co

nfid

ence b

elt, since th

ere are 4 requ

iremen

ts for a

valid b

elt:

(1) It mu

st cover.

(2) Fo

r every x, there m

ust b

e at least on

e µ.

(3) No

ho

les (on

ly valid fo

r sing

le µ).

(4) Every lim

it mu

st inclu

de its en

d p

oin

ts.

Gary F

eldm

an N

EP

PS

R 23 A

ug

ust 2002 20

Sen

sitivity

lT

he m

ain o

bjectio

n to

this w

ork h

as been

that an

experim

ent

that o

bserves few

er events th

an th

e expected

backg

rou

nd

may rep

ort a lo

wer u

pp

er limit th

an a (b

etter desig

ned

?)

experim

ent th

at has n

o b

ackgro

un

d.

lT

o ad

dress th

is pro

blem

and

to p

rovid

e add

ition

alin

form

ation

for th

e reader’s assessm

ent o

f the sig

nifican

ceo

f the resu

lts, we su

gg

ested th

at experim

ents th

at have

fewer co

un

ts than

expected

backg

rou

nd

also rep

ort th

eirsen

sitivity, w

hich

we d

efined

as the averag

e* u

pp

er limit th

atw

ou

ld b

e ob

tained

by an

ensem

ble o

f experim

ents w

ith th

eexp

ected b

ackgro

un

d an

d n

o tru

e sign

al. *Sh

ou

ld b

e med

ian

lW

e did

this in

the N

OM

AD

experim

ent an

d o

ther exp

erimen

tsh

ave been

do

ing

the sam

e thin

g.

Gary F

eldm

an N

EP

PS

R 23 A

ug

ust 2002 21

Visit to

Harvard

Statistician

s (1)

lT

ow

ards th

e end

of th

is wo

rk, I decid

ed to

try it ou

t on

som

ep

rofessio

nal statistician

s wh

om

I kno

w at H

arvard.

l T

hey to

ld m

e that th

is was th

e stand

ard m

etho

d o

f co

nstru

cting

a con

fiden

ce interval!

l I asked

them

if they co

uld

po

int to

a sing

le reference o

f an

yon

e usin

g th

is meth

od

befo

re.

l T

hey co

uld

no

t.

Gary F

eldm

an N

EP

PS

R 23 A

ug

ust 2002 22

Visit to

Harvard

Statistician

s (2)

lT

heir lo

gic:

lIn

statistical theo

ry there is a o

ne-to

-on

e corresp

on

den

ceb

etween

a hyp

oth

esis test and

a con

fiden

ce interval.

(Th

e con

fiden

ce interval is a h

ypo

thesis test fo

r eachvalu

e in th

e interval.)

lT

he N

eyman

-Pearso

n T

heo

rem states th

at the likelih

oo

dratio

gives th

e mo

st po

werfu

l hyp

oth

esis test.

lT

herefo

re, it mu

st be th

e stand

ard m

etho

d o

fco

nstru

cting

a con

fiden

ce interval.

Gary F

eldm

an N

EP

PS

R 23 A

ug

ust 2002 23

Ken

dall an

d S

tuart (1961)

lS

o I started

readin

g ab

ou

t hyp

oth

esis testing

.

lA

t the start o

f chap

ter 24 of K

end

all and

Stu

art’s Th

eA

dvan

ced T

heo

ry of S

tatistics (ch

apter 23 o

f Stu

art and

Ord

), I fou

nd

1 1/4 cryptic p

ages th

at pro

po

se this m

etho

dan

d its exten

sion

to erro

rs on

the b

ackgro

un

d.

lW

e were ab

le to in

clud

e a reference to

Ken

dall an

d S

tuart in

a no

te add

ed in

pro

of to

ou

r pap

er.

Gary F

eldm

an N

EP

PS

R 23 A

ug

ust 2002 24

Exten

sion

s

lT

his tech

niq

ue is m

ore g

eneral th

an th

e simp

le examp

lesd

escribed

here.

lT

he p

aper d

iscusses th

e app

lication

to n

eutrin

o o

scillation

s,in

wh

ich lim

its are set on

two

param

eters, sin22q an

d D

m2,

simu

ltaneo

usly.

lIt can

also b

e extend

ed to

cases in w

hich

the b

ackgro

un

ds

are no

t precisely kn

ow

n (b

ut w

e have n

ot yet p

ub

lished

this).

lIn

fact, I have yet to

find

a pro

blem

in th

e con

structio

n o

fclassical co

nfid

ence in

tervals and

regio

ns th

at is no

tso

lvable b

y the o

rderin

g p

rincip

le sug

gested

here.