THE NEW ELS TESTS - A REJOINDER
Dror Bar-Natan, Alec Gindis, Arieh Levitan, Brendan McKay
July 16, 1997
] Preliminary analysis and comments
] =================================
] on the report of new ELS tests
] ==============================
] by Dror Bar-Natan, Alec Gindis, Arieh Levitan, Brendan McKay
] ============================================================
]
] Eliyahu Rips
] June 19, 1997
SUMMARY.
We have studied in great depth the comments made by Prof Rips
concerning our experiment with the years of birth and death of
the famous rabbis. In this paper we give our reply. In addition,
our Report on the experiment will be revised in the near future.
Our conclusions are as follows.
1. We agree that errors were made in the collection of the data,
and for this we apologise. As promised in the protocols, the
calculations have been rerun. The results for the a-priori
parts of the experiment are still unambiguously negative.
2. The one smallish significance level that results for the first
list was never part of the a-priori experiment. Moreover, its
exposure requires two changes to the original protocol that
were each requested by Prof Rips some time after publication
of the protocols. We present evidence that Witztum and Rips
had prior knowledge of information which could reasonably have
lead them to suspect that the requested changes would help.
Despite this, the value is not small enough to be interesting
considering that it is only one value of very many.
3. Both dates and years had been examined by Prof Rips to some
degree by the time the appellations for the first list of rabbis
and the method of measuring closeness were first revealed. Given
that there is an ongoing controversy concerning those issues,
it is important to realise that the question of the years may
not be independent of it.
4. The arguments by which Prof Rips tries to focus the view on
the one value he likes are illogical and inconsistent. For
example, he claims that he is aiming for similarity with his
own experiment but also argues against some features which
we copied from him. Even more surprisingly, he argues against
one set of experiments that were requested by he himself.
5. Many of the criticisms made against our experiment apply just
as much or even more to experiments which are claimed to have
produced a positive result. However, we have no knowledge of
Prof Rips having made the same criticisms in those cases.
UNDERSTANDING THIS DOCUMENT.
The author of each line can be determined by the way it begins:
]> Dr Bar-Natan et al, in the original report
] Prof Rips
Dr Bar-Natan et al (if no special characters)
References to "Table 1", "List 1" and similar refer to the list
of rabbis from Table 1 in [WRR]. Similarly for "Table 2" etc..
]==== SUMMARY ====
]
]1. Mistaken data has been detected for both parts of the experiment
](on years and on books), in surprising quantities. Therefore all the
]results and conclusions given in this Report are invalid.
It is true that there are errors in the data, and we apologise for
that. The most common reason was that we took the Jewish year from
[EH] without checking if it was compatible with the secular year which
is also usually given there. There is a discrepancy surprisingly often,
and in a number of cases it is not clear from the rules what the correct
thing to do is. A secondary reason was the difficulty of communication
between Israel (where the data was collected) and Australia (where it
was entered onto the computer).
Two of us who did not do the initial collection have now reconstructed
the data from scratch then correlated it with the initial data.
We accept that it is desirable to find an expert to check the years.
We tried before, but did not succeed. (In our experience, the great
majority of professional people regard the entire subject as nonsense
and don't want to be involved.) We are trying again. Meanwhile, we
have repeated the calculations within the rules of the protocols.
]2. There is a fundamental flaw in their methodology: the authors of the
]report fail to say clearly IN ADVANCE, what will be considered a
]"successful" outcome (i.e. finding supporting evidence for ELS claims) and
]what will be considered a "failure".
While it might have been a good idea to define in advance an algorithm
for converting our many numbers into a single number, we don't see why
the absence of subjective {number} -> {true,false} mapping is a flaw.
No reader would be obliged to accept the value of that mapping anyway,
so what is the point?
In this connection it is worth noting that neither the published
StatSci paper, nor the Gans experiment, nor the three experiments
described by Witztum and Rips in "ELS, Part II" make any attempt
to define "success" and "failure" in advance. It seems that this
concern is a new one. The only precedent we are aware of is the
value of 1/1000 (after Bon-Ferroni) appearing in private
correspondence between Persi Diaconis and Yisrael Aumann, where the
meaning is clearly related to the acceptability for publication.
If that precedent is applied here, the experiment is an very clear
failure even for the very limited (and totally unjustified) subset
of our a-posteriori results selected by Prof Rips.
]3. This lack of a specified basis for judgement of the outcome widely
]opens the door to subjective interpretation of the results.
The door was already wide open, and that is the only way it can be.
]4. They claim the similarity of their experiments to that of [WRR].
]In reality only a small portion of their experiments are indeed similar.
This is a complete red-herring, and is introduced here only as a device
to dismiss various options that are in fact completely reasonable.
"Similar" is not a technical word. It is an informal description that
applies perfectly well to everything we did, and also to other SIMILAR
experiments like the Gans experiment.
]5. Having found a result 0.4% does not prevent them from concluding "The
]result was unambiguously negative. No indication of any extraordinary
]phenomenon was found." And this because they claim that they performed a
]"large number of computations". However, examination shows that only a
]small portion of the computations are relevant, so 0.4% cannot be so
]easily discounted.
This argument of "relevance" seems to be devoid of content. We do
not accept Prof Rips as judge and jury of what is relevant and what
is not. This is not his experiment, it is ours. What is relevant
is what is in the protocols, nothing more or less. The 0.4% did
not appear in the experiment defined by the protocols, so it is not
relevant. Prof Rips is attempting to move the target to underneath
the arrow after the arrow has been fired.
]Unfortunately, all the results are meaningless, because
]the computations were made with flawed data.
It has been rerun, results below.
]==== INTRODUCTION =====
]
]The summary of the "Report on new ELS tests" by Dror Bar-Natan, Alec
]Gindis, Arieh Levitan, Brendan McKay states:
]
]> We have performed two series of experiments similar to that
]> published by Witztum, Rips, and Rosenberg.
]> ...
]> In each case, the result was unambiguously negative.
]> No indication of any extraordinary phenomenon was found.
]
]We shall examine these declarations.
]
]The most serious deficiency in the proposed experiments is the authors'
]failure to state clearly in advance what will be considered a "success"
](i.e. finding supporting evidence for the ELS claims) and what will be
]considered a "failure" (i.e. not finding supporting evidence for the ELS
]claims). This failure opens the door to subjective interpretation of the
]outcome, as we shall see.
Not at all. Prof Rips' arguments about the outcome focus on a
computation that was not part of the experiment. How that is
interpretted has no bearing on whether the experiment succeeded
or failed.
]==== MISTAKEN DATA =====
In the case of Table 1, we will indicate the effect of the change
on Genesis. The change for the other books can be judged by
comparing the new results to the old ones.
]We have found that the experiments were performed with flawed data, which
]renders the reported results and conclusions invalid. We present here
]the outcome of our sample check of the data against the authors' own
]sources.
]
]1. In Years for Table 1, the data for #2, Rabbi Avraham, son of the
]Rambam, is indicated as follows:
]
]> #2 has 1+11 words
]> RBY)BRHM
]> BTTCX $NTTTCX B$NTTTCX DTTCX BDTTCX $NTDTTCX TTQMW BTTQMW
]> $NTTTQMW DTTQMW BDTTQMW
]
]However, his year of death is DTTQCX and not DTTCX; [EH], vol.1, page 313,
][ME], page 78. (According to the authors' data, Rabbi Avraham died 48
]years before he was born).
That is correct. It was a typo. However, since the only appellation
for this rabbi, RBY)BRHM, does not have a single ELS in all of Genesis,
the information available about name-year matches does not change at all.
In other words, this year is undesirable noise - undesirable because
it can change the result of the permutation test by a large amount
either up or down despite contributing no information. (This is one
of the many anomolous properties of the permutation test.)
]2. In Years for Table 1, the data for #4, Rabbi Eliyahu Bahur, is
]indicated as follows:
]
]> #4 has 3+4 words
]> RBY)LYHW HBXWR B(LHBXWR
]> $NT$X B$NT$X $NTH$X B$NTH$X
]
][EH], vol.3, page 553 indicates the Hebrew year of death as H$X, and the
]Gregorian year as 1649. This is self-contradictory, because H$X is
]1647/1648. Therefore, there is an obvious error in [EH]. The Report's
]protocol dictates using [ME] to resolve the error. [ME], page 46 indicates
]the year as being H$+.
In each case 16xx should read 15xx, but otherwise Prof Rips is correct.
In this case the change makes absolutely no difference without $NT as
the words are too short. The result with $NT becomes more negative.
Our reconstruction of the data uncovered three further problems
in Table 1. One was due to us missing an indication of uncertainty,
and two were due to a problem found many times in [EH], namely that
Jewish years have been translated into secular years incorrectly or
with the ambiguity resolved arbitrarily. Since the great majority
of the original sources are in Hebrew, one can expect Hebrew->secular
translation errors much more than the reverse.
A. In the case of Table 1, Rabbi #30, [EH] gives the birth year 1660
(5420/1). [ME] says "approximately 5420", so the uncertainty
cannot be resolved without an expert. We should have not used
this year within the protocols.
The year 5420 (HTK) is too short to produce any words of 5-8
letters without $NT. For forms with $NT, removing it makes
the result considerably weaker, as there is a perfect 1/125
score involved.
B. In the case of Table 1, Rabbi #31, [EH] gives the death year 1662
(5422/3). [ME] gives "Rosh Hodesh Adar 5423". 5423 is 1662/3,
which is consistent, though in 5423 there were two Adar's that
both fell in 1663. Until this matter is properly evaluated by
an expert, the most likely interpretation is that the "1662"
in [EH] is just a translation without care from 5423. (Such
a problem is seen in many places in the encyclopedia.)
Therefore, we should have taken the year 5423 (HTKG) instead
of what we used (5422=HTKB) which was incorrect anyway.
The effect of this change in Genesis for the date forms without
$NT is as follows:
Before: 0.343,0.368,0.496
After: 0.028,0.280,0.664
This change makes the result stronger. For the forms with
$NT, the result gets weaker.
C. In the case of Table 1 rabbi #33, [EH] gives the death year 1574
(5334/5). [Mar] gives 12 Kislev 5334, which is in 1573. As before,
the rules are not totally clear on what should be done, but until
expert advice is available we will consider the most likely
scenario, that a Jewish->secular year conversion error occurred.
This means we must replace 5335 (H$LH) by 5334 (H$LD).
The effect of this change in Genesis for the date forms without
$NT is as follows:
Before: 0.280,0.360,0.448,0.576
After: 0.456,0.744,0.912,0.920
This appears to make the result significantly weaker. For the
forms with $NT, the result also gets weaker.
We are puzzled as to why Prof Rips does not mention this example in
his letter, as he showed it to one of us at least a week beforehand.
]3. In Years for Table 2, the data for #1, Rabbi Avraham Av-Beit-Din of
]Narbonne, is indicated as follows:
]
]> #1 has 5+10 words
]> RBY)BRHM HR)BY HRB)BD HR)BD H)$KWL
]> TTQL+ BTTQL+ $NTTTQL+ DTTQL+ BDTTQL+ $NTTT( B$NTTT( BDTT(
]> $NTDTT( B$NTDTT(
]
][EH], vol.1, page 308 indicates that the birth year DTT( as uncertain.
]According to their protocols, rule R1, it should not be used.
Agreed.
]4. In Years for Table 2, the data for #3, Rabbi Avraham Ha-Malakh,
]is indicated as follows:
]
]> #3 has 2+11 words
]> RBY)BRHM HML)K
]> BTQLD $NTTQLD B$NTTQLD HTQLD BHTQLD $NTHTQLD $NTTQ) B$NTTQ)
]> BHTQ) $NTHTQ) B$NTHTQ)
]
][EH], vol.1, page 294 indicates the Hebrew year of death as TQLD, and the
]Gregorian year as 1776. This is self-contradictory, because TQLD is
]1773/1774. Therefore, there is an obvious error in [EH]. The Report's
]protocol dictates using [ME] to resolve the error. [ME], page 70 indicates
]the year as being TQLZ.
Agreed.
]5. In Years for Table 2, the data for #8, Rabbi David Ha-Nagid,
]is indicated as follows:
]
]> #8 has 2+0 words
]> RBYDWD DWDHNGYD
]
][EH], vol.12, page 77 and [ME], page 403 indicate the Gregorian year of
]death as 1300, and [ME] gives the Hebrew year HS, so according to the
]protocols the year of death to be taken is HS (and not missing).
Agreed.
]6. In Years for Table 2, the data for #9, Rabbi David Nieto, is indicated
]as follows:
]
]> #9 has 2+5 words
]> RBYDWD DWDNY+W $NTTPX B$NTTPX BHTPX $NTHTPX B$NTHTPX
]
][EH], vol.25, page 140 and [ME], page 405 indicate the Gregorian year of
]birth as 1654, and [ME] gives the Hebrew year TYD, so according to the
]protocols the year of birth to be taken is TYD (and not missing).
Agreed.
]7. In Years for Table 2, the data for #12, Rabbi Haim Capusi, is indicated
]as follows:
]
]> #12 has 4+0 words
]> RBYXYYM KPWSY B(LNS B(LHNS
]
]For Rabbi Haim Capusi there is no item in [EH]. [ME], page 519 indicates
]the year of death as $C). It is not clear from the protocols what action
]should be taken in such case. In any event, this situation should be
]mentioned in the section "Collection of the data" of their Report.
Later on an expert may tell us to insert $C, but for now we cannot
take it because the protocols do not tell us to consult [ME] in
this circumstance. Our action was correct, but Prof Rips is correct
in stating that we should have documented it.
]8. In Years for Table 2, the data for #14, Rabbi Yair Haim Bacharach,
]is indicated as follows:
]
]> #14 has 1+10 words
]> XWTY)YR
]> $NTTSG B$NTTSG BHTSG $NTHTSG B$NTHTSG $NT$CX B$NT$CX BH$CX
]> $NTH$CX B$NTH$CX
]
][EH], vol.8, page 721 and [ME], page 579 indicate the Hebrew year as TSB,
]so according to the protocols the year of death to be taken is TSB (and
]not missing).
It was not missing, it was wrong. We agree with HTSB.
]9. In Years for Table 2, the data for #15, Rabbi Yehudah Hasid,
]is indicated as follows:
]
]> #15 has 1+6 words
]> RBYYHWDH BTQL) $NTTQL) B$NTTQL) HTQL) BHTQL) $NTHTQL)
]
]I was not able to understand how TQL) is related to Rabbi Yehudah Hasid.
][EH], vol.19, page 212 and [ME], page 632 indicate the Gregorian year of
]death as 1700, and [ME] gives the Hebrew year TS), so according to the
]protocols the year of death is to be taken TS).
Agreed.
In addition, our regathering of the data uncovered the following:
D. For Table 2, Rabbi #21, [EH] gives only the secular year 1541.
[Mar] gives 1 Iyyar 5306, which is in 1546. The difference of five
years is too much for confident resolution, so we feel it is best
to remove this year until expert opinion is available.
E. For Table 2, Rabbi #24, [EH] has the year of birth as 1697, whereas
[Mar] gives 1698 and 5458. There is an inconsistency about the secular
year, but the Jewish year in [Mar] is consistent with the secular year
in [EH]. Thus, we take 5458 = HTNX for the year of birth.
A few other issues, which don't change the data, will be mentioned in
the second edition of the official report.
]10. In Books for Table 1, the data for #29, the Rambam, is indicated as
]follows:
]
]> #29 has 2+3 words
]> RBYM$H HRMBM SPRHMCWT YDXZQH M$NHTWRH
]
]However, in [EH], vol.24, page 543 and in [ME], page 1114, the name of
]his book is given as YDHXZQH and not YDXZQH.
Agreed. YDHXZQH has no ELS for any book of Torah. For Genesis, Numbers
and Deuteronomy, YDXZQH doesn't either. For Exodus, YDXZQH gave distances
0.345, 0.745 (uninteresting) and for Leviticus YDXZQH gave distances
0.143 and 0.036 (small, so the correction makes the result worse).
Prof Rips will need to demonstrate much more serious errors than
this one before we will withdraw the experiment on the books.
]==== THE PREMISES ====
]
]The protocols of the experiment published on April 17, 1997 give the
]following reason for the experiment:
]
]> Our aim is to further test the hypotheses made by Witztum, Rips,
]> and Rosenberg in [WRR].
]
]On April 20, I wrote to Professor McKay:
]
] "The interpretation of the outcome in case of success/failure. Please
] notice that we never claim that EVERY experiment should be successful.
] Indeed, in the preprint ELS II we report on an experiment that failed
] (out of 3 reported in this preprint).
]
] Therefore the expression "to further test the hypotheses" is not well
] defined as stated.. Our search is for a hidden text; we do not have
] knowledge in advance, what should be contained in it."
This lack of knowledge in advance is difficult to reconcile with
Prof Rips' certainty that (for example) using years with $NT is
inappropriate.
]On April 21 Professor McKay responded:
]
] "I would not claim that a negative result proves the absence of
] the phenomenon in general. I think it is logically impossible
] to disprove it. One can only find evidence for it, or fail to
] find evidence for it. Failing to find evidence for it is not
] the same as finding evidence against it. I think that is a
] fundamental asymmetry in statistical method....
]
] I mean "further test" only in the wide sense of continuing the
] general investigation. Both positive and negative results contribute
] to our understanding of it. I should have worded it more carefully"
]
]We see from the above discussion that both sides seem to be in agreement
]that the proposed experiment does not "further test the hypotheses made by
]Witztum, Rips and Rosenberg in [WRR]", in the narrow sense. This means
We don't know what point is being made.
]that IT IS AGREED THAT THE CLAIMS OF [WRR] ARE NOT DISPROVED IF THE
]EXPERIMENT FAILS TO PRODUCE SIGNIFICANT RESULTS.
The situation is just the same as in any where some experiments appear
to demonstrate a phenomenon and others do not. There are countless
examples in science. A primary way in which such issues are resolved
is by independent replication. If such attempts repeatedly fail,
most scientists will doubt the original experiment whether or not
they can identify anything specifically wrong with it. It is not
a question about proof; there is no such thing as proof in these
matters.
]==== THE DESIGN ====
]
]The "Report on new ELS tests" states:
]
]> We have performed two series of experiments similar to that
]> published by Witztum, Rips, and Rosenberg.
]
]The experiments conducted by the authors of the Report are claimed to be
]similar to those of [WRR].
]
]The authors of the Report have done many experiments. Upon closer
]examination, it turns out that only a small portion of their experiments
]are indeed similar to [WRR]:
All of this argument is futile. We do not accept a single word
of it, as we believe it is simply a rhetorical device. Our
experiments are OBVIOUSLY "similar" to the WRR experiments in
any reasonable sense of the word. They are more similar to them
than, for example, the experiment on the "nations". Furthermore,
as we shall demonstrate, this desire for "similarity" is applied
in a curiously selective way.
]1. Their protocols specify two methods, denoted (A) and (B). Method (B)
]was never used by [WRR] in any form, so experiments using (B) are not
]"similar" to [WRR]. In addition, we find (B) completely unacceptable on
]conceptual grounds (see below).
It is important to remember that this experiment was specifically
designed to be independent of Prof Rips. Therefore, while he is entitled
to his opinion as to what is "acceptable", we are not obliged to adopt
his views. Thus, while we acknowledge that (B) does not test for
precisely the same phenomenon, it does test for something related and
does so in a way that is less beset by mathematical difficulties.
We also note that we have seen many examples where (B) produces a
stronger significance level than (A), including the two smallest values
for this experiment. It would be bad science to restrict our tests to
those things which precisely match Prof Rips' intuition.
This is a good place to note that (B) is the most natural
interpretation of the experiment which WRR were asked to perform
in 1990 by Persi Diaconis (on behalf of the journal to which their
paper was first submitted). They failed to do so, but if they had
the experiment would not have passed the 1/1000 milepost set for them.
Whatever is the reason for it, the fact remains that [WRR] would quite
likely not have been published if the Prof Diaconis' instructions had
been followed.
]2. [WRR] is entitled "ELS's in the book of Genesis". Experiments performed
]on other texts, while possibly interesting, do not bear direct relevance
]to the study of ELS's in Genesis.
If one of our experiments on other Books yielded a strong positive
result, there is not the shadow of a doubt that Prof Rips would have
accepted it with great enthusiasm. Furthermore, claims of ELS
phenomena in other books of the Torah have been made a number of time
by Prof Rips and others in the past. Therefore, one can conclude that
the reason for dismissing the other books now is that they gave a
negative result. We note that no objection on this point was received
before the results were announced. On the contrary, note this exchange:
McKay to Rips, April 21 1997:
There is one question here: in Numbers there are two "inverted nuns".
The Koren electronic edition includes them as if they were ordinary
letters, but I was told they are generally considered as markers
(like special punctuation) rather than as normal text letters. Should
I include them or exclude them? Please make this choice for me.
Rips to McKay, May 1 (the only reply on this subject):
It seems to me that they are (kind of) markers.
Note two things. Firstly, that we asked Prof Rips to make the choice
in order to avoid being suspected of making a choice in our favour.
This is illustrative of the great care we took in the design of the
experiment, from which other experimenters can learn. Secondly, that
Prof Rips not only failed to complain about the proposal to test the
other books but even offered advice on how to do it properly!
In this context, we also note that we know of no tradition that
Genesis has a higher place than the other books of Torah.
]3. Another matter is the FORMAT of their Years experiment. Their protocol
]states:
]
]> Let yyy be the year within the millennium, and let myyy
]> be the same with the millennium indicated. The following
]> eight forms were approved by the linguist Professor
]> Michael Sokolov of Bar-Ilan University:
]>
]> F1: yyy
]> F2: Byyy ('in yyy')
]> F3: $NTyyy ('the year yyy')
]> F4: B$NTyyy ('in the year yyy')
]> F5-F8: The same as F1-F4 with myyy in place of yyy.
]
]We obviously agree with Professor Sokolov that it is legitimate to express
]a year by prefacing it with "the year" or "in the year". Nonetheless,
]nowhere does [WRR] append "the day" (i.e. YWM ) or "on the day" to their
]dates. Therefore, the only forms relevant to doing a test "similar" to
][WRR] are F1, F2, F5 and F6, i.e. the forms that do not have "the year" or
]"in the year" as a preface.
]
](It should be mentioned that on May 1 I wrote to Professor McKay:
]
] "I would like to suggest (in addition to the procedure R3) to consider
] the forms {F1,F2,F5,F6} (i.e. without $NT) separately and to consider
] the forms (F3,F4,F7,F8) (i.e. with $NT) separately.")
This is a very peculiar thing to mention as it totally demolishes Prof
Rip's own argument. Please read it carefully: it was Prof Rips himself
who asked for an experiment that uses the four $NT-forms without
the others, and at the same time he accepted the use of all eight.
It is also worth noting that we did not invent the idea of adding
a consistent prefix. We copied it from work of Gans and of Prof
Rips himself. More generally, our only nontrivial innovations appear
to have been of attempting to design the simplest possible protocol
that completely defines the data, and of publishing it widely before
doing the experiment. We only partially succeeded in the former aim,
but at least we tried.
Another difference between $NT and YWM is apparent. Without $NT,
many years are too short to fit the 5-8 letter bounds, and so are
not tested. (This is the same as happens in some of Prof Rips
experiments where he uses a prefix.) In the case of YWM, only
a small number of additional words are added; many get too long
and none at all are brought into the 5-8 range when they were
outside it before.
]We are left with the forms F1,F2, F5 and F6, these being similar to
]the forms used by [WRR].
On the contrary, we are left with the impression that Prof Rips is
arguing against everything that produced a result he does not like.
He is even arguing against his own suggestions!
]4. In principle, the idea of checking for the books corresponding to each
]Rabbi is interesting and worthy. However, such an experiment is not at all
]similar to [WRR].
It is just as similar as the "Gans" experiment, towards which Prof
Rips appears not to have made the same objection. We can also note
that in Jewish tradition the books written by these rabbis are
regarded as vastly more important than the precise dates of their
birth and death, or indeed the places of their birth and death.
]In addition, its design poses serious problems. These
]problems will be dealt with separately.
And answered separately.
]==== METHODOLOGY ====
]
]The authors of the "Report" conducted their experiments despite the fact
]that there was not (and still is not) a mutually agreed method of
]measurement. Their report states:
]
]> We have previously expressed criticism of Experiment A on various
]> mathematical grounds. However, since it was the method used in
]> [WRR] (other than minor changes), we included it in order to make
]> the present experiment independent of that debate.
]>
]> Experiment B has been severely criticised by E. Rips on the
]> grounds that it does not satisfactorily measure the phenomenon he
]> believes to occur in Genesis. Essentially, he is concerned that
]> the exceptionally small distances which occur occasionally may be
]> masked by averaging them with a larger number of ordinary
]> distances.
]
]1.Given this, it is hard to understand why the authors of the "Report"
]unilaterally decided to go ahead with their experiments. How will they
]interpret the outcome of their experiment? If method (A) succeeds then
]they'll say: "Well, we always said (A) is not a thing you should rely on."
]If (A) fails, then they'll say that even the authors of [WRR] will have
]to admit that the experiment failed.
The experiment was never intended to be cooperative. It was intended
to be independent. That is the correct scientific procedure to follow
in the case of a claim of an extraordinary phenomenon. When Pons and
Fleischman announced "cold fusion", there was no rush to perform joint
experiments. There was a rush to perform _independent_ experiments.
That is how science works.
(At this point Prof Rips mentions the question of the validity of
Experiment A. It deserves a more considered response than is
possible in this document, so we will make a separate reply later.)
]3. As stated earlier, (B) is a completely different experiment for which no
]"similarity" claims with our research can be claimed.
]
]Here is an explanation of the reasons for my criticism of (B), as given in
]my letter of May 1:
]
] "Experiment B is absolutely unacceptable for me, and let me explain why.
] This research is oriented towards checking the claim that there is a
] hidden text in Genesis which is based on ELS's. We do not know what
] should be contained in this hidden text, so we make guesses. The input
] of each guess is a pair of words (w,w'). For each such pair of words we
] compute some functional c(w,w'). The functional c(w,w') was designed as
] to reflect some intuitive idea ("close meeting between ELS's", where
] "close" is understood for some cylindrical metrics on the text). To have
] a "small" value of c(w,w') means "success" (a close meeting between
] ELS's detected), otherwise "failure." Now we have to count the number of
] successes per number of guesses in order to decide whether we encounter a
] "remarkable" deviation from randomness. Both statistics P1 and P2 do it...
]
] Now what does the procedure of the experiment B? It AVERAGES the values
] of c(w,w'), in other words it punishes the successes for the failures.
] (For example, I would be very happy to have SYSTEMATICALLY a 1/100 per
] every 10 guesses; even such an impressive result would be AVERAGED OUT!)"
]
]4. I was willing to negotiate with Professor McKay on finding a mutually
]agreed method of measurement. Below are some relevant excerpts of our
]correspondence.
Please note that this correspondence occurred well after the protocols
of the experiment had been announced. In order to maintain the
independent and a-priori nature of our experiment we were obligated
to not accept changes. We very generously accepted one request from
Prof Rips, though in retrospect we should not have.
]On April 20, I wrote to Professor McKay:
]
] "...2. In my opinion, the experiment cannot be performed before there is
] a measure to which both sides agree. As I already wrote, the experiment
] (B) simply does not measure the thing we are looking for; the experiment
] (A) is not satisfactory for you.
]
] I have a number of suggestions for alternative measure; probably you
] have you own suggestions. I think we should return to this question
] (immediately) after Pesach."
]
]And in another letter on May 14:
]
] "I suggested other ways to perform the randomizations, and I am prepared
] to negotiate on this matter without any delay. However, I am very
] strongly against performing this experiment in an absence of an
] agreement."
]
]The final position of Professor McKay is (in a letter from May 17):
]
] "Since it is now a considerable time since the protocols of
] the experiments were published, and most of the data has been
] collected, we do not feel it would be correct procedure to
] modify the "official" experiment now."
Exactly. To do otherwise would have been to destroy the experiment.
]==== RELIABILITY OF DATA ====
]
]The reliability of data is clearly a very important issue. The outcome of
]an experiment depends critically on correct data. It is well known
]that the biographical data about the birth of famous people is usually
]much less reliable that the data about their death. I suggested to Professor
]McKay to consult experts on the matters of biographical data, especially,
]the years of birth. However the authors of the "Report" chose to act
]otherwise.
This is not true. We have always accepted the desirability of having
the data checked, and made some unsuccessful attempts to find someone
to do it. We are still trying. However, we were not willing to delay
distribution of the results according to the letter of the protocols
(which were designed to not need an expert).
]Therefore, in analyzing the outcome of the experiment on years, we will only
]regard the years of death. When the years of birth have been checked by a
]proper authority, they can be taken into consideration.
They shall be checked, and also the death years. However, it should
be noted that the reason for including birth details was that [WRR]
also included them. Furthermore, we have no knowledge of any complaint
from Prof Rips about the accuracy of the data in other experiments for
which an expert could have been used (such as the Gans experiment,
which is much more problematic in this respect than ours).
]I bring here my discussion with Professor McKay.
]
]In my letter on April 20 to Professor McKay I wrote:
]
] "The years of birth and death should be established by an expert; the
] task of doing it should be assigned to an expert through an agreed
] procedure...
]
] It is especially important because it is well known that many of the
] birth years are only approximate."
]
]The response of Professor McKay (April 21) is noteworthy:
]
] "I wanted to do some independent tests as the next part of my personal
] investigation. We deliberately picked subjects that could be
] investigated without a (new) expert, that had no clear means available
] for manipulation of spelling (etc), and that could be checked in a short
] time by anyone.
] It is not true that we are not consulting any expert: we believe the
] editors of Hebraica would have chosen the best available expertise to
] write each of the entries. It would be different if the words we need
] were required to come from many different sources and be selected
] according to criteria which cannot be defined precisely. We
] deliberately avoided such examples in this case. Our experiment does
] not rely on a complicated set of arbitrary choices, but on rules
] intended to be very simple while still be precise enough that two people
] collecting the data should obtain precisely the same lists.
This was the hope, but in the event there were circumstances that
were not anticipated. At least two situations exist where the rules
are insufficiently clear. This illustrates the great difficulty
of defining the data exactly without looking at it first.
] Unfortunately, I don't think there are very many such sets of data
] available.
]
] Obviously not all experiments can be designed like this.
] Certainly I am much in favour of establishing a committee,
] especially for more complicated experiments."
]
]My answer on May 1:
]
] Let me explain the problem as I see it. I certainly believe that the
] editors of Hebraica employed very good experts. But please notice that
] prior to this experiment NOTHING CRITICALLY DEPENDED on whether the
] years are correct ones, or approximations, or plausible conjectures (or
] even wild guesses). The items of the encyclopedia do not indicate which
] of these cases holds (because nobody apparently needs to know this --
] except for the present experiment).
]
] So the expert should be asked not just to provide the data, but to
] provide clear documentary proof of its correctness, or to state (in a
] substantiated way) that there is no such proof available. In any case,
] the status of the data should be indicated.
]
]In a letter from May 15 forwarded to Professor McKay on May 18 I wrote:
]
] "I agree with using the years of death as indicated in the Encyclopedia
] (except for obvious mistakes).
] For me, it is fine to run the experiment only on years of death.
]
] But if the years of birth are to be taken, this requires CONSULTATION
] WITH EXPERT OR EXPERTS. A special Committee should formulate the
] question to the expert/s."
]
]The authors of the "Report" chose to proceed without consulting experts,
]however they do publish the results obtained using the death years
]without the birth years..
Only as an auxiliary computation that does not form part of the
a-priori experiment.
]REMARK. Professor McKay expresses the intention to have a simple and
]straightforward set of data for his experiment. In fact, in the seemingly
]straightforward Years experiment, certain subtleties are involved. The
]Books experiment is neither simple nor straightforward at all. The names
]of books involves complexities which are not addressed in the authors'
]protocols. This should be discussed thoroughly as a separate issue.
It is true that we did not anticipate many of the difficulties that
arose. From this you can infer that we did not look at the data
before writing the protocols.
]==== "...THE LARGE NUMBER OF COMPUTATIONS" ====
]
]As I mentioned before, the authors do not specify IN ADVANCE what will be
]the BASIS FOR JUDGEMENT OF THE OUTCOME. Let us now follow their text.
]
](Notations of rows and columns in {these brackets} are ours.)
]
]> We will only present the results for the Book of Genesis.
]> The results for years of death in the other books are even
]> less interesting.
]> {c1} {c2} {c3} {c4} {c5} {c6} {c7} {c8}
]> Books All year Without With
]> forms $NT $NT
]> Table 1
]> {r1} P1 946597/786917 343991/417444 268265/309097 518287/567437
]> {r2} 107933/188022 040073/059591 548529/622865
]> {r3} P2 897962/657465 288110/261657 079486/156629 683097/511968
]> {r4} 025506/032244 004150/020425 488451/310752
]> {r5} B 804395/558328 063461/046783 036526/026419 232923/127553
]> {r6} 010521/013124 010639/017908 400176/307306
Here are the same values (except for books) with the corrected data.
The other values will be in the revised report. The numbers with
"+" are computations within the protocols and those with "-" are
computations demanded much later by Prof Rips and presented by
us merely as a courtesy. Those with neither were added later by
us entirely to imitate [WRR].
All year Without With
forms $NT $NT
P1 +613506/648384 +095687/109361 +950585/951049
-147839/237286 -004139/006701 -925217/950297
P2 +791606/735377 +049616/091837 +995425/980593
-196423/203780 -001123/006585 -974054/921431
B +223271/277218 +052327/030738 +496591/495248
-038011/085133 -013334/013724 -636103/729298
We see that of the nine a-priori results, seven got worse and only
two got better. The best of them got worse.
As to the small value in the center, the reader will be amazed to
see that it has changed from 0.004 to 0.001. The reason will be
even more amazing. It is not because of change B, which has only
a tiny effect. It is because of change 1, even though that change
is to a year of a rabbi for which there are no names with ELS's!
This change, which adds NO INFORMATION WHATEVER, reduces the result
by a factor of 5. Here again, for the umpteenth time, we see how
misleading the permutation test is.
]> Table 2
]> {r7} P1 227835/100008 417339/414092 834959/778628 105783/149229
]> {r8} 677190/786334 819212/799814 346106/557166
]> {r9} P2 268628/194173 201746/274949 720029/722988 041576/075552
]> {r10} 434804/679920 626708/715350 268936/495786
]> {r11} B 713015/220562 244322/105014 442305/404187 033625/019805
]> {r12} 379753/319462 375565/410462 302149/145498
]>
]> Tables 1 and 2 together
]> {r13} P1 823043/463562 337511/362989 601181/536583 208691/289420
]> {r14} 301703/449190 322849/315978 402008/621510
]> {r15} P2 753366/437302 179017/190047 321917/389296 214528/186159
]> {r16} 100186/189912 093055/191680 344982/382331
]> {r17} B 848098/383244 079685/019538 108264/060895 038674/011613
]> {r18} 050206/026570 051700/049921 291890/116418
]>
]> Here again we see no reason to claim other than chance behaviour.
]> Removing of years of birth sometimes improves the result and
]> sometimes worsens it. Similarly for removing the names starting
]> with "Rabbi".
]> The smallest value 0.4% is not very small considering the large
]> number of computations we have performed.
]
]I think that this statement is the most remarkable in their entire Report.
]The value 0.4% is discounted "considering the large number of computations
]performed".
Our remark was completely correctly. However, since this value did not
form part of the a-priori experiment, there is no real need for any
argument to dismiss it.
]However, as we have already seen, only a small portion of these
]computations are relevant.
We have not seen any such thing. We reject this assertion totally.
]1. Recall that the authors say:
]
]> We have performed two series of experiments similar to that
]> published by Witztum, Rips, and Rosenberg.
And so we did.
]If similarity is claimed, we have to consider only:
]
]1) Experiments with method (A),i.e. P1 and P2 and not with method (B).
]2) Experiments on Genesis.
]3) Experiments with years (Book experiment is a separate issue).
]4) The year formats of F1, F2, F5, F6 (i.e. without $NAT), just like we
]did note consider the format with YWM.
As well as recalling that Prof Rips himself suggested using the forms
with $NT alone, and agreed also to using all eight forms, it is worth
repeating that we did not invent the idea of adding a consistent
prefix to a list of words. The Gans experiment does it, and several
experiments of WRR rely entirely on it.
We also wonder why Prof Rips omitted from his list the following,
as they were included by us precisely because [WRR] did.
5) Experiments with and without the names beginning RBY.
6) Experiments that use both births and deaths.
]What remains from the above table is:
What this means is "the part that Prof Rips likes the result of".
Note that if Prof Rips consistently argued for "similarity" with
[WRR], he would have to take the birth details just as [WRR] did.
]> {c5} {c6}
]> Without
]> $NT
]> Table 1
]> {r1} P1 268265/309097
]> {r2} 040073/059591
]> {r3} P2 079486/156629
]> {r4} 004150/020425
]>
]> Table 2
]> {r7} P1 834959/778628
]> {r8} 819212/799814
]> {r9} P2 720029/722988
]> {r10} 626708/715350
]>
]> Tables 1 and 2 together
]> {r13} P1 601181/536583
]> {r14} 322849/315978
]> {r15} P2 321917/389296
]> {r16} 093055/191680
]
]2. Now, as we stressed several times, only years of death constitute
]reliable data. Recall my letter from May 15:
]
] "I agree with using the years of death as indicated in the Encyclopedia
] (except for obvious mistakes).
] For me, it is fine to run the experiment only on years of death.
]
] But if the years of birth are to be taken, this requires CONSULTATION
] WITH EXPERT OR EXPERTS. A special Committee should formulate the
] question to the expert/s."
It is clear that further investigation of the death years is also
justified. We did not see much evidence that the birth years which
are available are less reliable than the death years, though
considerably fewer are available.
]Taking only the years of death, we obtain the following table:
There is nothing in the protocols about taking only the years
of death. Incidentally, we would like to ask Prof Rips who the
expert was who prepared the dates of birth and death for [WRR].
]> {c5} {c6}
]> Without
]> $NT
]> Table 1
]> {r2} P1 040073/059591
]> {r4} P2 004150/020425
]>
]> Table 2
]> {r8} P1 819212/799814
]> {r10}P2 626708/715350
]>
]> Tables 1 and 2 together
]> {r14} P1 322849/315978
]> {r16} P2 093055/191680
]>
]
]REMARK: The authors of the report say:
]
]> 3. On May 15, E. Rips requested that we use only the years
]> of death, not the years of birth and death together.
]> We did not agree to that change, but in any case we will
]> present the results of that experiment here also.
]I do not know the reason for their disagreement, and I also do not know
]why their disagreement makes the experiment with the years of death
]"non a priori".
This should be completely obvious. In the 12 years since the
investigation of dates of birth/death began, it is inconceivable
that no checking of years (at least informally) was done. In fact
the opposite can be proved: the book of Witztum contains four
examples of years of death of rabbis in the first list. One of
the examples (Rambam) actually gives the second smallest distance
in the sub-experiment producing the best result. Thus, we know
as a fact that Witztum and Rips had at their disposal information
which could reasonably lead them to suspect that removing the forms
with $NT and removing the birth years would boost the chance of a
small answer, which indeed it does. Further recall that both requests
were received considerably after the protocols were published.
We can also say that both dates and years were both already in evidence
the very first time that the StatSci method of measuring closeness
appeared, and the first time the appellations for Table 1 appeared
(in both cases, in the 1986 preprint of the StatSci research).
Therefore, especially for Table 1, it cannot be taken as a fact
that the development of the metric and the appellation list took
no account of years.
It is instructive to look further at the proposal by Prof Rips
to break the year forms into two groups. Here is a little more
of his reasoning (mail of 1 May 1997, Rips to McKay):
I would like to suggest (in addition to the procedure R3) to
consider the forms {F1,F2,F5,F6} (i.e. without $NT) separately
and to consider the forms (F3,F4,F7,F8) (i.e. with $NT) separately.
To a certain extent this suggestion if influenced by the finding
by Professor Bar-Natan that BdayBmonth does not contribute
significantly. For me, it has the meaning that the "support" of
the phenomenon can sit on a subset, so it makes sense also to
"divide". (My previous view was that one should only "accumulate").
Note that the argument he gave was for dividing the forms, not
for dividing them in a particular way. There was no sign of the
specific arguement against $NT that is presented now. Let us
consider how other divisions could have been argued:
1. "Hebrew years are written without the millenium much more
commonly than with. Therefore, try the four forms with the
millenium and the four forms without it".
2. "For Table 1, the strength of the result with the dates came
more from date forms with the preposition bet (in), whereas
in Table 2 the form without bet was the most important.
Therefore, try the four forms with bet and the four forms
without it".
Let us see the effect of these divisions at the point in question
(Genesis, Table 1, years of death only, P2 rank out of a million):
Undivided: 196423
Without $NT: 1123 <--- suggestions
With $NT: 974054 <--- of Rips
Without millenium: 662900
With millenium: 45397
Without bet: 564808
With bet: 95232
We will leave the interpretation of these numbers to the reader.
]3. The authors perform additional computations not included in their
]protocol:
Including using the years of death alone. Why is it not listed?
]> 1. The boundary between the two lists is an artifact of the
]> history of [WRR]. Therefore it makes sense to consider
]> the effect of using both lists together.
]>
]> 2. The strongest result in [WRR] was obtained after the
]> removal of appellations starting with the word "Rabbi".
]> Therefore it makes sense to try that here also.
]
]While these suggestions are reasonable, they cannot be counted in their
]"large number of computations", otherwise a real statistical Pandora's box
]is opened.
This argument is incredible. These two additional computations
are at least as reasonable as the exclusion of the birth years.
In fact, both have solid precedence in the work of Gans and/or
Rips, whereas exclusion of birth details is something entirely new.
It is fundamentally important to realise that none of these three
variations (exclusion of birth details, exclusion of "RBY ...", both
lists together) are part of the a-priori experiment. Given that these
additional computations were all absent from the protocols, they are
all merely anecdotal and cannot be used to draw firm statistical
conclusions.
One wonders why Prof Rips has suddenly forgotten his desire for
"similarity". The experiment with the names "Rabbi X" deleted
was ONLY performed because it is used in the WRR experiment.
This example proves that the concern about "similarity" was just
a device, to be overlooked when convenient. (We also note in
passing that in the experiment of Gans, ONLY the appellations
other than "Rabbi X" were used.)
]So we are left with:
We absolutely reject this assertion.
]> {c5}
]> Without
]> $NT
]> Table 1
]> {r2} P1 040073
]> {r4} P2 004150
]>
]> Table 2
]> {r7} P1 819212
]> {r9} P2 626708
]
]Of course, the reader should be reminded that this discussion is purely
]theoretical. The experiment was run with incorrect data.
]
]I must say that if these were actual numbers, I would not be able to say
]what their meaning is, because unilateral decisions were made. However, I
]do not accept the blame for this situation, because I constantly
]insisted that an agreement should be reached, and only then should the
]experiment be performed. It was the decision of the authors of the Report
]to run ahead with their experiments despite the lack of an agreed way to
]judge the outcome.
What principle in science says that experimenters must first seek
the approval of the originator of the hypothesis being tested?
]==== SUBJECTIVE INTERPRETATION =====
]
]Here are some examples, and little comment is needed.
]
]> The smallest value 0.4% is not very small considering the large
]> number of computations we have performed. In fact, a close look
]> shows just how weak it is. There are 72 defined values c(w,w')
]> for which w and w' belong to the same rabbi. If they were
]> independent random variables with uniform distribution on (0,1),
]> the expected values of the smallest two would be 0.0137 and 0.0274.
]> The actual smallest two values are larger: 0.0172 and 0.0320.
This is still true with the corrected data.
]> Hence, this example certainly does not support the hypothesis that
]> very small distances are unusually common.
]
]In other words, NOT TO HAVE "OUTLIERS" is bad. But at other occasions we
]were told that TO HAVE "OUTLIERS" is bad.
Prof Rips seems to be quoting from a document long ago rewritten.
Nevertheless, we can address the point. The value of P2 is very
sensitive to the few smallest values. That is a fact, and it is
just as true for mediocre results as for stronger results. (Changing
the two smallest distances to "undefined" makes the result jump up
by a factor of 7.) However, the main reason we included that
paragraph was to show that this sub-experiment in no way supports
the hypothesis about the 'codes' that Prof Rips employs to dismiss
Experiment B.
]Or, what does the following argument tell?
]
]> It is hard to reconcile these facts with the score of 0.4%,
]> but it seems to be due to the small number of smallish distances
]> (8 at most 0.05) being unevenly distributed: there are 3 for rabbi
]> #22 and 2 for rabbi #5. Removing rabbi #22 alone is enough to raise
]> the P2-rank by a factor of more than 8.
With the corrections, it is now worth a factor of 13. In other words,
more than a whole order of magnitude of this smallish value rests on the
shoulders of a single rabbi. Would you buy shares in a drug which in
clinical trials only cured one patient?
One should also note that if this value indicates a genuine phenomenon
one must explain why the exact same case for Table 2 gives such a bad
result (worse than average). Recall that for the StatSci experiment
Table 2 gave a stronger result than Table 1.
]However, let me remind once again, that this experiment IS YET TO BE DONE.
]
]==== REFERENCES ====
]
][WRR] D. Witztum, E. Rips, and Y. Rosenberg, Equidistant
] Letter Sequences in the Book of Genesis, Statistical
] Science Vol 9 (1994) 429-438.
][ME] M. Margaliot (ed.), Encyclopedia of Great Men of Israel.
][EH] Encyclopedia Hebraica.
]
]
]The text of the "Report on new ELS tests of Torah" by Dror Bar-Natan, Alec
]Gindis, Arieh Levitan, Brendan McKay is posted at:
]
]http://www.math.gatech.edu/~jkatz/religions/Numerics/report.html
A revised report will be published soon. Meanwhile, here are
the results of the a-priori experiment with the new data.
Each number is the rank out of one million.
All
Names of year Without With
Books forms $NT $NT
Book = Genesis
Table = 1
P1 : 946597 613506 095687 950585
P2 : 897962 791606 049616 995425
B : 804395 223271 052327 496591
Table = 2
P1 : 227835 278681 680601 105543
P2 : 268628 244274 755110 053046
B : 713015 221061 448561 008056
Book = Exodus
Table = 1
P1 : 497040 777723 458105 861311
P2 : 248726 674284 512277 705649
B : 184635 530838 702930 014562
Table = 2
P1 : 732340 963584 823230 944385
P2 : 666454 771410 824091 544362
B : 553204 981849 707582 803153
Book = Leviticus
Table = 1
P1 : 539882 885629 547929 913279
P2 : 885291 962741 612366 969463
B : 670833 903563 970253 705664
Table = 2
P1 : 191194 505995 581853 402682
P2 : 073466 423324 446715 439863
B : 494075 867109 777571 646880
Book = Numbers
Table = 1
P1 : 761420 106021 113520 281993
P2 : 412348 149744 125674 376755
B : 569661 733519 271861 519378
Table = 2
P1 : 467604 295100 658790 161007
P2 : 305941 045448 224390 064491
B : 422428 316707 686831 525477
Book = Deuteronomy
Table = 1
P1 : 612340 319108 474433 261805
P2 : 770759 247574 368112 262818
B : 627681 577148 753026 643072
Table = 2
P1 : 437418 486239 481636 488050
P2 : 334035 378378 412209 383925
B : 192979 222604 656561 116402
It is seen that the smallest value occurs for Experiment B
and is quite normal for the smallest out of 120 values.
The results from the complete set of additional computations
will be given in the revised official report. In that case also,
the smallest value is consistent with the number of values.
Epilogue...... During work on another project, we discovered
a new error in Mr Rosenberg's program ELS2.C --- the most recent
of many errors we have found in it. We had been under the
impression that we had duplicated his errors exactly, but one
escaped notice. Consequently, we cannot say for sure that our
results are the same as ELS2.C would obtain, though probably
they would be similar. We do not believe it is worth rerunning
our experiment just to copy another of Mr Rosenberg's errors.
[Details: the variable Result000_tot should be declared float.]
------------
Overview on numerical features in different scriptures
Answering Islam Home Page