Comparison of perfect table cryptanalytic tradeoff algorithms

Abstract

The performances of three major time memory tradeoff algorithms were compared in a recent paper. The algorithms considered there were the classical Hellman tradeoff and the non-perfect table versions of the distinguished point method and the rainbow table method. This paper adds the perfect table versions of the distinguished point method and the rainbow table method to the list, so that all the major tradeoff algorithms may now be compared against each other. Even though there are existing claims as to the superiority of one tradeoff algorithm over another algorithm, the algorithm performance comparisons provided by the current work and the recent paper mentioned above are of higher practical value. We provide comparisons of algorithms at parameters that achieve a common success rate of inversion and which take both the cost of pre-computation and the efficiency of the online phase into account. The comparisons are based on the average case execution behaviors rather than the worst case situations, and non-negligible details such as the effects of false alarms and various storage optimization techniques are no longer ignored. A large portion of this paper is allocated to analyzing the execution behavior of the perfect table distinguished point method. In particular, we obtain a closed-form formula for the average length of chains associated with a perfect distinguished point table.

Appendices

Appendix 1: Effects of chain length bounds and parallelization on the DP tradeoff

Let us briefly comment on the two techniques for the DP tradeoff that were not considered during our complexity analyses. Specifically, these were the use of upper and lower bounds on the chain lengths and the parallel processing of tables during the online phase.

The use of chain length bounds will increase the number of pre-computation chains that are discarded, so that the amount of pre-computation must be increased in order to maintain the success rate. The effect of chain length bounds on the online efficiency is unclear, but making the pre-computation chain lengths shorter has the tendency to reduce chain merges and false alarms, so this could have a positive effect. However, a quick review of Fig. 4 shows that no small enhancement in the online efficiency is likely to make the perfect DP tradeoff more preferable over the perfect rainbow tradeoff, even if no penalty on the pre-computation cost was involved. Hence, there seems to be little reason to consider the perfect DP tradeoff variant that utilizes chain length bounds, except possibly at low success rates.

The work [12, 24] suggested that all the pre-computation tables be processed in parallel, rather than sequentially, during the online phase. Parallel processing causes the shorter online chains to be treated before the longer ones, and since the online phase is likely to terminate with the correct answer before any of the pre-computation tables are fully processed, this leads to a larger portion of the online computation being spent on processing the shorter chains. Since shorter chains are less likely to induce false alarms, this has a positive effect of reducing the cost of dealing with alarms. However, the recent analysis on the non-perfect parallel DP tradeoff [16] indicates that one cannot hope to see any drastic improvement of the perfect DP tradeoff performance through parallelization. For now, there seems to be no reason to expect the parallel perfect DP tradeoff to perform better than the perfect rainbow tradeoff, especially under the high success rate requirements which are of interest.

Appendix 2: Appropriate chain length upper bound

Let us analyze how often one can expect to see over-length chains when an upper bound on the chain length is set. This information will give us an indication of how large a bound qualifies as a sufficiently large chain length bound.

We first wish to write down the probability for a chain generated from a given starting point to fall into an infinite loop without reaching a DP within its first $\hat{t}$ iterations. The chain can fall into a loop if the first iteration lands on the given starting point itself. The probability for such an event can be stated to be $\frac{1}{\mathsf{N }}$, when the chain is treated as a random walk. If the first iteration lands on a non-DP that is different from the starting point, which happens with probability $1-\frac{1}{t}-\frac{1}{\mathsf{N }}$, then the chain can fall into a loop if the second iteration lands on either of the first two points, which happens with probability $\frac{2}{\mathsf{N }}$. Continuing, the probability in question may be written as

$$\begin{aligned} \begin{aligned} \frac{1}{\mathsf{N }}&+ \left( 1-\frac{1}{t}-\frac{1}{\mathsf{N }}\right) \frac{2}{\mathsf{N }} + \left( 1-\frac{1}{t}-\frac{1}{\mathsf{N }}\right) \left( 1-\frac{1}{t}-\frac{2}{\mathsf{N }}\right) \frac{3}{\mathsf{N }} +\cdots \cdots \\&\qquad \cdots \cdots + \left( 1-\frac{1}{t}-\frac{1}{\mathsf{N }}\right) \left( 1-\frac{1}{t}-\frac{2}{\mathsf{N }}\right) \cdots \left( 1-\frac{1}{t}-\frac{\hat{t}-2}{\mathsf{N }}\right) \frac{\hat{t}-1}{\mathsf{N }}. \end{aligned} \end{aligned}$$

(35)

If we assume $t\hat{t}\ll \mathsf{N }$, the $\frac{i}{\mathsf{N }}$ term appearing inside each set of parentheses can be ignored and the above may be approximated by

$$\begin{aligned} \frac{1}{\mathsf{N }} + \left( 1-\frac{1}{t}\right) \frac{2}{\mathsf{N }} + \cdots + \left( 1-\frac{1}{t}\right) ^{\hat{t}-2}\frac{\hat{t}-1}{\mathsf{N }}. \end{aligned}$$

(36)

Arguing as before, we can state that the probability for a chain not to reach a DP within its first $\hat{t}$ iterations, without falling into a loop, is

$$\begin{aligned} \left( 1-\frac{1}{t}-\frac{1}{\mathsf{N }}\right) \left( 1-\frac{1}{t}-\frac{2}{\mathsf{N }}\right) \cdots \left( 1-\frac{1}{t}-\frac{\hat{t}-2}{\mathsf{N }}\right) \left( 1-\frac{1}{t}-\frac{\hat{t}-1}{\mathsf{N }}\right) , \end{aligned}$$

(37)

and this can be approximated by

$$\begin{aligned} \left( 1-\frac{1}{t}\right) ^{\hat{t}-1}, \end{aligned}$$

(38)

under the assumption that $t\hat{t}\ll \mathsf{N }$.

The sum of (36) and (38)

$$\begin{aligned} \frac{1}{\mathsf{N }} + \left( 1-\frac{1}{t}\right) \frac{2}{\mathsf{N }} + \cdots + \left( 1-\frac{1}{t}\right) ^{\hat{t}-2}\frac{\hat{t}-1}{\mathsf{N }} + \left( 1-\frac{1}{t}\right) ^{\hat{t}-1}, \end{aligned}$$

(39)

is the probability for a chain not to reach a DP within its first $\hat{t}$ iterations. If we further assume that $\frac{\hat{t}}{t}$ is not too large, we can approximate and rewrite this in the form

$$\begin{aligned} \frac{t^2}{\mathsf{N }} \left( \frac{1}{t}+ e^{-\frac{1}{t}}\frac{2}{t} + e^{-\frac{2}{t}}\frac{3}{t} + \cdots + e^{-\frac{\hat{t}-2}{t}}\frac{\hat{t}-1}{t} \right) \frac{1}{t}+ e^{-\frac{\hat{t}-1}{t}}, \end{aligned}$$

(40)

and approximate the above once more with the definite integral

$$\begin{aligned} \frac{t^2}{\mathsf{N }}\int _0^{\frac{\hat{t}}{t}} e^{-u}u \;du + e^{-\frac{\hat{t}}{t}} = \frac{t^2}{\mathsf{N }}\left\{ 1-\left( 1+\frac{\hat{t}}{t}\right) e^{-\frac{\hat{t}}{t}}\right\} + e^{-\frac{\hat{t}}{t}}. \end{aligned}$$

(41)

This probability approaches $\frac{t^2}{\mathsf{N }} = O\big (\frac{1}{m}\big )$ very quickly, as $\frac{\hat{t}}{t}$ is increased. For example, even at the moderately large chain length bound of $\hat{t}= 15\,t$, the probability for a randomly generated chain to be discarded due to its length is $\frac{t^2}{\mathsf{N }}0.999995 + 3.05902\times 10^{-7}$, which is small enough for our purposes.

As a word of caution, we add that, due to the first term of $O\big (\frac{1}{m}\big )$ order, the number of long chains expected during the generation of a full DP matrix cannot be made arbitrarily close to zero by increasing the chain length bound, since each DP matrix contains m chains. Only the fraction of over-length chains among all chains approaches zero with the increase of the chain length bound.

Appendix 3: Rigorous proof of Lemma 3

The very short proof of Lemma 3 given in the main body of this paper will seem sufficient to most cryptographers. However, there are subtle issues involving random functions that were ignored in the proof. This section is an attempt at resolving these issues. We strongly urge any interested reader to review [14, Appendix B] before reading this section.

Recall that we are using $\mathscr {D}_k(F)$ to denote the set of elements of $\mathscr {N}$ that are k-many $F$-iterations away from their closest DPs, and suppose that $D_0$, $D_1$, ..., $D_k$ are any collection of mutually disjoint subsets of $\mathscr {N}$, with $D_0$ denoting the set of all DPs. Let us consider any function $F: \mathscr {N}\rightarrow \mathscr {N}$ and discuss when the series of conditions

• C1: $D_i = \mathscr {D}_i(F)$, for $i = 1, \dots , k$

would be satisfied by the function. Note that the omitted condition $D_0 = \mathscr {D}_0(F)$ is always satisfied.

The series of conditions C1 is satisfied by the function $F$ if and only if the following three items are all satisfied:

• C2: $D_i = \mathscr {D}_i(F)$, for $i=1, \dots , k-1$.

• C3: $F(D_k) \subset D_{k-1}$.

• C4: $F(\mathscr {N}\setminus D_k) \cap D_{k-1} = \emptyset$.

Repeating this argument, we see that the satisfaction of Condition-C1 by an $F$ is equivalent to the satisfaction of the following two series of conditions.

• C5: $F(D_k) \subset D_{k-1}$, ..., $F(D_2) \subset D_1$, and $F(D_1) \subset D_0$.

• C6: $F(\mathscr {N}\setminus D_k) \cap D_{k-1} = \emptyset$, ..., $F(\mathscr {N}\setminus D_2) \cap D_1 = \emptyset$, and $F(\mathscr {N}\setminus D_1) \cap D_0 = \emptyset$.

Suppose that we are now trying to construct a function that satisfies these conditions. Then the condition $F(D_k) \subset D_{k-1}$ would certainly places a restriction on how we may define $F$ on $D_k$. However, since the set $D_k$ does not overlap with any of the other sets $D_0, \dots , D_{k-1}$, none of the other sub-conditions of C5 places any restriction on the definition of $F$ on $D_k$. Next, the sub-condition $F(\mathscr {N}\setminus D_k) \cap D_{k-1} = \emptyset$ of C6 is certainly irrelevant to the definition of $F$ on $D_k$. The other sub-conditions of C6 are not quite so irrelevant, but since $D_{k-1}$ does not overlap with any of the sets $D_0, \dots , D_{k-2}$, as long as the condition $F(D_k) \subset D_{k-1}$ is adhered to, they are automatically satisfied and do not places any additional restriction of how we may define $F$ on $D_k$. More generally, for each $i = 1, \dots , k$, the condition $F(D_{i}) \subset D_{i-1}$ is the only restriction placed on the definition of $F$ on $D_{i}$ by C5 and C6. In other words, asking for C1 to be satisfied by a function $F$ places no restriction on the definition of $F$ on $D_i$ other than that $F(D_i)\subset D_{i-1}$ be satisfied.

The discussion given so far can be reinterpreted as follows. Let $D_0, D_1, \dots , D_k$ be any collection of mutually disjoint subsets of $\mathscr {N}$, with $D_0$ denoting the set of all DPs. By choosing a function $F: \mathscr {N}\rightarrow \mathscr {N}$ satisfying Condition-C1, one determines functions $F_i:D_{i} \rightarrow D_{i-1}$ for each $i=1,\dots , k$. Let us fix any i. If the choice of $F$ is made uniformly at random from the set of all function satisfying Condition-C1, the distribution of the corresponding $F_i: D_i \rightarrow D_{i-1}$, defined on the set of all functions having domain $D_i$ and co-domain $D_{i-1}$, also becomes the uniform distribution.

We are now ready to prove the lemma. Choose any explicit family of disjoint sets $D_0$, ..., $D_k$ such that $D_0$ is the set of all DPs. For a random function satisfying Condition-C1, since each $F_i: D_i \rightarrow D_{i-1}$ is a random function, we can expect to see

$$\begin{aligned} \frac{|F(D')|}{|D_{i-1}|} = 1 - \exp \left( -\frac{|D'|}{|D_{i-1}|} \right) \end{aligned}$$

(42)

for every set $D' \subset D_i$ and $i=1,\dots , k$. So far, we have fixed $D_0$, ..., $D_k$ first and have claimed information on iterated image sizes only for functions $F$ satisfying Condition-C1, which is associated with the sets $D_0$, ..., $D_k$. However, observe that this iterative equation depends only on the set sizes $|D'|$, $|D_0|$, ..., $|D_k|$ and not on the specific sets $D', D_0, \dots , D_k$. Also recall that (11) is expected of a random function. Interpreting this as (11) holding accurately for the vast majority of functions $F$, the truth of Lemma 3 follows. A review of [14, Appendix B] is required to fully understand this final statement.

Appendix 4: Optimal rainbow tradeoff parameters

In this section we discuss the parameter set for the perfect rainbow tradeoff that the previous analysis [2] suggested and claimed to be optimal. Let us explain the steps they take to fix the parameters, in the language of this work. The required success rate $\bar{\mathtt {R}}_{\mathrm{ps}}$ and the total number of table entries M are treated as the externally supplied parameters. First (28) is used to fix the number of tables $\ell = \lceil -\frac{1}{2}\ln (1-\bar{\mathtt {R}}_{\mathrm{ps}})\rceil$. This forces the number of starting points per table to $m = \frac{M}{\ell }$ and we know from (26) that the remaining parameter t must be fixed to

$$\begin{aligned} t = -\frac{\mathsf{N }}{m\ell }\ln (1-\bar{\mathtt {R}}_{\mathrm{ps}}) = -\frac{\mathsf{N }}{M}\ln (1-\bar{\mathtt {R}}_{\mathrm{ps}}), \end{aligned}$$

(43)

if the requested success rate is to be achieved. In short, the number of tables is chosen to be as small as possible, subject to the condition (28), and the rest of the parameters are set to what they must be in order to satisfy the externally given requirements.

It is not hard to show that the tradeoff coefficient $\bar{\mathtt {R}}_{\mathrm{tc}}$ of Theorem 2, when combined with (27), is an increasing function of $\ell$ in the range $\ell \ge 1$ and $0\le \bar{\mathtt {R}}_{\mathrm{ps}}< 1$. This implies that the parameters suggested by [2] are optimal in the sense that it gives the smallest possible tradeoff coefficient among those achieving the intended success rate. An easier approach would be to state that, the optimal parameter set of [2] corresponds to the rightmost empty circle in each box of Fig. 4, which is clearly the lowest dot in each box.

However, we want to emphasize that the rightmost circle of each box is just one of the many options that are available to the tradeoff algorithm implementer, and that this specific option is not likely to be favored over others when the pre-computation cost is taken into account, except possibly at low success rates. Furthermore, Fig. 6 demonstrates that there are success rates for which using parameters corresponding to the rightmost dot is certainly impractical.

Fig. 6

Pre-computation cost versus tradeoff coefficient for the perfect rainbow tradeoff at success rates slightly lower than $1-\frac{1}{e^2}=86.466$ %

The parameter set that is most efficient in terms of the online resource requirements is very often not the most practical or reasonable parameter set option among those made available by a tradeoff method. The current work provides the information which enables the tradeoff implementer to decide on the most sensible, rather than the most online-efficient, parameter to use.