Breaking Captcha System with Minimal Exertion through Deep Learning: Real-time Risk Assessment on Indian Government Websites

3.1 Type of Captchas

The first type of captchas (Type I) are image-based text captchas, the most commonly used captcha system worldwide, as well in most Indian government websites. Such types of captchas collected from the websites are shown in Figure 1. Type II captcha includes instruction separately in text format along with the image-format character sequence, which is evaluated following the instruction. The character sequence depicted in the image is referred to as expression text. This type of captcha is hard to crack as the understanding of both text and image are required and evaluated for obtaining the captcha answer. Figure 2 depicts the example of Type II captchas. Type III captcha is similar to Type II, except here text instructions are embedded into the image along with character sequence expression text, shown in Figure 3. Type III type of captcha are more complex to solve as the character sequence depicted in the image include instruction text and expression text together. First the segregation of instruction text and expression text is required to evaluate the captcha answer.

Fig. 1.

View Figure

Fig. 2.

View Figure

Fig. 3.

View Figure

3.2 Major Websites

In the work, we consider seven Indian government websites based on the importance of information, likely to be extracted from the websites, during an attack. The importance or value of information in context with a website is dependent on two factors. First is the significance of the information that can be extracted using web scraping. The second factor is the real-time impact due to the stoppage of service provided by the particular website. From the attacker’s perspective, the maximum gain is proportional to the high value of information scraped from the website. In Appendix A, we consider the websites having high information value and provide details of the captcha systems along with the corresponding information type. The other government websites use captcha systems, too, but they have low information value, as mentioned in Appendix B. It is important to note that none of the websites mentioned in Appendix B has a different captcha type other than the types mentioned in Section 3.1.

3.3 Deep Network Background

Over the past few years, deep learning techniques have been used as a major development step in the field of machine learning [15, 24]. Breakthrough improvements have been observed in many applications associated to image recognition [16, 37], image captioning [22, 42], language translation [45], and natural language processing [49]. Neural networks are proved to be productive in creating and solving captcha systems [13, 27, 41, 48], due to the effectiveness of deep learning models, the use of which enhances the adaptation capacity of the models. On updating the captcha system, the model can be retrained without significant change in the architecture. We use encoder-decoder architecture inspired from sequence-to-sequence models in the field of language modeling and translation [28, 39]. Both RNNs and CNNs are used as components of the model architecture for captcha solvers.

Traditional RNNs are susceptible to the vanishing gradient problem. The two most widely used variant networks that solve the vanishing gradient problem are Long Short Term Memory (LSTM) [19] and Gated Recurrent Unit [8]. We use LSTM as a decoder unit and bi-directional LSTM [14] for a sequence encoding unit, the basic components of the architectures. For future reference and explanation, we represent the LSTM by function “\(\vec{\Gamma }\)” for a given set of sequence vector “\(\vec{\theta }\),” described in Equation (1), (1) \(\begin{equation} \vec{Y}= \Gamma ^{LSTM}(\vec{\theta }). \end{equation}\)

For the encoder-decoder model with text sequence, we also use the attention mechanism for decoder prediction [2]. It primarily helps to memorize long-range sentences in neural sequence prediction. Apart from using hidden state of the encoder, the decoder also has the weighted vector of all the encoder output states. We use attention mechanism proposed in Reference [29], which is improvement of the initial attention mechanism [2].

CNN is used as an image feature extractor for captcha image, represented by function “\(\vec{\Lambda }\)” in Equation (2) where vector “\(\vec{\phi }\)” is the input set of 2D vectors. We use Exponential Linear Unit (ELU) as activation function in CNN [9] to alleviate the vanishing gradient problem that improves the learning characteristics and speed up the learning process. ELU performs slighty better compared to rectified linear units (ReLUs), leaky ReLUs, and parametrized ReLUs [34], (2) \(\begin{equation} \vec{Y}= \Lambda ^{CNN}(\vec{\phi }). \end{equation}\)

FC networks are used for label prediction and represented by function “\(\vec{\Delta }\)” in Equation (3) using linear vector “\(\vec{\psi }\)” [21], (3) \(\begin{equation} \vec{Z}= \Delta ^{FC}(\vec{\psi }). \end{equation}\)

4.1 Type I Captcha Model

The Type I captcha is the most used form of captcha, which contain an image displaying a sequence of characters. The character sequence in the image needs to be recognized to solve the captcha. The general solver architecture for the Type I captcha is shown in Figure 4 and is referred to as Architecture I. The input to the network is the raw image, and output labels are the corresponding list of characters present in the image. The model has a similar basic encoder-decoder structure, proposed earlier in Reference [13]. The image is passed through the convolution unit, which consists of three basic layers. A combination of convolution layer and max-pooling layer followed by a batch normalization function represent a convolution unit. Batch normalization helps the process in two ways. First, it allows us to have higher learning rates, which significantly reduces the training time of the model [20]. Second, it has been used as an alternative to dropout, which in turn function as a regularizer. The “ELU” activation function is used in the convolution unit. A color channel image input is passed through a set of convolution units that act as an encoded feature extractor for Type I captchas. On the decoder side, an LSTM is used to predict the output label. In the proposed modified version for the usage of architecture in different captcha schemes of Type I captchas, the size of the LSTM decoder unit is kept variable. The length of the LSTM varies according to the captcha type and image size used in that scheme. The decoder predicts a single character at a time, as shown in Figure 4, and returns a \(\lt EOS\gt\) tag when the sequence ends. Equation (4) explains the image encoder and label decoder process for input image vector \(\vec{\phi }\) and output label vector \(\vec{Z}\). Two major aspects we focus on experimentally are the number of convolution units and the variable vs. static size of the LSTM decoder.

Fig. 4.

View Figure

Generally, the FC layer is used to reduce the size of the encoded vector obtained from the convolution unit, which is then passed to initialize the LSTM decoder unit. The FC layer helps in keeping the size of the LSTM decoder static for every type of Type I captcha scheme. However, the presence of a static size LSTM decoder unit requires a larger amount of manually generated label data compared to the variable or dynamic length decoder. The experimental result is provided in Section 6 to demonstrate the effect of the static and dynamic decoder length with the number of captcha data-points. The direct flattened vector length of the LSTM decoder tends to perform better. We use three convolution layers for effective performance improvement over two layer convolution network used by Garg et al. [13], (4) \(\begin{equation} \vec{Z}_{Label}= \Delta ^{FC}\Gamma ^{LSTM}_{Decoder}\left(\Lambda ^{CNN}_{Captcha Image}(\vec{\phi })\right). \end{equation}\)

4.2 Type II Captcha Model

Type II captcha consists of text instructions, representing the task to perform to solve the captcha based on an expression. For example, the task in the text instructions is “Evaluate the expression,” while the expression that is present in the image depicts “\(6+8=.\)” To solve the captcha, the network has to predict the answer, “14.” We propose two neural network pipelines for solving this type of captcha. The first architecture has two encoders and one decoder, as shown in Figure 5, and is referred to as Architecture IIA. In Architecture IIA, one encoder has been implemented using a Convolution Unit and used for feature extraction from the image, while the other one is used for encoding the task. The convolution encoder used here for image encoding is similar to the encoder used in Architecture I. The task encoding is performed by a stacked bi-directional LSTM, where the encoded vector is formed by concatenating the last hidden state of both the forward and backward LSTMs, as shown in the Task Encoder Unit of Figure 5. The feature vector has been obtained after concatenating the encoder vector of the Task Encoder Unit and Convolution Unit, containing the final encoded information. The feature vector is represented by Equation (5) for input image vector \(\vec{\phi }\) and the text word sequence embedding vector \(\vec{\theta }\). The LSTM decoder unit is responsible for performing reasoning and evaluation of encoded information, required to solve and produce the answer label for the Type II captcha. The captcha answer label sequence vector \(\vec{Z}\) is obtained using the feature vector \(\vec{T}\), as shown in Equation (6). The backpropagation algorithm [18] has been applied for evaluating the error on predicting the captcha answer label, (5) \(\begin{equation} \vec{T}= \Lambda ^{CNN}_{Captcha Image}(\vec{\phi }) \oplus \Gamma ^{LSTM}_{Encoder}(\vec{\theta }), \end{equation}\) (6) \(\begin{equation} \vec{Z}_{Label}= \Delta ^{FC}\left(\Gamma ^{LSTM}_{Decoder}(\vec{T})\right). \end{equation}\)

Fig. 5.

View Figure

The Architecture IIA performs reasonably well when the number of different types of task is smaller in text instructions. Another architecture that has been proposed in this article for solving Type II captcha is shown in Figure 6, which is referred to as Architecture IIB. We have two types of label data in Type II captcha. One is the result of the captcha answer label (“14”), evaluated based on the task, similarly used to train the Architecture IIA. The other available output information for solving the Type II captcha is the optical character sequence expression label, depicted in the image (“\(6+8=\)”), which is not used in Architecture IIA. The character sequence expression is predicted as an output from the raw image using CNN and an LSTM decoder unit. The Architecture IIB has two encoder networks performing image encoding by the Convolution Unit and task encoding by the Task Encoder Unit, as shown in Figure 6. It has an additional LSTM decoder unit that predicts the character sequence expression labels from the image, called Image to Expression Decoder Unit. Equation (7) evaluates the predicted character sequence expression labels from the image using the 2D image vector \(\vec{\phi }\). The character sequence expression labels are then encoded using a bi-directional LSTM encoder unit, as shown in the Expression Encoder Unit of Figure 6. The final feature vector is obtained by concatenating the encoder vectors of the Task Encoder Unit and Expression Encoder Unit. Finally, the feature vector is passed through the Label Decoder Unit to obtain the Type II captcha answer label. Equation (8) represents the final feature vector \(\vec{T}\), and Equation (9) outputs the captcha answer labels \(\vec{Z}_{Label}\) for the network model, (7) \(\begin{equation} \vec{Y} = \Delta ^{FC}\left(\Gamma ^{LSTM}_{Decoder}\left(\Lambda ^{CNN}_{Captcha Image}(\vec{\phi })\right)\right), \end{equation}\) (8) \(\begin{equation} \vec{T}= \Gamma ^{LSTM}_{Encoder}(\vec{\theta }) \oplus \Gamma ^{LSTM}_{Encoder}(\vec{Y}), \end{equation}\) (9) \(\begin{equation} \vec{Z}_{Label}= \Delta ^{FC}\left(\Gamma ^{LSTM}_{Decoder}(\vec{T})\right). \end{equation}\)

Fig. 6.

View Figure

In Architecture IIB, the Convolution Unit and the Image to Expression Decoder Unit together work as an independent optical character recognition unit for a specific captcha scheme. Therefore, both the unit can be trained simultaneously using character sequence expression labels from the image. Later, the predicted character sequence expression labels are encoded by Expression Encoder Unit, which is utilized to predict the captcha answer. In Architecture IIB, learning in the complete network takes place with two type of information (captcha answer labels, expression sequence labels) while one type of information (captcha answer labels) is used in Architecture IIA. Hence, Architecture IIB performs more effectively with few numbers of captcha. The results of the effectiveness of both architectures are shown and discussed in Section 6. In Architecture IIB, the first backpropagation (represented as “Backpropagation 1” in Figure 6) is performed on the sequence loss between the computed character sequence expression label and the original character sequence expression label. The second backpropagation algorithm (“Backpropagation 2”) is applied by calculating the loss between the original captcha answer label and the computed captcha answer label.

4.3 Type III Captcha Model

The Type III captcha is the most complicated captcha among the three. Like Type I captcha, it has only one input, that is, the raw image, and similarly to Type II captcha, it consists of task instruction to perform on the expression. Both the task instruction and character sequence expression are embedded in the image. So we split the task into three sub-tasks as follows: (i) image-to-character label prediction, (ii) character sequence–to–possible independent word prediction, and (iii) the final independent word–to–captcha answer label prediction. The proposed network architecture pipeline is shown in Figure 7 and referred to as Architecture III. The first sub-task is to decode the character from the image, similarly to Architecture IIB. Equation (10) evaluates the predicted character sequence label \(\vec{X}\) using the image vector \(\vec{\phi }\). The second sub-task is to use the attention-based encoder-decoder unit for prediction of an independent word from the predicted character sequence of the input image using the Character encoder Unit and Character to Independent Word Decoder Unit in Figure 7. The attention-based encoder-decoder unit is important for separating the words representing the task instruction and the characters, representing the expression. The attention module helps to predict the word based on the character occurrence in the sequence. Equation (11) evaluates the independent word sequence \(\vec{Y}\) given the character sequence \(\vec{X} as input\). The independent word sequence \(\vec{Y}\) is now passed through the last encoder-decoder module of Figure 7 to produce the captcha answer label, as given in Equation (12), (10) \(\begin{equation} \vec{X}= \Delta ^{FC}\left(\Gamma ^{LSTM}_{Decoder}\left(\Lambda ^{CNN}_{Captcha Image}(\vec{\phi })\right)\right), \end{equation}\) (11) \(\begin{equation} \vec{Y}= \Delta ^{FC}\left(\Gamma ^{LSTM}_{Decoder}\left(\Gamma ^{LSTM}_{Encoder}(\vec{X})\right)\right), \end{equation}\) (12) \(\begin{equation} \vec{Z}_{Label}= \Delta ^{FC}\left(\Gamma ^{LSTM}_{Decoder}\left(\Gamma ^{LSTM}_{Encoder}(\vec{Y})\right)\right). \end{equation}\)

Fig. 7.

View Figure

The use of three encoder-decoder units helps to learn the model in three levels using backpropagation. The first backpropagation (represented as “Backpropagation 1” in Figure 7) is based on the loss between the predicted character sequence label and the original character sequence label present in the image. The second backpropagation (“Backpropagation 2”) is performed on the difference of predicted independent word sequence obtained from the character to the real independent word sequence. The loss between the computed captcha answer label and the original captcha answer label is used for the third backpropagation (“Backpropagation 3”). The learning from all three levels is possible using a single manually annotated captcha. The inference of the network requires only the raw 2D captcha image.

5.1 Network Parameters

The aim of experimentation is to keep the number of network parameters or the network depth as low as possible. Fewer network parameters reduce inference and training time on a low-end resource. Ye et al. [48] mentioned that the requirement of the number of manually labeled captcha is also significantly reduced with minimum network parameters. For each of the proposed architectures, the network parameters are explained below in detail.

The Architecture I captcha model solver consists of one convolution unit and one decoder unit. There are many well-known models for image recognition, starting with basic models like LeNet [26] to advance architecture like ResNet [17], VGG [37], and Inception [40]. The advanced models are difficult to train due to the large number of trainable parameters that require a sufficient number of manually trained labels. An extension of LeNet architecture is considered in this article for feature extraction from the captcha image. The simplicity of the network requires the least training data, and it performs the fastest prediction. Each convolution unit consists of three convolution layers along with batch normalization and an ELU activation function after each layer. The window size of (3 * 3) with strides as (1, 1), (2, 2), and (3, 3) are used in three convolution layers, respectively. To obtain faster inference, the filter size has been adjusted according to the performance of the model for each individual website. The number of layers is kept constant to have a generalized architecture that works efficiently for all the websites using Type I captcha. The decoder consists of an LSTM unit that recursively produces the captcha text label. After the convolution unit, the dynamic size LSTM decoder is used instead of an FC layer (which are normally used in fixing the LSTM size) for every Type I captcha. The FC layer leads to high convergence time and a less-accurate model. Figure 8 provides accuracy vs. epoch graph depicting the same effect. It is important to note that the dynamic size decoder LSTM does not change the architecture pipeline as the size is mapped automatically according to the size of the captcha image. Table 1 provides details about the maximum length of predicted sequence label for the Indian government websites that use the Type I captcha system. Table 1 also presents one sample captcha for each website and the corresponding label size, which represents the number of classes or unique characters used in the captcha scheme.

Fig. 8.

View Figure

Table 1.

View Table

Table 1. Websites for Type I Captcha with Label Detail

In this article, two architectures are discussed for the Type II captcha solver model. The model in Figure 5 has a bidirectional LSTM of size 256 unit for task instruction word encoding. The GloVe [35] 100-dimensional word embedding vectors are used to represent the corresponding word in the instruction. The three convolution unit has window of size (3 * 3), (5 * 5), and (3 * 3) with the number of units as 64, 32, and 16, respectively. The strides used in each convolution layer are (1, 1), (2, 2), and (2, 2) and max-pooling with window size (3, 3). This type of captcha is used in the Indian Post website (A.6). The flattened vector obtained from the convolution unit is combined with the output of bi-directional LSTM. The concatenated vector is used for prediction of a dynamic size sequence label to obtain the answer of final captcha. The second model for the Type II captcha described in Figure 6 has the same convolution unit configuration in all three layers. The expression character sequence is obtained from the decoder. The character sequence is then encoded using the bi-directional LSTM of size 1,024 unit. The size of the bi-directional task encoder unit is kept at 64 on the basis of the experimental result. The final architecture, shown in Figure 7, is used for solving Type III captcha, deployed on the IRCTC website (A.7). The three convolution layers with a window of size (3 * 3) has been used while strides are (1, 1), (2, 2), and (1, 1) with the number of unit 64, 32, and 8, respectively, considered in three successive layers. The size of the bi-directional LSTM character encoder unit and character to independent word decoder LSTM unit are set to 256 unit. The third encoder-decoder unit also has a size of 256 unit.

5.2 Dataset

The manually labeled data are created by capturing original captcha from each of the major Indian government websites considered in this work. The process of extraction of real-time captchas began in August 2021. Annotating 1,000 data-points by scraping a website requires nearly one and a half working hours. As the time required by annotator varies from person to person for the same amount of work, a working hour is roughly calculated as the average time spent to annotate 1,000 data-points for a specific website by five human annotators. The number of data-points annotated for each website along with the working hours that are spent to do the same is shown in Table 2. The Type I captcha requires less time for manual annotation compared to Type II and Type III captcha, as it requires annotation of only captcha text characters present in the image. The time requirement for Type I captchas also varies according to the number of characters and complexity of text present in the images, as described in Table 1. In annotation of the Type II and Type III captchas, the annotator need to register task instruction, captcha solution, and the text character sequence depicted in the image. We also prepare a synthetic dataset for testing the Type III captcha architecture, as there is a class imbalance on the type of task information. The synthetic dataset is created by superimposing the task information of Type II captcha along with expression in an image to convert into Type III captcha. The synthetic dataset is used for architecture validation only.

5.3 Executing Environment

Training and validation of the architecture are executed using the Tensorflow [1] version 1.8 library in python 3.6 [36]. The hardware system used for training the network consists of a single Nvidia Tesla V100 GPU. The inference and testing process is carried out in a smaller desktop Nvidia 1050Ti GPU.

7.1 Analysis

We calculate the robustness of the captcha system to rate the vulnerabilities for each particular website. Robustness of the captcha system is directly proportional to the human efforts required to solve the system. With the context of deep architectures, the robustness of the captcha system mainly depends on two important factors. The factors are the complexity of architecture and the number of manually annotated captcha required to train an effective captcha solver. The complexity of an architecture is measured on the number of encoder-decoder module used in the respective architecture as shown in Table 2. The human effort required to annotate each captcha type differs and depends on factors like the presence of text instructions, the maximum number of characters in the label, label type, and type of operation as per instruction. Working hours for annotating each captcha dataset is specified in Table 2. The working hours are considered for evaluation of the complexity of the captcha system. The other important aspect is the accuracy; the minimum 80% of test accuracy has been considered as a threshold for an effective captcha solver model. The minimum number of annotated captcha required to obtain 80% accuracy for each model is the evaluation criteria, which is considered in this article.

The Robust score (\(\gamma\)) for a captcha system is evaluated by adding the Architecture complexity value (\(\alpha\)) and manual annotating working hours required to achieve 80% accuracy for the model (\(\beta\)), defined in Equation (13). Table 5 shows the Robust score for each dataset and the corresponding model architecture. The higher Robust score represents a more complex and hard-to-break captcha system, (13) \(\begin{equation} \begin{split} Robust\hspace{2.84526pt}Score(\gamma) = \log (\alpha) * \log (\beta + 1), \end{split} \end{equation}\) where \(\alpha\) is the number of the Encoder-Decoder unit in the solver architecture and

Table 5.

View Table

Table 5. Robust Score for the Website Dataset and their Corresponding Model Architecture

\(\beta\) is the approximate working hours require to manually annotate the data-points to reach 80% accuracy.

The risk to a website is dependent on the importance of that website and how vulnerable it is to attacks. The risk to a website is measured by subdividing it into two major categories (i) intrinsic properties corresponding to the vulnerability of the captcha system and (ii) the extrinsic properties to launch an attack.

The first property (i.e., intrinsic properties corresponding to the vulnerability of the captcha system) is inversely proportional to the Robust score, which measures the strength of the captcha system, whereas the extrinsic properties, i.e., the non-captcha factors, which attract or discourage an attacker from launching an attack on a website, is determined through the calculation of the Website value score. The Website value Score of a government website is dependent on three properties, which are whether the website has important scraping content, is susceptible to a DoS attack, and has complex fields in the web page. Complex fields on the web page refer to the text boxes, which are hard to predict by random guessing. Equation (14) represents the formulation of Website value score, and the values for the corresponding websites are shown in Table 6, (14) \(\begin{equation} {\it Website} \hspace{2.84526pt}{\it value}\hspace{2.84526pt}{\it score} (\delta)= \phi + \chi - (0.5 * \psi), \end{equation}\) where \(\begin{align*} \hspace{56.9055pt} & \phi = {\left\lbrace \begin{array}{ll} 1, & \text{if } website \hspace{2.84526pt} has \hspace{2.84526pt} important\hspace{2.84526pt}scraping\hspace{2.84526pt} content\\ 0, & \text{otherwise} \end{array}\right.} \\ & \chi = {\left\lbrace \begin{array}{ll} 1, & \text{if } website \hspace{2.84526pt} susceptible \hspace{2.84526pt} to \hspace{2.84526pt} DoS \hspace{2.84526pt} attack\\ 0, & \text{otherwise} \end{array}\right.} \\ & \psi = {\left\lbrace \begin{array}{ll} 1, & \text{if } website \hspace{2.84526pt}has \hspace{2.84526pt}complex \hspace{2.84526pt}field \hspace{2.84526pt}in \hspace{2.84526pt}the \hspace{2.84526pt}web-page \hspace{2.84526pt}along \hspace{2.84526pt}with\hspace{2.84526pt} Captcha \hspace{2.84526pt} system \hspace{2.84526pt}\\ 0, & \text{otherwise}\end{array}\right.}. \end{align*}\)

Table 6.

View Table

Table 6. Risk Assessment for the Organizational Websites

The final Risk score for each of the considered website is shown in Table 6 and calculated using Equation (15),

7.2 Inferences and Suggestions

Based on the analysis, a few inferences have been derived from Tables 5 and 6. The solver for Type II and Type III captcha has more complex architecture and tends to have a high Robust score. However, the system using Type I captcha has high variance on the Robust score. It has been observed that the captcha systems using variable length labels are hard to converge and have a high Robust score, as shown in Table 5. On the contrary, the fixed-character Type I captchas are easy to train and have a low Robust score.

From the analysis, it is clear that a well-designed variable character Type I captcha system works at par with Type II captcha. But the Type III captcha is the most complex and most difficult to break. There is still some room for improvement while designing Type I captcha by making changes in the images. The Type II and Type III captchas both have a considerable amount of space for improvement. While designing, the possible improvement is not only limited in the images—there is a scope of improvement in text instructions and expression. One such scenario is to use a more complex mathematical expression. In the original captcha dataset of the website, the only arithmetic operations are addition and subtraction. An extended operation like multiplication, division, or any other mathematical expression that is easy to evaluate for humans is encouraged.

The combination of natural language and image in captcha could be the future of the captcha system, and a lot of improvement is possible. But the focus of our work is to explore and escalate the vulnerabilities involved in the government websites where the current captcha systems are deployed. The proposed architectures are able to solve different types of captcha systems with high efficiency using the original captcha dataset of the websites.

7.3 Justification to State of the Art

In this work, we use neural network models to solve text and text instructions–based captcha. The neural network models are highly effective in solving text captchas, as reported in the previous studies [41, 48]. The neural network models are easy to implement because of its black box nature, and with minimum effort and domain knowledge could be used by an attacker. The other significant advantage of using the neural network models is the inference speed of neural network pipelines. The work reported by the team of Gao [10, 11], Ye et al. [48], and Tang et al. [41] provide enough practical evidence that a neural network model has much faster inference speed compared to the traditional methods. Therefore, the neural network pipeline is equally effective in solving text and text instructions captchas, which we analyzed and presented in thisarticle. From the viewpoint of an attacker, the inference speed always plays a pivotal role when a web-server is highly vulnerable with the frequency of attacks. In the study, we observe that the IRCTC website (A.7) that provides critical service during Tatkal time could be highly affected with fast attacks. Along with critical service disruption, data extraction is faster with a high-speed inference model. Attacker plans to scrape data from a website by filling text fields in the web-page with a well-defined random value and solving the required captcha. Therefore, to scrape data from a large government website, the neural network models with low inference time is favored. In this article, we attempt to develop and propose neural network models for its effectiveness and inference speed, which contributes to the factor for solving the captchas.

The model used for Type I captcha is similar to the work proposed by Garg and Pollett [13] and Ye et al. [48]. Both studies have a similar basic CNN structure known as LeNet [26]. Garg and Pollett considered a complete synthetic grey captcha image dataset to train the model. The synthetic dataset contains 1 million simple images of fixed-length 5, 2 million complex images of fixed length 5, and 13 million images of variable length. However, it is not feasible to create such a large dataset on real-time captcha system for each website. Though the architecture is simple, they overestimate the requirement of the size of the dataset. Our analysis shows that the architecture could also be effective, using only a few thousand data points. Our model is a modified version to the one proposed by Garg and Pollett, consisting of three convolution layer instead of two. In their CNN model, they did not use batch normalization, which reduces the need for a huge number of training samples and training time. In this article, we demonstrate that the fine-tuned architecture could be adequately trained from scratch to break a captcha system with a limited amount of training datasets. From the analysis, we exhibit that the model is equally effective with color captcha images. Another work, as proposed by Ye et al. [48] (explained in 2) for Type I captcha uses GAN architecture, has a major bottleneck in the viewpoint of an attacker. The attacker needs to put an extensive amount of effort to design the base classifier by considering various captcha scheme. Moreover, an attacker has to effectively implement the complex fine-tuned process of captcha layers for each new captcha system. The number of captcha layers that need to be retrained adds another new hyper-parameter to the task. The model also uses a static FC layer for prediction of captcha labels in place of a dynamic decoder, which makes the architecture less adaptable to a new captcha scheme with variable length. In this article, the model considered for the analysis of Type I text captcha is simple, effective, and easy to implement. The model architecture can be implemented for any captcha system from scratch and independently from any other captcha scheme, unlike the approach used by Ye et al. [48], which is dependent on various other captcha schemes used for training of the base solver.

The text instructions–type captchas are relatively new and less explored. To the best of our knowledge, there is no solution architecture proposed to date for text instructions captcha. A novel architecture pipeline with faster inference process has been proposed in this article for text instructions (Type II and Type III) captchas. The architecture is easy to understand and effectively use multiple encoder-decoder models to predict the correct captcha answer. It is important to note that for Type II captcha, the statistical or rule-based system could be combined with the neural network model for correct label prediction, where the rule-based system can be applied for interpreting the instructions. However, the system fails when the number of instruction is large enough. The proposed models will thrive in such a situation, as the architecture incorporates the language model or language understanding approach [30, 38], which eliminates the rule-based reasoning process. The use of a neural network pipeline architecture also significantly increases the inference speed. In Type III text instructions captcha, the text instructions needs to be extracted from the image along with the text expression. The LSTM attention module helps to predict the word, which, in turn, is used to predict the text instructions. An attention module also plays a vital role in differentiating between expression and text instructions present in the single image. Having a single pipeline is the major challenge in solving such captcha and the model proposed in the work effectively satisfy the need, demonstrating the novelty of the work.