Fact-checked by Grok 2 weeks ago

Textual entailment

Textual entailment, also known as recognizing textual entailment (RTE), is a fundamental task in natural language processing (NLP) that involves determining whether the meaning of one text fragment, called the hypothesis, can be inferred from another fragment, called the premise, based on their semantic content.^[1] This directional relationship is asymmetric, meaning that entailment from premise to hypothesis does not imply the reverse, and it often incorporates a probabilistic or "likely" inference rather than strict logical deduction to account for real-world linguistic variability.^[2] RTE serves as a unified evaluation framework for assessing a system's ability to capture semantic inferences across diverse NLP applications, such as question answering, summarization, and information extraction.^[2] The concept of textual entailment traces its roots to early efforts in computational semantics, with the FraCaS project (1994–1996) introducing a test suite of 350 inference problems focused on linguistic phenomena like quantifiers, anaphora, and ellipsis to evaluate semantic theories.^[2] RTE was formally defined and popularized in 2006 through the PASCAL RTE Challenge, which aimed to standardize evaluation of semantic inference by creating datasets of text-hypothesis pairs drawn from real-world sources, labeling them as entailment or not-entailment.^[1] Subsequent annual challenges from 2007 to 2013 introduced three-way labeling (entailment, contradiction, or unknown) starting as a pilot in 2007 and main task from 2009, expanding the datasets to over 10,000 pairs and fostering advancements in machine learning approaches to RTE.^[2]^[3] In the deep learning era, RTE evolved into natural language inference (NLI), with large-scale crowdsourced datasets like the Stanford Natural Language Inference (SNLI) corpus (2015), containing over 570,000 sentence pairs annotated for entailment, contradiction, or neutral relations, enabling training of neural models such as BiLSTMs and transformers. The Multi-Genre NLI (MultiNLI) dataset (2018) further broadened this by including 433,000 examples from ten diverse genres, including fiction, telephone conversations, and government reports, to improve generalization across domains. These resources have driven state-of-the-art performance, with models like BERT achieving over 90% accuracy on SNLI, though human baselines remain higher at around 95%.^[2] RTE's importance lies in its role as a proxy for broader semantic understanding, underpinning tasks like recognizing contradictions in fact-checking or generating paraphrases in machine translation, and it highlights challenges such as lexical ambiguity, world knowledge gaps, and dataset biases that lead to spurious correlations in models.^[2] Recent advances include specialized datasets targeting phenomena like negation and monotonicity, as well as multimodal extensions incorporating images; as of 2025, large language models have pushed NLI performance near human levels on standard benchmarks but continue to struggle with adversarial examples and domain-specific inference, spurring new resources like the chemistry-focused CRNLI dataset.^[2]^[4]

Fundamentals

Definition

Textual entailment (TE) is a fundamental task in natural language processing that involves determining whether a text T semantically entails a hypothesis H, such that the truth of T guarantees the truth of H based on their natural language meanings.^[5] This directional relationship captures whether the information conveyed by T implies H, relying on linguistic interpretation and background knowledge to assess inferential validity.^[5] TE is typically approached as either a binary classification, distinguishing entailment from non-entailment, or a three-way classification that additionally identifies contradictions (where T implies the falsity of H) and neutral cases (where no clear inference holds).^[6] The binary formulation focuses on the presence or absence of entailment, while the three-way variant provides finer-grained evaluation of semantic relations.^[6] Within natural language understanding (NLU), TE occupies a pivotal position at the intersection of semantics and pragmatics, testing systems' ability to perform inferences that extend beyond strict logical deduction to include contextual and commonsense reasoning.^[7] It addresses the inherent variability in real-world texts, where expressions of the same idea can differ in wording, structure, or implied knowledge, making robust inference essential for applications like question answering and summarization.^[8] Recognizing textual entailment (RTE) emerged as a standardized challenge to benchmark progress in this area.^[5]

Formalization

Textual entailment is formally modeled in logical terms as a semantic entailment relation, where a text T entails a hypothesis H, denoted T \models H, if every model satisfying T also satisfies H when both are represented in first-order logic (FOL).^[9] This model-theoretic approach draws from classical semantics, treating entailment as holding in all possible worlds where T is true, thereby capturing strict logical implication without reliance on pragmatic factors.^[10] Systems implementing this formalization often translate natural language sentences into FOL via discourse representation structures and use automated theorem provers to verify the implication.^[9] A probabilistic formalization relaxes the strictness of logical entailment by defining T entails H if the conditional probability P(H \mid T) exceeds a high threshold \tau, such as 0.9, indicating that H is highly likely to be true given T.^[11] This approach incorporates uncertainty from world knowledge and language variability, often interpreted in Bayesian terms where prior probabilities P(H) are updated by evidence from T.^[11] For instance, lexical overlap models compute P(H \mid T) via generative probabilities over terms, enabling approximation of entailment in noisy data.^[11] Extensions to three-way classification refine the binary framework by distinguishing entailment, contradiction, and neutral relations. Entailment holds as before; contradiction occurs when T \models \neg H, meaning H is false given T; and neutral applies when neither entailment nor contradiction is true, leaving H's truth uncertain relative to T.^[12] This classification, widely adopted in datasets like SNLI, better reflects natural language inference by accounting for cases requiring additional context.^[12] Entailment relations in textual settings exhibit monotonicity, where inferences preserve directionality based on context polarity: upward-monotone contexts (e.g., affirmative clauses) allow weakening via hypernyms, while downward-monotone contexts (e.g., under negation) require strengthening via hyponyms.^[13] However, textual entailment is often defeasible, meaning plausible inferences can be overturned by new information, unlike strict monotonic logical entailment, due to reliance on probabilistic world knowledge rather than exhaustive models.^[14] This defeasibility underscores the pragmatic nature of the task, prioritizing typical human judgments over absolute certainty.^[14]

Examples

Entailment Cases

Textual entailment cases exemplify scenarios where the meaning conveyed by a text (T) justifies a human reader in inferring the truth of a hypothesis (H), typically without requiring additional assumptions beyond standard linguistic or background knowledge.^[15] Simple lexical entailment arises from semantic relations like hyponymy or synonymy between corresponding terms in T and H, while preserving overall structural alignment. For instance, consider T: "A man is playing soccer" and H: "A person is playing a sport." The entailment holds because "man" is a hyponym of "person" (a specific type of human), and "soccer" is a hyponym of "sport" (a particular athletic activity), with entities (the individual) and core relations (playing) directly aligning to support inference.^[16] This type of case succeeds due to straightforward lexical substitution that maintains semantic compatibility without altering propositional content.^[11] Syntactic entailment involves structural rephrasing or transformation in natural language that does not change the underlying meaning, often through nominal-to-verbal shifts or clause adjustments. An illustrative pair is T: "Hepburn, a four-time Academy Award winner, died last June in Connecticut at age 96" and H: "Hepburn, who won four Oscars, died last June aged 96." Here, the nominal phrase "a four-time Academy Award winner" aligns syntactically with the relative clause "who won four Oscars" (noting "Oscar" as a synonym for "Academy Award"), while entities (Hepburn), events (winning, dying), and temporal details match precisely.^[17] Success in such cases depends on parse tree alignments that confirm equivalent predicate-argument structures across syntactic variations.^[18] World knowledge entailment requires integrating commonsense or factual background information to bridge T and H. For example, T: "Birds can fly" and H: "Eagles can fly" is entailed because eagles are a specific subtype of birds, and the general capability attributed to birds applies downward to this hyponym via taxonomic knowledge. Similarly, T: "Norway’s most famous painting, ‘The Scream’ by Edvard Munch, was recovered Saturday" entails H: "Edvard Munch painted ‘The Scream’," drawing on cultural knowledge of artistic attribution where the credited creator implies authorship.^[19] These pairs succeed through entity coreference (e.g., the painting and artist) and relational extension (authorship to painting action), grounded in verifiable external facts that reinforce rather than contradict T's assertions.^[15] In all these entailment cases, the key to success lies in robust alignment of core entities, preserved relational semantics, and minimal reliance on probabilistic or context-dependent interpretations, ensuring the inference is directionally reliable from T to H.^[20]

Non-Entailment Cases

Non-entailment in textual entailment arises when the hypothesis cannot be reliably inferred as true from the premise, encompassing both contradiction and neutral relations. In contradiction cases, the premise and hypothesis describe scenarios that cannot both be true simultaneously, leading to a semantic opposition. For instance, given the premise "The event was canceled" and the hypothesis "The event took place," the direct opposition in outcomes results in a contradiction, as the cancellation explicitly precludes the event occurring.^[16] Similarly, in the premise "A man inspects the uniform of a figure in some East Asian country" paired with the hypothesis "The man is sleeping," the active inspection conflicts with the state of sleep, exemplifying a contradiction through incompatible actions.^[16] Neutral relations occur when the premise provides insufficient information to confirm or deny the hypothesis, leaving its truth possible but not entailed. A representative example is the premise "Some dogs are brown" and the hypothesis "All dogs are brown," where the partial commitment in the premise does not support the universal claim in the hypothesis, resulting in neutrality due to lack of full semantic coverage. Another case involves the premise "An older and younger man smiling" and the hypothesis "Two men are smiling and laughing at the cats playing on the floor," where the additional details about laughing and cats introduce elements not implied or contradicted by the premise, yielding a neutral relation.^[16] Cases of partial overlap failure further highlight non-entailment through semantic mismatch, where shared elements do not yield implication. Consider the premise "John entered the room" and the hypothesis "John left the room"; while both involve John and the room, the directional actions oppose each other without any inferential link from entry to exit, creating a contradiction rooted in incompatible spatial transitions. These examples underscore how non-entailment stems from failures in semantic alignment, such as opposition or incomplete information, without requiring external knowledge.

Challenges

Linguistic Ambiguity

Linguistic ambiguity poses significant challenges to textual entailment (TE) by introducing multiple possible interpretations of text, which can lead to inconsistent or uncertain judgments about whether one text entails another. In TE, where the task requires determining if the meaning of a hypothesis can be inferred from a premise, ambiguities at various linguistic levels disrupt the reliable mapping of semantic content, often resulting in false positives or negatives in automated systems. These issues highlight the need for robust semantic representations that account for interpretive variability without relying on external context. Lexical ambiguity occurs when a word or phrase has multiple senses, making it difficult to establish a direct entailment relation between premise and hypothesis. For example, consider a premise stating "John walked along the bank" and a hypothesis "John was near a financial institution"; the entailment fails if "bank" refers to a riverbank rather than a place for deposits, illustrating how polysemy can invalidate assumed inferences. This type of ambiguity is prevalent in natural language inference tasks, where shallow lexical overlaps fail to capture sense distinctions, leading to errors in early benchmarks. Seminal work on logical inference for TE emphasized the role of word sense disambiguation in improving accuracy, as unresolved lexical variants propagate uncertainty through the entailment pipeline. Syntactic ambiguity arises from multiple possible parse structures for a sentence, altering the relational dependencies and thus the entailed meanings. A classic case is the sentence "I saw the man with the telescope," which can be parsed as either the speaker using a telescope to see the man or the man holding the telescope, potentially changing whether it entails "The man used optical equipment." Such parsing indeterminacies complicate TE by requiring systems to evaluate all viable syntactic interpretations, as single parses may overlook valid inferences or introduce spurious ones. Research on syntax-aware models for natural language inference has shown that handling these ambiguities through multi-parse evaluation can boost performance, though parser errors exacerbate the problem in real-world applications. Scope ambiguity involves the interaction of quantifiers, modals, or negation, where the order of operators affects entailment direction. For instance, "Every student read some book" may or may not entail "Some book was read by every student," depending on whether universal or existential scope dominates, leading to non-monotonic inferences that defy simple textual alignment. This ambiguity challenges TE systems by necessitating deep semantic scoping, as surface-level comparisons ignore operator precedence. Studies on semantic role labeling in TE have noted that scope resolution is critical for accurate predicate-argument inference, with unresolved cases contributing to systematic failures in datasets involving quantified expressions. The impact of linguistic ambiguities on TE was recognized early in the field's formalization, particularly during the inaugural PASCAL Recognizing Textual Entailment (RTE-1) Challenge in 2005, which highlighted inference challenges stemming from natural language variability in its benchmark dataset creation. This challenge established TE as a task sensitive to such linguistic phenomena, influencing subsequent evaluations to incorporate ambiguity as a core difficulty.

Contextual and World Knowledge Issues

Textual entailment often hinges on resolving coreferences, where pronouns or noun phrases refer to entities introduced earlier in the discourse, requiring contextual linking to establish inference. For instance, in the pair where the text states "John entered the room. He sat down," the hypothesis "John sat down" entails only if "he" is resolved to refer to John; failure to do so can lead to incorrect non-entailment judgments.^[21] Studies emphasize coreference resolution as a prerequisite for accurate entailment, particularly in longer texts, though empirical evaluations show mixed impacts on overall performance due to resolution errors.^[22] Temporal and causal inferences further complicate entailment by necessitating world knowledge about event sequences and effects beyond explicit textual cues. Consider the text "It rained heavily last night" and hypothesis "The ground is wet today," where entailment relies on commonsense understanding that rain typically causes ground wetness, absent direct mention. Similarly, temporal aspects like tense and duration introduce challenges; for example, "Jane has arrived in London" entails "Jane is in London now" due to the present perfect's implication of ongoing relevance, whereas "Jane arrived in London" does not, as past events may no longer hold.^[23] Causation adds layers, as in preconditions where an action's completion infers membership or state change, such as "Once welcomed, they belong to the organization."^[24] Cultural or domain-specific knowledge gaps exacerbate these issues, as entailment judgments vary based on shared assumptions not universal across contexts. In scenarios requiring situational awareness, such as interpreting "A half-hour drive is near" as entailing proximity in a suburban U.S. context but not in dense urban settings, models falter without cultural priors.^[24] Domain expertise, like geographical facts (e.g., "Paris is in France" entailing European location), or professional norms, further demands external knowledge, leading to failures when systems lack such embeddings.^[24] Quantification of these challenges reveals their prevalence; analyses of RTE datasets indicate that 16.5% of entailment problems involve geographical world knowledge, 8.7% functionality, and 2.1% cultural/situational assumptions, with causal and precondition types contributing to broader world knowledge dependencies estimated at 20-30% of cases.^[24] The FraCaS test suite highlights this through sections on causation, temporal inferences, and world knowledge, where problems like nationality-based generalizations or event ordering (e.g., "Smith left after Jones left; Jones left after Anderson left" entailing "Smith left after Anderson left") expose systematic failures in capturing non-linguistic inferences.^[25] Such benchmarks underscore the importance of integrating contextual and commonsense knowledge to address these persistent challenges. In recent years, as of 2024-2025, additional challenges have emerged in RTE and NLI, particularly with large language models. These include dataset artifacts like word overlap leading to spurious correlations, inconsistencies in transitive entailment predictions, and difficulties in multilingual and low-resource settings. For instance, models often exploit superficial cues rather than deep semantics, resulting in poor generalization, as highlighted in evaluations of transformer-based systems.^[26] Efforts to mitigate these involve adversarial datasets and self-consistency checks to better capture true inferential capabilities.^[27]

Approaches

Traditional Methods

Traditional methods for detecting textual entailment rely on rule-driven and knowledge-intensive techniques that emphasize interpretability and do not require large-scale training data, focusing instead on linguistic structures and semantic resources to determine if a hypothesis can be inferred from a text.^[28] These approaches emerged prominently in the early RTE challenges from 2005 to 2010, where systems often achieved baseline performance through shallow comparisons and hand-engineered rules.^[7] Lexical alignment methods measure surface-level overlap between the text and hypothesis to gauge entailment, typically using metrics like word overlap or edit distance to identify shared terms while accounting for synonyms and semantic relations.^[29] For instance, directed lexical similarity calculates the proportion of hypothesis words that match or are semantically related to text words, often enhanced by resources like WordNet to incorporate hypernym-hyponym relations and gloss overlaps via extensions of the Lesk algorithm for word sense disambiguation.^[30] These techniques provide a simple baseline, detecting entailment when overlap exceeds a threshold, but they struggle with paraphrasing and complex inferences.^[28] Syntactic parsing approaches represent sentences as dependency or constituent trees and compute structural similarities to assess entailment, capturing relational alignments beyond mere word matches.^[28] A key method employs tree edit distance, which quantifies the minimum cost of transforming the text's parse tree into the hypothesis's via operations like insertion, deletion, and substitution, with costs modulated by lexical similarity from resources such as IDF weights or dependency-based thesauri.^[31] For example, in the 2005 PASCAL RTE challenge, this approach on dependency trees yielded competitive results by identifying low-cost edits as evidence of entailment.^[31] Such methods highlight structural consistency but require accurate parsing and may falter on syntactic variations.^[32] Knowledge-based inference leverages ontologies and semantic frames to verify consistency between text and hypothesis, drawing on external world knowledge for deeper semantic matching.^[28] Systems using FrameNet map predicates and arguments to semantic frames (e.g., "COMMERCE_GOODS_TRANSFER" for buying/selling scenarios), then compare frame-role alignments to detect entailment through overlap in evoked structures.^[33] Early implementations, such as those in the 2006 PASCAL RTE workshop, integrated FrameNet with syntactic parses to normalize expressions and check for subsumption, improving handling of implicit relations.^[33] Similarly, broader ontologies like Cyc enable rule application over logical forms, though their use in RTE has been more exploratory for consistency checks.^[34] Rule-based systems employ hand-crafted patterns and inference rules to transform texts or match specific entailment patterns, often targeting domain-specific or syntactic phenomena.^[35] For example, dependency-based rules derived from resources like DIRT (e.g., "receive obj award" ≈ "award obj") are applied to tree skeletons—simplified subtrees from overlapping nodes—to propagate entailments along paths.^[35] These were refined in RTE systems from 2005–2010, achieving high precision on covered cases by manually curating lexical and syntactic transformations, though coverage remained limited to explicit rules.^[36]

Machine Learning Methods

Machine learning methods for textual entailment represent a shift toward data-driven inference, employing statistical models to learn entailment patterns from annotated pairs or large corpora, often outperforming rigid rule-based systems on RTE challenge benchmarks. These techniques, prominent from the mid-2000s to mid-2010s, emphasized hand-crafted features and classical classifiers to address the binary decision of whether a text entails a hypothesis, achieving accuracies typically ranging from 55% to 70% on datasets like those from the PASCAL RTE challenges. By focusing on lexical overlap, structural alignment, and shallow semantics, they provided scalable solutions for practical NLP tasks while highlighting the need for richer representations. Feature engineering formed the foundation of these methods, transforming text-hypothesis pairs into numerical vectors suitable for classifiers. Bag-of-words (BoW) representations encoded sentences as multisets of words, capturing basic lexical presence without order, while term frequency-inverse document frequency (TF-IDF) weighted terms by their specificity across a corpus to prioritize discriminative vocabulary. Alignment features extended this by quantifying matches between hypothesis elements and text constituents, such as word overlaps or dependency links, often computed via similarity metrics like Dice coefficients. These features were commonly input to support vector machines (SVMs) or Naive Bayes classifiers; for example, SVMs with radial basis kernels excelled at separating entailment from non-entailment classes using high-dimensional feature spaces. In a seminal 2006 study, MacCartney and Manning extracted alignment-based features (e.g., coverage and monotonicity scores) alongside BoW and fed them into a logistic regression classifier, attaining 62.5% accuracy on preliminary RTE data and establishing alignment as a key predictor of valid entailments. Similarly, a 2007 RTE-3 system integrated string kernel similarities with SVMs, leveraging TF-IDF for weighting to achieve 68.2% accuracy on the test set. Naive Bayes variants, treating entailment as a probabilistic generative process, were applied in early supervised setups for their efficiency on sparse BoW features.^[37]^[38] Supervised models trained directly on RTE datasets, which comprised binary-labeled pairs (entailment or not) drawn from diverse sources like news articles and question-answering corpora, totaling around 800-1,000 examples per challenge from 2005 to 2010. These datasets enabled end-to-end learning of classifiers using objectives like hinge loss for SVMs or cross-entropy for logistic regression, optimizing for the entailment probability given feature vectors. Cross-entropy loss facilitated probabilistic outputs, allowing thresholding for binary decisions and integration with ensemble methods. Systems trained on RTE-1 through RTE-5 data demonstrated that supervised approaches scaled with feature richness, with top performers combining lexical and syntactic cues to reach 65-70% accuracy, though generalization remained challenged by dataset sparsity. The inaugural PASCAL RTE challenge overview by Dagan et al. (2006) reported supervised classifiers averaging 58% accuracy across participants, underscoring their edge over unsupervised baselines. Unsupervised alignment methods discovered entailment patterns from unlabeled corpora, bypassing annotation costs by iteratively refining mappings between text fragments. The Expectation-Maximization (EM) algorithm was particularly useful here, modeling latent alignments as hidden variables to maximize the likelihood of observed co-occurrences indicative of entailment. In the E-step, EM computes posterior probabilities for potential alignments (e.g., word substitutions implying hyponymy), and in the M-step, updates parameters like substitution probabilities from a large corpus such as Wikipedia or news archives. This enabled extraction of entailment rules, such as "X causes Y" entailing "Y occurs," with precision up to 80% for high-confidence patterns. A 2013 unsupervised framework by Szpektor et al. applied EM-like iterative alignment on web text to acquire paraphrase and entailment relations. Such techniques were vital for bootstrapping knowledge in resource-poor settings.^[39] Hybrid approaches merged machine learning with shallow semantic tools to incorporate syntactic structure, enhancing feature expressiveness beyond pure lexical methods. A prominent example combined classifiers with Combinatory Categorial Grammar (CCG) supertagging, which assigns words supertags—rich categories encoding argument structure—to derive partial parses for alignment and compositionality checks. During 2010-2015, CCG supertagging advanced RTE by providing syntactic proofs of entailment, such as type-raising for monotonicity preservation. These hybrids exemplified the synergy of statistical learning and formal grammars, achieving up to 72% accuracy in ensemble configurations while remaining interpretable.

Deep Learning Methods

Deep learning methods for textual entailment (TE) emerged prominently after 2015, leveraging neural architectures to learn distributed representations of text that capture semantic relationships between premise and hypothesis pairs. These approaches shifted from hand-crafted features to end-to-end trainable models, enabling better handling of lexical and syntactic variations through representation learning. Early advancements focused on sentence encoders that produce fixed-length embeddings for similarity computation, evolving into more sophisticated transformer-based systems that incorporate contextual attention. Sentence encoders, often built using Siamese networks, represent a foundational deep learning technique for TE by encoding premise and hypothesis sentences into shared embedding spaces for comparison. For instance, the InferSent model employs a Siamese LSTM architecture trained on natural language inference data to generate universal sentence embeddings, achieving strong performance on TE tasks by focusing on directional semantic relations. Similarly, convolutional neural networks (CNNs) in Siamese setups extract n-gram features for similarity scoring, as seen in models that combine CNNs with max-pooling to align sentence pairs. A notable extension is the Enhanced Sequential Inference Model (ESIM), which integrates LSTM-based encoding with multi-hop attention and soft alignment to refine raw representations, improving inference over sequential dependencies in TE pairs.^[40] Transformer-based models marked a significant leap starting in 2018, utilizing self-attention mechanisms to model bidirectional context and long-range dependencies in text. BERT, pre-trained on masked language modeling and next-sentence prediction, when fine-tuned on NLI datasets, achieves state-of-the-art TE results by classifying entailment through contextualized embeddings derived from transformer layers. This fine-tuning process adapts the model's attention heads to detect subtle inference patterns, outperforming prior recurrent models on benchmarks like SNLI and MNLI. Subsequent pre-trained language models built on the transformer paradigm have further advanced TE through optimized training and architectural refinements. RoBERTa, which removes next-sentence prediction and uses dynamic masking during pre-training, enhances TE performance via robust fine-tuning on NLI tasks, yielding higher accuracy in capturing nuanced entailments. DeBERTa introduces disentangled attention to separately model content and position, leading to superior TE results by better disambiguating relative positions in sentence pairs. Adaptations of these models also support prompt-based inference, where TE is reformulated as a masked prediction task to leverage zero-shot or few-shot capabilities without full fine-tuning. Recent trends up to 2025 emphasize extensions to multimodal TE and efficiency enhancements. Multimodal TE incorporates visual elements alongside text, as in visual entailment tasks where models like fine-tuned LLaMA 3.2 Vision assess inference between image-caption pairs and hypotheses, probing vision-language alignment. For efficiency, knowledge distillation techniques compress large models; DistilBERT, a distilled version of BERT, retains over 97% of its TE performance while reducing parameters by 40%, facilitating deployment in resource-constrained settings. These developments underscore the progression toward scalable, cross-modal inference systems.

Applications

Natural Language Inference

Natural Language Inference (NLI) serves as a generalized framework in natural language processing for determining the semantic relationship between a premise and a hypothesis, typically classifying it into three categories: entailment (the premise supports the hypothesis), contradiction (the premise opposes the hypothesis), or neutral (the relationship is undetermined).^[41] Textual entailment (TE), in contrast, represents a subset of NLI specifically focused on binary hypothesis testing, where the task is to assess whether the premise entails the hypothesis without considering contradictions or neutral cases.^[42] This distinction allows TE to emphasize directional inference in targeted applications, while NLI provides a broader evaluation of inferential capabilities.^[43] In dialogue systems, TE plays a key role in detecting implied meanings within conversations by evaluating whether a response logically follows from the preceding context, thereby ensuring coherence and relevance.^[44] For instance, by treating the conversation history as the premise and a generated response as the hypothesis, TE-based metrics can verify consistency and identify subtle implications that maintain natural flow, aiding in more robust response generation.^[44] This approach enables scalable, interpretable assessments that approximate human judgments of dialogue quality without exhaustive manual annotation.^[44] TE integrates with advanced reasoning mechanisms in large language models (LLMs) through techniques like chain-of-thought (CoT) prompting, where step-wise inference relies on successive entailment relations to build complex logical chains.^[45] In CoT, LLMs generate intermediate reasoning steps that implicitly test entailment between sequential thoughts, enhancing performance on multi-step tasks that require inferential alignment.^[45] This reliance on TE principles allows models to decompose problems into verifiable entailment steps, improving accuracy in reasoning-heavy applications. A notable case study is the GLUE benchmark, introduced in 2018, which incorporates TE as a core component in its NLI subtasks to evaluate model generalization across natural language understanding challenges.^[41] Subtasks like RTE directly test binary textual entailment on curated premise-hypothesis pairs, while others such as Multi-Genre NLI (MNLI) extend to three-way classification, highlighting TE's foundational role in broader inference evaluation.^[41] These subtasks demonstrated TE's impact by revealing gaps in early models' ability to handle diverse inferential scenarios, spurring advancements in NLU systems.^[41]

Question Answering

Textual entailment plays a crucial role in question answering (QA) systems by validating candidate answers against supporting passages, particularly in extractive tasks like those in the SQuAD dataset. In this setup, a hypothesis (H) derived from the candidate answer is checked for entailment by the text (T) from the passage, determining if the answer is supported or unanswerable. For instance, systems employ an answer verifier module, often based on models like BERT, to classify the legitimacy of extracted spans post-prediction, improving handling of unanswerable questions in SQuAD 2.0.^[46] This approach reduces false positives by filtering answers not entailed by the context, enhancing overall accuracy in reading comprehension.^[47] In generative QA, textual entailment aids in recognizing and filtering hallucinations—unsupported or fabricated claims in model outputs—by treating the generated answer as a hypothesis and the retrieved evidence as the premise. Natural language inference models, such as fine-tuned RoBERTa-large, classify the relationship as entailment (supported), contradiction (intrinsically false), or neutral (extrinsically unverifiable), enabling the detection of factual inconsistencies.^[48] This method has demonstrated superior performance, achieving an F1 score of 0.81 on hallucination detection benchmarks like XSumFaith++, outperforming prior systems by up to 12%.^[48] Hybrid QA systems integrate textual entailment with retrieval mechanisms, such as dense passage retrieval (DPR), to rerank candidate passages based on entailment scores. For example, queries are reformulated into existential claims (e.g., "There exists a human who stepped on the moon" for "Who first stepped on the moon?"), and passages are scored for whether they entail the claim, refining retrieval relevance beyond lexical matching.^[49] This entailment tuning boosts metrics like mean reciprocal rank (MRR) by 1-3% on datasets including Natural Questions (NQ).^[49] In multi-hop QA benchmarks like HotpotQA, incorporating entailment for evidence extraction and reranking has led to F1 score improvements of 5-10% in joint answer and supporting fact prediction from 2018 to 2023 evaluations, with models like Query Focused Extractor (QFE) raising evidence F1 from 37.7 to 44.4 in full-wiki settings.^[50]

Information Extraction and Summarization

Textual entailment plays a key role in relation extraction by reformulating the task as determining whether a hypothesis describing a specific relation between entities is entailed by the input text. For instance, given a text stating "Alice founded the company in 2010," a system can check if the hypothesis "Alice works for the company" is entailed to infer an employment relation. This approach leverages verbalizations of relations—simple templates like "X [relation] Y"—combined with pretrained entailment models, enabling zero-shot performance of 63% F1 on benchmarks like TACRED and 69% F1 in few-shot settings with only 16 examples per relation.^[51] Such methods reduce reliance on annotated data and outperform traditional supervised systems in low-resource scenarios by effectively discriminating between relation types and identifying non-relations. In abstractive summarization, textual entailment ensures factual consistency by verifying that generated summary sentences are entailed by the source document, mitigating hallucinations or unsupported claims. One prominent technique employs reinforcement learning where an entailment model provides feedback rewards during training, optimizing summaries for faithfulness while balancing salience and conciseness; this yields significant improvements in human-evaluated faithfulness scores on datasets like XSum and CNN/Daily Mail.^[52] In domain-specific applications, such as legal rulings, entailment modules assess multiple candidate summaries derived from different text "views" (e.g., full vs. segmented documents), selecting those with high fidelity to the source and boosting ROUGE scores across metrics by filtering unfaithful outputs.^[53] Recent extensions as of 2024 include applications in legal textual entailment with large language models for improved robustness.^[54] For coreference in information extraction, textual entailment facilitates merging entities across sentences by checking entailment relations between mentions, such as verifying if "the president" entails "Joe Biden" in context to link them as the same referent. This integration addresses gaps in inference where coreference resolution impacts 44% of entailment pairs, enabling substitution or merging transformations that enhance extraction accuracy; for example, in RTE-5 datasets, 73% of discourse references involve coreference, which, when resolved via entailment checks, reduces knowledge dependencies and improves overall system performance in tasks like event argument extraction.^[55] Advancements in multi-document summarization leverage textual entailment to reduce redundancy by identifying entailed content across documents, a challenge addressed in frameworks like the PASCAL RTE challenge and DUC evaluations from the 2000s onward. Systems compute entailment scores between sentences to detect paraphrases and subsumptions, omitting redundant material while preserving coherence; for instance, an extractive method using TE relations and sentence compression via knapsack optimization achieves up to 5% higher ROUGE F-measure on DUC datasets by prioritizing non-overlapping, salient content.^[56] These techniques, evolving through DUC's question-directed summarization tasks, enable scalable handling of document clusters.

Datasets and Evaluation

Key Datasets

The Recognizing Textual Entailment (RTE) challenges from 2005 to 2011 produced a series of benchmark datasets that established the foundational evaluation framework for textual entailment systems. The first three annual datasets, labeled RTE-1 through RTE-3, were organized under the PASCAL Network of Excellence, while RTE-4 through RTE-7 were held as part of the Text Analysis Conference (TAC). Each dataset consisted of approximately 800 to 1,000 human-annotated sentence pairs drawn from diverse sources such as news articles and encyclopedic texts, with binary labels indicating entailment or non-entailment in RTE-1 to RTE-3 and three-way labels (entailment, contradiction, unknown) in RTE-4 to RTE-7. These datasets emphasized manual annotation by linguists to ensure high-quality judgments, focusing on real-world inference tasks without additional context, and they influenced subsequent NLI benchmarks by prioritizing concise premise-hypothesis pairs.^[5]^[57] The Stanford Natural Language Inference (SNLI) corpus, introduced in 2015, marked a significant scale-up in dataset size for textual entailment research, comprising 570,000 English sentence pairs crowdsourced from image captions via Amazon Mechanical Turk workers. Each pair includes a premise derived from a caption, a hypothesis generated by the annotator, and one of three labels—entailment, contradiction, or neutral—achieving inter-annotator agreement of around 81% for the three-way task. This dataset's construction process involved workers writing hypotheses conditioned on premises to simulate natural inference scenarios, making it widely used for training supervised models due to its balanced distribution and focus on commonsense reasoning.^[16]^[58] Building on SNLI, the Multi-Genre Natural Language Inference (MultiNLI) dataset, released in 2017, expanded coverage to 433,000 sentence pairs across ten diverse genres, including fiction, telephone conversations, news, and academic texts, to address domain-specific inference challenges. Like SNLI, it employs three-way labeling through crowdsourcing, but incorporates matched and mismatched evaluation splits to test generalization beyond the training distribution, with annotators providing hypotheses based on premises from varied sources. MultiNLI's genre diversity revealed performance gaps in models trained solely on image-caption data, promoting more robust entailment systems.^[59]^[60] More recent advancements have introduced specialized datasets to probe limitations in existing NLI models. The Adversarial NLI (ANLI) benchmark, launched in 2020, features over 100,000 sentence pairs collected through an iterative human-model collaboration process across three rounds of increasing difficulty, where humans craft challenging examples to fool state-of-the-art models, resulting in three-way labels that highlight adversarial robustness issues. ANLI's construction emphasizes deception and complex reasoning, often involving world knowledge or subtle linguistic traps, and has become a key resource for evaluating model brittleness.^[61]^[62] ChaosNLI, also from 2020, augments subsets of SNLI and MultiNLI development sets with 100 annotations per example, totaling 464,500 labels, to stress-test annotation reliability and reveal human disagreement in entailment judgments. By crowdsourcing multiple perspectives on the same pairs, it exposes label noise and ambiguity in standard datasets, such as cases where neutral labels split between entailment and contradiction, aiding in the development of uncertainty-aware inference models.^[63] The WANLI dataset, introduced in 2022, comprises 107,885 NLI examples generated via a hybrid worker-AI pipeline, where large language models like GPT-3 produce initial premise-hypothesis pairs from seed data, followed by human revision and three-way labeling to incorporate weak supervision signals. This collaborative approach yields diverse, high-quality instances that capture nuanced entailment patterns, including those from AI-generated perturbations, and supports scalable dataset creation for improving model generalization.^[64]^[65]

Evaluation Metrics

The evaluation of textual entailment (TE) systems primarily relies on classification metrics adapted to the task's binary or three-way setups, where pairs of text are labeled as entailing, contradicting, or neutral. Accuracy measures the proportion of correctly classified pairs and serves as the standard metric for balanced datasets in three-way TE, as it directly reflects overall performance in distinguishing semantic relations.^[16] For instance, in the SNLI dataset, models are evaluated using three-way accuracy on held-out test sets to assess generalization.^[16] F1-score, the harmonic mean of precision and recall, complements accuracy by balancing the trade-off between false positives and false negatives, particularly useful when label distributions vary slightly across classes. In multi-class TE tasks, F1-scores are typically computed with macro-averaging (unweighted average across classes) to treat all labels equally, or micro-averaging (weighted by class support) to account for dataset balance; macro-F1 is preferred for three-way setups to avoid bias toward majority classes like neutral.^[66] These averaging methods ensure robust assessment, as seen in benchmarks where macro-F1 highlights deficiencies in minority classes such as entailment or contradiction. TE datasets often exhibit class imbalance, with neutral or non-entailment labels outnumbering others, necessitating metrics that penalize poor performance on rare classes. Precision (ratio of true positives to predicted positives) and recall (ratio of true positives to actual positives) are computed per class and averaged, providing granular insights into model reliability for entailment detection.^[67] The Matthews Correlation Coefficient (MCC), ranging from -1 to +1, offers a balanced measure for imbalanced scenarios by incorporating all confusion matrix elements, making it suitable for TE where binary approximations (entailment vs. not) amplify skew effects.^[68] MCC is particularly valuable in TE evaluations to ensure models do not overfit to dominant non-entailment cases.^[69] Challenge sets like FraCaS test linguistic competence through hand-crafted problems focusing on phenomena such as quantifiers and anaphora, reporting entailment success rates as the percentage of correctly resolved yes/no/unknown inferences.^[70] Early systems achieved around 70-80% success on FraCaS sections, while natural logic-based models reached up to 82% precision on known entailments, underscoring gaps in compositional reasoning.^[13] These rates reveal model limitations beyond large-scale data, emphasizing targeted linguistic coverage.^[2] Performance trends show rapid gains with deep learning: BERT achieved approximately 85% accuracy on SNLI in 2018, establishing a baseline for transformer-based TE. By 2024, advanced models and ensembles pushed state-of-the-art accuracies beyond 94% on SNLI, reflecting saturation on crowd-sourced benchmarks but persistent challenges on adversarial or linguistic test suites.^[71] These improvements, often exceeding 90% on datasets like SNLI, highlight the impact of scaled pretraining, though evaluations stress the need for diverse metrics to capture nuanced inference.^[43]

References

[1]
The PASCAL Recognising Textual Entailment Challenge
The PASCAL Recognising Textual Entailment Challenge. 179. Table 1. Examples of Text-Hypothesis pairs. ID. TEXT. HYPOTHESIS. TASK VALUE. 568 Norway's most famous ...<|control11|><|separator|>
[2]
[PDF] A survey on Recognizing Textual Entailment as an NLP Evaluation
Recognizing Textual Entailment (RTE) was proposed as a unified evaluation framework to compare semantic understanding of differ- ent NLP systems.
[3]
The PASCAL Recognising Textual Entailment Challenge
The PASCAL Recognising Textual Entailment Challenge. Conference paper. pp 177–190; Cite this conference paper. Download book PDF.
[4]
TAC 2008 RTE Track Guidelines
Textual entailment recognition is the task of deciding, given a T-H pair, whether T entails H. The three-way RTE task is to decide whether: T entails H - in ...
[5]
Recognizing textual entailment: Rational, evaluation and approaches
Nov 17, 2009 · More concretely, the applied notion of textual entailment is defined as a directional relationship between pairs of text expressions, denoted by ...Missing: seminal | Show results with:seminal
[6]
Inherent Disagreements in Human Textual Inferences
Nov 1, 2019 · ... semantics” (Montague, 1970). Computational work on recognizing textual entailment ... pragmatics and world knowledge (e.g., inferences about ...
[7]
[PDF] Recognising Textual Entailment with Logical Inference
Recognising Textual Entailment with Logical Inference. Johan Bos. School of Informatics. University of Edinburgh. 2 Buccleuch Place. Edinburgh, EH8 9LW jbos@inf ...
[8]
https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00293/43531/Inherent-Disagreements-in-Human-Textual-Inferences
[9]
[PDF] A Probabilistic Lexical Approach to Textual Entailment - IJCAI
Abstract. The textual entailment problem is to determine if a given text entails a given hypothesis. This paper describes first a general generative ...Missing: formalization | Show results with:formalization
[10]
None
### Definition of Three-Way Classification for NLI
[11]
[PDF] Natural Logic for Textual Inference - Stanford NLP Group
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The. PASCAL Recognising Textual Entailment Challenge. In. Proc. of the PASCAL RTE Challenge Workshop.
[12]
[PDF] Chapter 26: Textual Entailment - NLPado - the Pado homepage
Aug 29, 2012 · The three-way decision can be seen as a refinement of the original binary decision, as shown by the curly brackets at the top, by splitting the ...
[13]
[PDF] Defining Textual Entailment - CORE
We have seen that D11 predicts failures of monotonicity in cases involving inconsistent texts. However, there are other cases involving inconsistent texts where ...Missing: defeasibility | Show results with:defeasibility
[14]
[PDF] A large annotated corpus for learning natural language inference
A man is driving down a lonely road. A soccer game with multiple males playing. entailment. E E E E E. Some men are playing a sport.
[15]
[PDF] Effectively Using Syntax for Recognizing False Entailment - Microsoft
Hypothesis: ''Hepburn, who won four Oscars...'' Text: ''Hepburn, a four-time Academy Award winner...'' Hepburn. Noun win. Verb. Tsub. Hepburn. Noun. String.<|control11|><|separator|>
[16]
[PDF] Syntactic/Semantic Structures for Textual Entailment Recognition
In this paper, we describe an approach based on off-the-shelf parsers and semantic re- sources for the Recognizing Textual Entail-.
[17]
[PDF] The PASCAL Recognising Textual Entailment Challenge - CiteSeerX
The PASCAL Recognising Textual Entailment Challenge. 179. Table 1. Examples of Text-Hypothesis pairs. ID. TEXT. HYPOTHESIS. TASK VALUE. 568 Norway's most famous ...
[18]
(PDF) The PASCAL recognising textual entailment challenge
Aug 7, 2025 · Conference PaperPDF Available. The PASCAL recognising textual entailment challenge. January 2005; Lecture Notes in Computer Science. DOI:10.1007 ...
[19]
[PDF] Recognizing Textual Entailment - umich.edu
Dagan, O. Glickman, and B. Magnini. The PASCAL Recognising Textual. Entailment Challenge. 3944, 2006. [14] H.T. Dang and K. Owczarzak. Overview of the tac ...
[20]
[PDF] Coreference Resolution: to What Extent Does it Help NLP ...
Jun 25, 2012 · In this study we seek to establish whether the employment of coreference resolution to NLP applications is beneficial. The investigation has ...
[21]
[PDF] Temporal and Aspectual Entailment - ACL Anthology
In this paper we propose a novel entailment dataset and analyse the ability of a range of recently proposed NLP models to perform inference on temporal ...
[22]
[PDF] Types of Common-Sense Knowledge Needed for Recognizing ...
We attempt to characterize the kinds of common-sense knowledge most often involved in recognizing textual entailments.
[23]
[PDF] The FraCaS test suite 1 Generalized quantifiers - CLASP
action and causation, world knowledge, interaction between plurality, genericity and temporal/aspectual phenomena etc. Some of the inferences are very basic ...
[24]
Recognizing textual entailment: A review of resources, approaches ...
The review aims to examine the current state of recognizing textual entailment (RTE) research and summarize the state-of-the-art methods.Missing: seminal | Show results with:seminal
[25]
[PDF] Recognizing Textual Entailment Using Lexical Similarity
Our method is based on calculating “directed” sentence simi- larity: checking the directed “semantic” word overlap between the text and the hy- pothesis. We use ...
[26]
[PDF] An approach using Named Entities for Recognizing Textual Entailment
Word sense disambiguation using the. Lesk algorithm [8], based on Wordnet definitions. 5. A semantic similarity matrix between words in T and H is defined ...
[27]
Tree edit distance for textual entailment - ResearchGate
This paper addresses Textual Entailment (i.e. recognizing that the meaning of a text entails the meaning of another text) using a Tree Edit Distance algorithm ...
[28]
[PDF] Tree Edit Models for Recognizing Textual Entailments, Paraphrases ...
A popular method for such tasks is Tree Edit Dis- tance (TED), which models sentence pairs by find- ing a low or minimal cost sequence of editing oper- ations ...
[29]
[PDF] Approaching Textual Entailment with LFG and FrameNet Frames
Approaching Textual Entailment with LFG and FrameNet Frames. Aljoscha Burchardt. Dept. of Computational Linguistics. Saarland University. Saarbrücken, Germany.
[30]
[PDF] Towards a Rewriting Framework for Textual Entailment
Jul 24, 2014 · The most basic criterion for success as far as a semantics of natural language sentences is concerned is that of textual entailment, which is of ...
[31]
[PDF] Inference Rules for Recognizing Textual Entailment - ACL Anthology
Given a premise p and a hypothesis h, the lexical-syntactic component marks all lexical noun alignments. For every pair of alignments, the paths between the two ...
[32]
(PDF) Inference Rules and their Application to Recognizing Textual ...
This paper addresses syntax-based paraphrasing methods for Recognizing Textual Entailment (RTE). In particular, we describe a dependency-based paraphrasing ...
[33]
[PDF] Learning to recognize features of valid textual entailments
J. Bos and K. Markert. 2005. Recognising textual entailment with logical inference. In EMNLP-05. I. Dagan, O.
[34]
[PDF] Learning Textual Entailment using SVMs and String Similarity ...
We present the system that we submitted to the 3rd Pascal Recognizing Textual Entail- ment Challenge. It uses four Support Vector. Machines, one for each ...
[35]
Unsupervised acquisition of entailment relations from the Web
Jul 30, 2013 · In this work, we adopt the Web as our textual resource, due to its huge size, its heterogeneous content, and the amount of redundant information ...Missing: maximization | Show results with:maximization
[36]
Refining Raw Sentence Representations for Textual Entailment ...
Jul 11, 2017 · In this paper we present the model used by the team Rivercorners for the 2017 RepEval shared task. First, our model separately encodes a pair of ...
[37]
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural ...
GLUE is a tool for evaluating and analyzing NLU models across diverse tasks. It is model-agnostic and incentivizes sharing knowledge.Missing: NLI | Show results with:NLI
[38]
Improving the Precision of Natural Textual Entailment Problem ...
We apply this method to a subset of the Recognizing Textual Entailment datasets. ... natural-language inference systems. Anthology ID: 2020.lrec-1.844 ...
[39]
A Study of the State of the Art Approaches and Datasets for ...
Oct 25, 2024 · Textual entailment defines a semantic relationship between a pair of sentences in terms of entailment, contradiction, and neutral relationships.
[40]
Evaluating Coherence in Dialogue Systems using Entailment
Evaluating Coherence in Dialogue Systems using Entailment. In Proceedings of the 2019 Conference of the North American Chapter of the Association for ...
[41]
Chain-of-Thought Prompting Elicits Reasoning in Large Language ...
Jan 28, 2022 · Experiments on three large language models show that chain of thought prompting improves performance on a range of arithmetic, commonsense, and ...
[42]
[PDF] Answer Validation for SQuAD 2.0 - Stanford University
We explore approaches to extractive reading comprehension with unanswerable questions, as defined in the SQuAD 2.0 dataset. In particular, we explore answer.
[43]
[PDF] Machine Reading Comprehension with Unanswerable Questions
After the answer is extracted, an answer verifier is used to compare the answer sentence with the question, so as to recognize local textual entailment that ...
[44]
[PDF] NLI to the Rescue: Mapping Entailment Classes to Hallucination ...
These three classes of textual entailment are mapped to in- trinsic, extrinsic, and non-hallucinated respec- tively. We fine-tune a RoBERTa-large model on NLI ...
[45]
[PDF] Improve Dense Passage Retrieval with Entailment Tuning
Nov 12, 2024 · Our experiments demonstrate the effectiveness of the entailment tuning method in dense passage retrieval tasks. In summary, our contribution is ...Missing: reranking | Show results with:reranking
[46]
[PDF] Multi-task Learning for Multi-hop QA with Evidence Extraction
Although designed for RC, it also achieves a state-of-the-art evidence extraction score on FEVER, which is a recognizing textual entailment task on a large ...
[47]
[PDF] The Third PASCAL Recognizing Textual Entailment Challenge
This paper presents the Third PASCAL. Recognising Textual Entailment Chal- lenge (RTE-3), providing an overview of the dataset creating methodology and the.
[48]
A large annotated corpus for learning natural language inference
Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings ...
[49]
A Broad-Coverage Challenge Corpus for Sentence Understanding ...
Apr 18, 2017 · This paper introduces the Multi-Genre Natural Language Inference (MultiNLI) corpus, a dataset designed for use in the development and evaluation of machine ...
[50]
A Broad-Coverage Challenge Corpus for Sentence Understanding ...
MultiNLI is a 433k example dataset for sentence understanding, with ten genres, designed for machine learning models and is more difficult than Stanford NLI.
[51]
Adversarial NLI: A New Benchmark for Natural Language ...
Abstract. We introduce a new large-scale NLI benchmark dataset, collected via an iterative, adversarial human-and-model-in-the-loop procedure.
[52]
[1910.14599] Adversarial NLI: A New Benchmark for Natural ... - arXiv
Oct 31, 2019 · We introduce a new large-scale NLI benchmark dataset, collected via an iterative, adversarial human-and-model-in-the-loop procedure.
[53]
What Can We Learn from Collective Human Opinions on Natural ...
Oct 7, 2020 · We collect ChaosNLI, a dataset with a total of 464,500 annotations to study Collective HumAn OpinionS in oft-used NLI evaluation sets. This ...
[54]
WANLI: Worker and AI Collaboration for Natural Language Inference ...
WANLI is a dataset created using worker and AI collaboration. AI generates examples, which are then revised and labeled by human crowdworkers.Missing: weakly supervised
[55]
WANLI: Worker and AI Collaboration for Natural Language Inference ...
Jan 16, 2022 · The resulting dataset, WANLI, consists of 107,885 NLI examples and presents unique empirical strengths over existing NLI datasets.Missing: weakly supervised
[56]
[PDF] GLUE: A MULTI-TASK BENCHMARK AND ANALYSIS PLATFORM ...
GLUE is a benchmark for evaluating NLU models across diverse tasks, including question answering and sentiment analysis, with an online platform for analysis.
[57]
Tour of Evaluation Metrics for Imbalanced Classification
May 1, 2021 · There are standard metrics that are widely used for evaluating classification predictive models, such as classification accuracy or classification error.Missing: entailment | Show results with:entailment
[58]
The advantages of the Matthews correlation coefficient (MCC) over ...
Jan 2, 2020 · In this article, we show how MCC produces a more informative and truthful score in evaluating binary classifications than accuracy and F 1 score.Missing: entailment | Show results with:entailment
[59]
On the Performance of Matthews Correlation Coefficient (MCC) for ...
The Matthews correlation coefficient (MCC) was used to measure the quality of the classifications generated by the models (Zhu, 2020 The null hypothesis is ...
[60]
[PDF] A Type-Theoretical system for the FraCaS test suite - ACL Anthology
The most well-known platforms are the FraCaS test suite (Cooper et al., 1996), the Pascal Recog- nizing Textual Entailment tasks (RTE) (Dagan et al., 2006), and ...
[61]
https://aclanthology.org/2020.acl-main.441/