A PHP Error was encountered

Severity: Warning

Message: file_get_contents(https://...@gmail.com&api_key=61f08fa0b96a73de8c900d749fcb997acc09): Failed to open stream: HTTP request failed! HTTP/1.1 429 Too Many Requests

Filename: helpers/my_audit_helper.php

Line Number: 143

Backtrace:

File: /var/www/html/application/helpers/my_audit_helper.php
Line: 143
Function: file_get_contents

File: /var/www/html/application/helpers/my_audit_helper.php
Line: 209
Function: simplexml_load_file_from_url

File: /var/www/html/application/helpers/my_audit_helper.php
Line: 3098
Function: getPubMedXML

File: /var/www/html/application/controllers/Detail.php
Line: 574
Function: pubMedSearch_Global

File: /var/www/html/application/controllers/Detail.php
Line: 488
Function: pubMedGetRelatedKeyword

File: /var/www/html/index.php
Line: 316
Function: require_once

A PHP Error was encountered

Severity: Warning

Message: Attempt to read property "Count" on bool

Filename: helpers/my_audit_helper.php

Line Number: 3100

Backtrace:

File: /var/www/html/application/helpers/my_audit_helper.php
Line: 3100
Function: _error_handler

File: /var/www/html/application/controllers/Detail.php
Line: 574
Function: pubMedSearch_Global

File: /var/www/html/application/controllers/Detail.php
Line: 488
Function: pubMedGetRelatedKeyword

File: /var/www/html/index.php
Line: 316
Function: require_once

Approaches to analyzing binary data for large-scale A/B testing. | LitMetric

Approaches to analyzing binary data for large-scale A/B testing.

Contemp Clin Trials Commun

Department of Biostatistics & Informatics, University of Colorado, United States.

Published: April 2023

AI Article Synopsis

  • A collaboration between industry and academia was formed to assess the appropriate statistical tests and study designs for A/B testing in larger-scale experiments, highlighting the prevalent use of the -test by the industry partner.
  • The study emphasizes the need to understand the impact of interim analysis on the -test’s effectiveness, especially since interim analyses often use only part of the sample size and can affect key properties like power and type I error rates.
  • Simulation studies compared the performance of the -test, Chi-squared test, and Chi-squared test with Yate's correction for binary outcomes, revealing that while the -test performed well, naïve interim monitoring without adjustments can significantly detract from study performance.

Article Abstract

An industry-academic collaboration was established to evaluate the choice of statistical test and study design for A/B testing in larger-scale industry experiments. Specifically, the standard approach at the industry partner was to apply a -test for all outcomes, both continuous and binary, and to apply naïve interim monitoring strategies that had not evaluated the potential implications on operating characteristics such as power and type I error rates. Although many papers have summarized the robustness of the -test, its performance for the A/B testing context of large-scale proportion data, with or without interim analyses, is needed. Investigating the effect of interim analyses on the robustness of the -test is important, because interim analyses rely on a fraction of the total sample size and one should ensure that desired properties are maintained when a -test is implemented not just at the end of the study, but for making interim decisions. Through simulation studies, the performance of the -test, Chi-squared test, and Chi-squared test with Yate's correction when applied to binary outcomes data is evaluated. Further, interim monitoring through a naïve approach with no correction for multiple testing versus the O'Brien-Fleming boundary are considered in designs that allow early termination for futility, difference, or both. Results indicate that the -test achieves similar power and type I error rates for binary outcomes data with the large sample sizes used in industrial A/B tests with and without interim monitoring, and naïve interim monitoring without corrections leads to poorly performing studies.

Download full-text PDF

Source
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC9982610PMC
http://dx.doi.org/10.1016/j.conctc.2023.101091DOI Listing

Publication Analysis

Top Keywords

interim monitoring
16
a/b testing
12
interim analyses
12
interim
8
naïve interim
8
power type
8
type error
8
error rates
8
robustness -test
8
chi-squared test
8

Similar Publications

Want AI Summaries of new PubMed Abstracts delivered to your In-box?

Enter search terms and have AI summaries delivered each week - change queries or unsubscribe any time!

A PHP Error was encountered

Severity: Notice

Message: fwrite(): Write of 34 bytes failed with errno=28 No space left on device

Filename: drivers/Session_files_driver.php

Line Number: 272

Backtrace:

A PHP Error was encountered

Severity: Warning

Message: session_write_close(): Failed to write session data using user defined save handler. (session.save_path: /var/lib/php/sessions)

Filename: Unknown

Line Number: 0

Backtrace: