Severity: Warning
Message: file_get_contents(https://...@gmail.com&api_key=61f08fa0b96a73de8c900d749fcb997acc09): Failed to open stream: HTTP request failed! HTTP/1.1 429 Too Many Requests
Filename: helpers/my_audit_helper.php
Line Number: 143
Backtrace:
File: /var/www/html/application/helpers/my_audit_helper.php
Line: 143
Function: file_get_contents
File: /var/www/html/application/helpers/my_audit_helper.php
Line: 209
Function: simplexml_load_file_from_url
File: /var/www/html/application/helpers/my_audit_helper.php
Line: 3098
Function: getPubMedXML
File: /var/www/html/application/controllers/Detail.php
Line: 574
Function: pubMedSearch_Global
File: /var/www/html/application/controllers/Detail.php
Line: 488
Function: pubMedGetRelatedKeyword
File: /var/www/html/index.php
Line: 316
Function: require_once
Severity: Warning
Message: Attempt to read property "Count" on bool
Filename: helpers/my_audit_helper.php
Line Number: 3100
Backtrace:
File: /var/www/html/application/helpers/my_audit_helper.php
Line: 3100
Function: _error_handler
File: /var/www/html/application/controllers/Detail.php
Line: 574
Function: pubMedSearch_Global
File: /var/www/html/application/controllers/Detail.php
Line: 488
Function: pubMedGetRelatedKeyword
File: /var/www/html/index.php
Line: 316
Function: require_once
The rise of the Internet of Things (IoT) and Industry 2.0 has spurred a growing need for extensive data computing, and Spark emerged as a promising Big Data platform, attributed to its distributed in-memory computing capabilities. However, practical heavy workloads often lead to memory bottleneck issues in the Spark platform. This results in resilient distributed datasets (RDD) eviction and, in extreme cases, violent memory contentions, causing a significant degradation in Spark computational efficiency. To tackle this issue, we propose an adaptive memory reservation (AMR) strategy in this article, specifically designed for heavy workloads in the Spark environment. Specifically, we model optimal task parallelism by minimizing the disparity between the number of tasks completed without blocking and the number completed in regular rounds. Optimal memory for task parallelism is determined to establish an efficient execution memory space for computational parallelism. Subsequently, through adaptive execution memory reservation and dynamic adjustments, such as compression or expansion based on task progress, the strategy ensures dynamic task parallelism in the Spark parallel computing process. Considering the cost of RDD cache location and real-time memory space usage, we select suitable storage locations for different RDD types to alleviate execution memory pressure. Finally, we conduct extensive laboratory experiments to validate the effectiveness of AMR. Results indicate that, compared to existing memory management solutions, AMR reduces the execution time by approximately 46.8%.
Download full-text PDF |
Source |
---|---|
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC11639302 | PMC |
http://dx.doi.org/10.7717/peerj-cs.2460 | DOI Listing |
Enter search terms and have AI summaries delivered each week - change queries or unsubscribe any time!