Governmentwide Findings: Key Compliance Outcomes

Key Compliance Outcomes

It has been 37 years since the Rehabilitation Act Amendments added Section 508 and 25 years since the expanded Rehabilitation Act Amendments of 1998, and the federal government is overarchingly still procuring, developing, maintaining, and using non-conformant ICT. The government as a whole is not meeting the minimum standard or legal obligation to provide equal access to all members of the public and federal employees with disabilities.

Successful Section 508 programs have policies, processes, and procedures in place that ensure conformance with ICT requirements. The following sections analyze both conformance outcomes and maturity inputs to holistically evaluate Section 508 compliance and better understand where deficiencies exist and why.

To gauge if testing methodologies and respondent internal processes translated into Section 508 conformant ICT, the Assessment included outcome-based questions related to conformance of digital content, including web, electronic documents, videos, hardware, software, mobile apps and kiosks. On average, governmentwide conformance is Low (1.79 on a 5-point scale).

The Assessment asked respondents to report on Section 508 web page testing performed throughout the reporting period. The Assessment criteria asked seven questions on public internet web pages and seven questions on internal intranet web pages to determine how many of each the respondent owned and operated, how many were tested for Section 508 conformance, how many web pages fully conformed, and what testing methodologies were used. The data showed:

Figure 1. Percentage of respondents submitting conformance testing results for the top intranet and internet web pages, electronic documents, and videos (Q78 to Q81)

Respondents reported relying heavily on only automated accessibility testing tools to perform conformance testing on public web pages, while utilizing a more manual approach on intranet sites. This was expected as automated testing tools historically have run into barriers in testing intranet content due to firewalls or technology stack.

With heavy reliance on solely automated testing for public web pages, there is risk of inflation of conformance scores due to the inability of automated tools to comprehensively test for all Section 508 requirements. Automated testing omits of over half of the Section 508 requirements, and fewer requirements tested means fewer chances for failure. However, reporting entities’ data nevertheless indicate similar levels of conformance with intranet pages, which tend to incorporate more manual inspection along with automated testing. Our speculated reasons for the similar conformance level patterns include:

Figure 2. Percentage of fully conformant ICT for all top intranet web pages, internet web pages, electronic documents, and videos tested (Q78 to Q81)

• Possible inflation of conformance from automated testing suggesting public web pages may be less conformant than intranet pages.

• Configuration of automated tools may flag defects that aren’t Section 508 failures (e.g. best practices), leading to an increase in failures reported.

• Implementation of a hybrid- testing methodology strategically utilizes automated and manual tools, thus percentages by type of testing tool is not statistically significant.

• Intranet pages may contain more non-web content, such as PDFs, that were not evaluated, which translates to less opportunities for failures.

• Other reasons we cannot determine without additional queries or investigation. GSA will need additional information to understand the alignment in conformance levels and trade-offs from automated, hybrid, and manual testing methods.

Within the reporting period, respondents were asked to identify and test the top viewed internal intranet pages, publicly available internet pages, publicly viewed electronic documents and videos (public or internal) for full conformance. As Figure 1 depicts, about half of all respondents noted resources were unavailable to test the ICT in order to provide results for these questions. As Figure 2 shows, the results for top viewed ICT displayed low conformance with less than 30% of the most viewed ICT tested reported as fully conformant.

Compliance of Top Viewed Intranet and Internet Web Pages 

Compliance of Top Viewed Electronic Documents and Videos

Compliance of Other ICT

The criteria included questions on if specific federal organization’s ICT, including public or internal use, fully conformed to Section 508 standards. Although 100% conformance is the standard, the highest percent conformant ICT was enterprise email applications reported to be at about 62% conformant across government. The least conformant widely-used ICT included in the Assessment was W2s at just over 31%. Out of the 97 reporting entities who reported having custom developed desktop software or native mobile applications used by the public, only 23% reported fully conformant ICT. Additionally, out of the 24 reporting entities who reported having kiosks used by the public, only 46% reported fully conformant kiosks. Figure 3 shows the percentage of additional ICT that fully conform to Section 508 standards and Table 1 details responses for conformance of ICT from the Assessment criteria.

Figure 3. Percentage of reporting entities noting full conformance by products used internally and by the public (Q82 to Q85, Q89 to Q92, respectively)

In addition, the Assessment asked about fully conformant ICT provided as a shared service, reimbursable service, or e-gov line of business to other agencies or public entities. Out of the 64 respondents who provided such ICT services:

• 23 respondents (36%) reported that all ICT services fully conform to Section 508 standards

• 25 respondents (39%) reported only some ICT services fully conform to Section 508 standards

• 3 respondents (5%) reported that none of the ICT services fully conform to Section 508 standards, and 

• 13 respondents (20%) reported they did not know if the ICT services fully conform. 

Although only about 25% of respondents provide a shared service, the downstream effects of ensuring these products and services are fully conformant could be exponential given the number of employees and members of the public these services could impact. Ensuring accessible shared service products should be a high priority item for all 64 agencies providing such services. 

Respondents overarchingly reported Section 508 defects in the ICT procured, developed, maintained, or used by the public and federal employees. With overall low compliance, the majority of ICT, including web pages, software, kiosks, electronic documents, and videos fail to meet Section 508 standards. Every instance of non-compliance with Section 508 captured in this data is an indicator that federal employees with disabilities and members of the public are impeded or excluded in their interactions with federal ICT. Findings further detail the factors within reporting entities that are impacting low Section 508 compliance.

Correlation between Maturity and Conformance

To comprehensively and quantitatively assess Section 508 compliance across the federal government, we generated two indices based solely on responses to the Assessment criteria. The business function maturity index (referred to as “maturity” or “m-index”) quantifies reporting entity responses to criteria across nine dimensions with equal weighting of each criteria.¹⁰ The second index, operational conformance (referred to as “conformance” or “c-index”) quantified 16 specific criteria in the Conformance dimension to assess how well reporting entities performed with respect to Section 508 implementation.¹¹ The indices are on a 5-point scale (from zero to five). These indices are fully described in Methods, Descriptive Analysis. 

As Figure 4 below depicts, reporting entities fall across the scale from zero to five on both axes, with a heavy concentration of reporting entities located in the bottom and left of the graph. The red trend line shows an upward trend, indicating that generally, as a reporting entity’s maturity increased, their conformance also tended to increase. Thus, reporting entities with greater capacity to execute activities related to accessibility, such as testing or training, tended to have more conformant ICT products.

Figure 4. Percentage of respondents submitting conformance testing results for the top intranet and internet web pages, electronic documents, and videos (Q78 to Q81)

To conduct a more detailed analysis, we created brackets around the conformance and maturity indices. The brackets range from Very Low (index values ranging from 0-1) to Very High (index values ranging from 4-5) and are reflected in the gridlines as shown in Figure 4. The full breakdown of bracket value ranges is shown in Table 2.

The “heat map” table below depicts the total number of reporting entities that fell into each category. The cells become progressively darker blue as more reporting entities fall into the combination of maturity and conformance index brackets. 

The combination of the maturity index and the conformance index can then be assessed as a reporting entity’s overall performance (referred to throughout this section as “category”). In the findings below, these categories are titled by the maturity index first (referred to throughout this section as “maturity”) and the conformance index (referred to throughout this section as “conformance”) second. For instance, for reporting entities that fell into the High-Low Overall performance category, their maturity fell into the “High” bracket, while their conformance fell into the “Low” bracket. As you can see, there are five such reporting entities that fell into this category. The categories with the highest number of reporting entities are Low-Low (39 reporting entities) and Moderate-Low (37 reporting entities). The categories with no reporting entities are Very Low-Very High, Very Low-Moderate, and Very High-Very High.

Reviewed/Updated: December 2023