Follow

Formal and High Level Audit Comparison

This document provides an overview of the key differences between Formal and High Level Audits provided by SSB BART Group (“SSB”).

Testing Approach

A Formal Audit uses SSB’s comprehensive testing methodology the Unified Audit Methodology. The testing methodology for Formal Audits focuses on the selection of a representative set of modules – in conformance with the Module List Creation Guide – and formally-scripted use cases – in conformance with the Use Case Creation Guide. The testing approach for the modules requires full testing of all relevant accessibility requirements across all modules and formal reporting of the use case testing results across all relevant assistive technologies.

In contrast, the testing approach for a High Level Audit focuses on the identification of unique violations found in the application. This approach is a “broad but shallow” or “quick fail” testing approach focused on identifying the same set of violations as the formal audit but without the population and percentage statistics for the issue provided by the formal audit. In lieu of formal use cases, SSB employs an ad-hoc combination of assistive technology testing that includes the leading AT products—JAWS screen reader, ZoomText screen magnifier, and Dragon Naturally Speaking speech input product—along with other manual methods including keyboard-only usage, testing in High Contrast mode, and similar methods. This testing approach allows for an accurate claim about the set of issues that exist in a system but no corollary claim relating to the frequency with which these issues occur.

The impact of these different testing approaches is that when delivering a Formal Audit SSB is able to certify a particular level of Section 508 compliance or WCAG conformance for the system given that the frequency of all violations is known.

Compliance Claims

The formal audit provides Independent Verification and Validation (IV&V) of the application and obligates SSB to work with our clients to defend any claims of compliance challenged by end customers. Most often this means working on behalf of the client directly with procuring US Federal Government agencies to ensure they clearly understand the claims and justifications for the claim of compliance within the audited system.

Pricing

By its nature the Formal Audit requires more extensive testing and auditing of applications pursuant to providing detailed metrics about the frequency and severity of issues within the user interface. This is in contrast to the “quick fail” approach of the High Level Audit were the testing effort is constrained by the amount of time allocated to the project.

For smaller applications the difference between these two testing approaches is minimal. As the size and complexity of the application increases the price difference between the two audits increases to a maximum point of around a sixty (60) percent price premium for a Formal Audit for complex, enterprise class applications. Ultimately there is no universally applicable rule of thumb for the pricing difference between a Formal and High Level Audit and the pricing difference must be determined on a case-by-case basis.

Deliverables

The written report for a Formal Audit consists of a general audit report accompanied by a series of appendix documents. This information is outlined below in the Formal Audit Deliverables portion of this page. In contrast the High Level Audit report is limited to approximately five pages, consisting of a 1-2 page summary of the accessibility and compliance issues followed by an illustration and discussion of the most significant problems.

Formal Audit appendix results are provided in both Word document format and via SSB’s Accessibility Management Platform (“AMP”). This provides a report format that can easily be escrowed and stored by the client should later regulatory audits or compliance challenges require that the particular key audit documents be produced. In contrast, High Level Audits are only delivered via AMP and Word documents are not provided by SSB.

Finally, at the conclusion of a Formal Audit SSB can create a third party certification of a level of compliance of the system. This certification is provided via a third party VPAT developed and signed by SSB and delivered on SSB corporate letterhead. This certification can be provided since the testing scope of a formal audit is guaranteed to be representative in nature as determined by an independent third party in conformance with the requirements defined in of the VPAT Creation Guide.

At the conclusion of a High Level Audit SSB can create an Accessibility Review Document that will discuss the compliance level of the application with each Section 508 paragraph or WCAG conformance requirement. The Accessibility Review Document constitutes SSB’s recommendation to the client regarding what types of issues should be put by the client into a VPAT document. It does not, however, constitute a binding claim of compliance on the part of SSB. Accessibility Review Documents are meant only to state that a particular set of issues exists at a given point in time – they make no claims as to the frequency or relative import of the issue in terms of the overall compliance of the application.

Selection Approach

Given that, what audit is the right audit for your organization? Generally speaking organizations that are early in the accessibility process should focus on High Level Audits. The high-level detail provided by the audit of the same name is more than sufficient to impart an understanding of the types of accessibility and compliance issues that exist in the application and determine the type of challenges to be addressed in future remediation or redesign activities. Further, the High Level Audit allows SSB to develop an Accessibility Review Document that reflects the general, starting level of Section 508 compliance or WCAG conformance and is often enough to ensure continued access to public sector markets.

Organizations that have a more highly developed level of accessibility or compliance or those that have specific regulatory conformance requirements – such as having faced a compliance compliant – are more likely served by a Formal Audit and third party VPAT. This ensures a reputable 3rd party has endorsed the claims of the organization provided in the VPAT. This provides government procurement officials a professional and well-written VPAT to assist in market research and prepare for any lack of accessibility or accommodations if the product is procured.

Deliverable Comparison

Formal Audit Deliverables

Audit – The Audit Report is the comprehensive report detailing the findings, violations, and compliance level of the tested application. It includes an Executive Summary, compliance metrics with the selected accessibility standards, product-specific examples of significant violations, and extensive technical documentation about every type of accessibility best practice that was violated. These examples and best practice descriptions include generic examples of compliant and non-compliant code, recommendations (including specific syntax), and the specific accessibility paragraph or checkpoint it falls under.

Appendix A – The Modules by Total Violation document lists the module name and the number of violations on each module.

Appendix B – The Modules Detail document depicts what violations are in each module. In short, this appendix is a detail of Appendix A. The details may include a short description, whether it is part of a violation pattern, and the line number of the rendered HTML page source.

Appendix C – The Violations by Priority appendix displays the severity, frequency, noticeability, and tractability (i.e. the typical degree of difficulty to fix) of each discovered violation. These factors are combined to establish a rough prioritization of the discovered violations. The violations are ranked on a scale from 1 (lowest) to 10 (highest), which offers a gauge of the development effort that may be required to remediate the problems.

Appendix D – The Use Case Results appendix documents the use cases that are meant to define the core functionality that is in use within the application and the results from the testing of these use cases. Accessibility problems that were encountered at any step of any use case are identified, along with any relevant AT-specific settings or notes. The results of each use case are ranked on a scale of 1 (lowest, meaning the use case was a total failure that could not be completed) to 5 (highest, completely successful). As with the manual testing, violation patterns are also identified, as well as AT-specific compatibility and configuration issues.

Appendix E – The Module List document lists and illustrates the modules that are tested. This ensures that the client’s developers and testers can cross-reference each module with the problems documented in Appendices A and B.

Appendix F – The Violation Patterns table describes accessibility problems that are found on many, most, or all modules. It is a convenient alternative to, and a subset of, the much fuller Appendix B, which also includes the results from automated testing and problems that are found only on individual pages. A significant amount of analysis goes into the identification of these Patterns, which are typically used as the primary examples pictured and discussed in the main Audit Report.

AMP Licenses Subscription – SSB’s AMP User Licenses provide “Accessibility-on-Demand” for the development teams in a self-serviced, self-paced easy to comprehend format which drastically reduces the cost of administering an accessibility program. AMP is a web-based platform for the implementation and management of accessibility across enterprise-class development organizations spanning multiple development platforms, deployment environments, and user interface patterns.

Voluntary Product Accessibility Template (VPAT) – The VPAT provides the compliance level with each Section 508 paragraph. In addition to a formal “grade”, it also describes specific accessibility features, documentation, and some types of accessibility problems.

High Level Audit Deliverables

Deliverables in Word Document Format

High Level Audit – SSB will deliver a High Level Audit report for the application. The report will begin with a 1-2 page Summary of the overall findings. Following the summary, and using a table format, examples of specific problems found in the application will be identified. This will include the page(s) where the problem was found, the accessibility Best Practice(s) that were violated, a screenshot and description of the problem, and the recommended changes.

Accessibility Review Document – The Accessibility Review Document provides the compliance level with each Section 508 paragraph. In addition to a formal “grade”, it also describes specific accessibility features, documentation, and some types of accessibility problems.

Module List – The Module List document lists and illustrates the modules that are tested. This ensures that the client’s developers and testers can cross-reference each module with the problems documented in the Modules by Total Violation and Modules Detail data from AMP.

Deliverables in AMP

Module List – This lists the Name and Location of each Module tested, providing a cross-reference to the screenshots in the Word-based Module List document.

Module Detail – This depicts what violations are in each module. The details include the accessibility Best Practice that was violated and a description of the specific problem that was encountered. Depending on the type of audit and type of product, it may also include additional results from automated testing tools, whether the problem was part of a Pattern or was Global in scope, and the line number of the rendered HTML page source. In AMP, the user can select whether to display these Module-specific details for a selected Module only or for all Modules.

Violations List – This lists the accessibility Best Practices that were found to have been violated during testing. It also identifies the “Media Type” to which each Best Practice belongs (e.g. Keyboard Accessibility, Forms), presented in a sortable list to group related requirements together.

Violations by Priority – This displays the Severity, Noticeability, and Tractability (i.e. the typical degree of difficulty to fix) of each accessibility Best Practice that was violated. These factors are combined to establish a rough prioritization of the discovered violations. The violations are normalized on a scale from 1 (lowest priority) to 10 (highest priority), which offers a gauge of the development effort that may be required to remediate the problems and the order in which they should be remediated.

Violations by Standard – For each accessibility Best Practice that was violated, this identifies which accessibility standard(s) that Best Practice is mapped to. Examples include Section 508’s Technical standards and its separate Functional Performance Criteria, and the Web Content Accessibility Guidelines for Level A and Level AA compliance. This can increase the understanding of the accessibility standards, and may also serve as a partial basis for prioritizing the remediation efforts.

Compliance Reports – For the accessibility standards against which the product was tested, these reports identify the specific paragraphs or checkpoints that relate to the violations that were found during the audit. This helps identify whether the problems found in the audit are more broad or narrow in nature.

AMP Licenses Subscription – SSB’s AMP User Licenses provide “Accessibility-on-Demand” for the development teams in a self-serviced, self-paced easy to comprehend format which drastically reduces the cost of administering an accessibility program. AMP is a web-based platform for the implementation and management of accessibility across enterprise-class development organizations spanning multiple development platforms, deployment environments, and user interface patterns.

Comparison Table

Comparison Table

Formal Audit

High Level audit

Testing Approach

Comprehensive

  • Representative modules
  • Use cases
  • Three assistive technologies

Identification of unique violations

  • broad but shallow
  • Ad hoc AT testing

Compliance Claims

IV&V

Non-IV&V accessibility review document for internal use

Pricing

Cost is greater for larger applications

Cost is similar to formal audit when application is small

Deliverables

  • General audit Report
  • Appendix documents
    • Lists all instances of violations
  • VPAT/Certification

5 pages

  • ½ page summary
  • Illustration and discussion of most significant issues
  • Accessibility Review Document

Deliverable Format

  • MS Word
  • AMP
  • MS Word
  • AMP

Selection Criteria

  • Require legal document
  • Invested significant resources into accessibliity
  • Early in process
  • Looking for a general measurement of accessibility

Deliverable details

  • Audit
    • Executive summary
    • Compliance chart
    • Example of significant violations
    • Best practice details
  • Appendix A – Modules by total violations
  • Appendix B – Module detail
  • Appendix C – Violations by Priority
  • Appendix D – Use case results
  • Appendix E – Module List
  • Appendix F – Violation Patterns
  • AMP License subscription
  • VPAT
  • High level Audit Document
    • ½ page summary
    • Illustration and discussion of most significant issues
  • Accessiblity Review Document (optional?)
  • Module List

AMP Deliverables

  • Module List
  • Module Details
  • Violation List
  • Violation by Priority
  • Violation by Standard
  • Compliance Reports
  • AMP License Subscription
  • Module List
  • Module Details
  • Violation List
  • Violation by Priority
  • Violation by Standard
  • Compliance Reports
  • AMP License Subscription
Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

Comments

Help Desk
www.ssbbartgroup.com | 800.889.9659
© 2005 - 2015 - SSB BART Group. All rights reserved.
Privacy | Security | Credits | License
Powered by Zendesk