arrow_back_ios

Main Menu

See All Acoustic End-of-Line Test Systems See All DAQ and instruments See All Electroacoustics See All Software See All Transducers See All Vibration Testing Equipment See All Academy See All Resource Center See All Applications See All Industries See All Insights See All Services See All Support See All Our Business See All Our History See All Our Sustainability Commitment See All Global Presence
arrow_back_ios

Main Menu

See All Actuators See All Combustion Engines See All Durability See All eDrive See All Production Testing Sensors See All Transmission & Gearboxes See All Turbo Charger See All DAQ Systems See All High Precision and Calibration Systems See All Industrial electronics See All Power Analyser See All S&V Hand-held devices See All S&V Signal conditioner See All Test Solutions See All DAQ Software See All Drivers & API See All nCode - Durability and Fatigue Analysis See All ReliaSoft - Reliability Analysis and Management See All Test Data Management See All Utility See All Vibration Control See All Acoustic See All Current / voltage See All Displacement See All Load Cells See All Pressure See All Strain Gauges See All Torque See All Vibration See All Accessories for Vibration Testing Equipment See All Power Amplifiers See All LDS Shaker Systems See All Vibration Controllers See All Training Courses See All Acoustics See All Asset & Process Monitoring See All Custom Sensors See All Data Acquisition & Analysis See All Durability & Fatigue See All Electric Power Testing See All NVH See All Reliability See All Smart Sensors See All Vibration See All Weighing See All Automotive & Ground Transportation See All Calibration See All Installation, Maintenance & Repair See All Support Brüel & Kjær See All Release Notes See All Compliance See All Our People
arrow_back_ios

Main Menu

See All CANHEAD See All GenHS See All LAN-XI See All MGCplus See All Optical Interrogators See All QuantumX See All SomatXR See All BK Connect / Pulse See All API See All Microphone Sets See All Microphone Cartridges See All Acoustic Calibrators See All Special Microphones See All Microphone Pre-amplifiers See All Sound Sources See All Accessories for acoustic transducers See All Experimental testing See All Transducer Manufacturing (OEM) See All Non-rotating (calibration) See All Rotating See All CCLD (IEPE) accelerometers See All Charge Accelerometers See All Electroacoustics See All Noise Source Identification See All Environmental Noise See All Sound Power and Sound Pressure See All Noise Certification See All Industrial Process Control See All Structural Health Monitoring See All Electrical Devices Testing See All Electrical Systems Testing See All Grid Testing See All High-Voltage Testing See All Vibration Testing with Electrodynamic Shakers See All Structural Dynamics See All Machine Analysis and Diagnostics See All Dynamic Weighing See All Calibration Services for Transducers See All Calibration Services for Handheld Instruments See All Calibration Services for Instruments & DAQ See All On-Site Calibration See All Resources See All Software License Management

Statement: Standard ReliaSoft Software Validation & QA Practices

 

ReliaSoft often receives questions/requests from customers/third parties regarding software validation (usually related to FDA requirements). This document was created to address such requests.


 

General Statement

All of ReliaSoft's shrink-wrap products are thoroughly tested and completely validated before commercial release. Our strict validation process, along with our extensive documentation on both the use of the software and the underlying mathematics, ensures (with a very high probability) that all results provided by ReliaSoft applications are valid and correct.

 

ReliaSoft's validation and quality assurance (QA) procedures were developed independently of the Food and Drug Administration (FDA) validation requirements. Although we believe that ReliaSoft's validation testing is of a much broader nature and far more intensive than the FDA requirements, it is of course the responsibility of each organization to determine whether the use of ReliaSoft's software will be in compliance with any regulatory guidelines they may be subject to. We expect that this document will provide the majority of the information about ReliaSoft's procedures that will be required for you to conduct your evaluation.

 

End User Independent Validation

With all standard shrink-wrap products, ReliaSoft provides extensive documentation in the form of user's guides, help files and theoretical textbooks. The purpose of the documentation is to present the methodologies used (equations and formulations), provide examples and familiarize the end user with the underlying theory.

 

An end user can do the following:

 

  • Independently confirm that the application behaves as described in these guides.
  • Use the provided textbooks to perform additional independent validation of the software results, either by using the formulations given or by using the many examples and solutions provided within these textbooks (many of which are from published papers).

Additionally, comparisons between ReliaSoft's numerical results and published results (by other authors) are also provided with the printed documentation.

 

Some of these textbooks are posted on ReliaSoft's public wiki. Specifically, the following references are available:

 

The following sections present an overview of the requirements for software development at ReliaSoft.

 


Theoretical Formulation and Theory Development

All theory and formulations created or implemented by ReliaSoft must be theoretically sound and correct, accepted by academia and industry experts, and also must be thoroughly validated before becoming part of a standard software product (SSP).

 

Theory and formulations that are not created by ReliaSoft must meet the following standards:

 

  • Must have been published in reputable academic journals or textbooks and must have had sufficient time for peer review.
  • Must have been re-derived and validated by ReliaSoft scientists to assure correctness of the original work.
  • All assumptions made during the formulation must be scrutinized, well documented and understood.
  • Must be reviewed and approved by ReliaSoft’s theoretical review board and by at least one other independent academic expert on the subject.

Theory and formulations that are developed by a ReliaSoft scientist must meet the following standards:

 

  • Must document all assumptions made during the formulation.
  • Must be reviewed and approved by ReliaSoft’s theoretical review board and at least two other independent academic experts on the subject.
  • If release of the formulation is not considered to be a competitive advantage, then it should be published in reputable academic journals for peer review. If a competitive advantage exists, then it should be published after the product release either in journals or in ReliaSoft textbooks.

Code Development

All products and product components developed by ReliaSoft must adhere to the industry-wide accepted standards for developing Windows-based object-oriented applications, including industry standards for graphical user interfaces (GUIs). Multiple authors, including Microsoft and Microsoft Press publications, offer a wide range of articles and books that detail accepted practices. Additionally, such code development must adhere to the guidelines for software development per ReliaSoft's internal document "Coding Practices and Procedures for SSP Software," and to current practices, methods and documents posted in the Development section of ReliaSoft’s intranet.

Microsoft guidelines for developing Windows applications (as detailed in several documents, such as "Application Specification for Microsoft® Windows® 2000") are adhered to. Deviation, if any, from these guidelines must be approved by ReliaSoft’s technical review board and reasons for the deviation must be documented.

Periodic design reviews are conducted with members of the code development team, technical writers and quality assurance. In addition, theoretical and technical review boards are conducted.


Source Code Management

The guidelines implemented by ReliaSoft for source code management are as follows:

  • All source code, including modifications to code, is tracked and controlled through the use of third party source control software.  Under no circumstances may code not tracked through source control  be introduced into the application.
  • All product builds are versioned using a standard Major.Minor.Revision numbering scheme (e.g., 3.1.2).
  • All compiled code (components) used in ReliaSoft applications are controlled through ReliaSoft’s configuration and version control processes. All components are capable (through an event or property) of identifying their version and build to a host application.

 

QA and Testing Procedures Overview

ReliaSoft has always adhered to the highest quality and reliability standards for all of its software products and services. The quality assurance (QA) and testing procedures for all ReliaSoft products, including custom software, are based on a scaled agile framework which is facilitated using CA Agile Central. The scaled agile process involves detailed and comprehensive testing efforts by multiple individuals and teams over time specific iterations; thorough documentation of all issues identified during each iteration; and independent validation of all methods, theory and calculated results.

Unit Testing

These tests include low level testing at 'unit' and 'integration' levels. They generally test libraries etc. and are therefore performed by developers. Their purpose is to ensure that the software produces the correct results (e.g. 2+2 = 4). They must be completed before System Tests can be concluded. This unit testing ensures that components and models function as intended and also ensures robustness. This testing is the responsibility of the individual developers and it involves a test-analyze-and-fix (TAAF) process.

Integration Testing

Integration testing involves testers looking for bugs within the relationships and interfaces between a pair of components or group of components. (An example is how Weibull++ is integrated with ALTA.) Integration testing is not required if the application is made up of individual utilities that do not share data or invoke one another. However, if the software uses API, shares data or passes control from one component to another, then integration testing becomes an important method to verify that the components are working together properly. Integration testing requires that the tester has a firm understanding of how the components are intended to work together.

System Testing

Once all modules/components have been tested, compiled and linked without any errors or warnings, wider testing is implemented on the complete application or system. Formal logs of system testing activities are kept in our Bug Logging and Tracking system - CA Agile Central. In general, any and all issues found are corrected as they are identified and testing resumes immediately with the corrected version to assure that the fix did not create any regression issues. Special emphasis is given to the component or issue that was corrected. This is repeated throughout each iteration of the development phase.

Installation and Environmental Testing

The last phase of testing is installation and configuration testing. This is done in-house on multiple test systems including most versions of Windows® (e.g., Windows 7 & 8.1, Windows 10). When appropriate, feedback is also obtained from outside testers. In the case of software that will reach foreign markets, foreign OS (operating system) testing is also performed (e.g., Chinese Windows, Korean Windows).

Results Validation Testing

In parallel with the software functionality testing, multiple engineers, scientists, and statisticians working independently perform testing and validation of all calculated results. Validated results are thoroughly documented and many of the examples are then illustrated in textbooks that are released with the software. Any issues found are corrected, re-tested and re-validated using multiple data scenarios.

Release Decision

When ReliaSoft releases a product, it has been thoroughly tested and ReliaSoft is highly confident of its quality and reliability. After all issues have been resolved and validated, a release decision is made. This decision is based on the number and frequency of issues found. Before software release can commence, the Release Manager must ensure that all test documentation has been signed-off (and approval for release has been given) by the Test Manager.

Concluding Remarks

In the case of custom systems, we expect the client to test the product to assure that the system is functioning as intended in the client environment. For systems that incorporate a non-ReliaSoft database to support functionality and calculations, the client is responsible for taking steps to maintain the integrity of the data stored in the database. An overview of the testing procedures employed by ReliaSoft is presented in the next table.


Software Testing (QA) Summary Table

 

Testing Phase Testing Type Responsibility Testing Details

Development Phase

Unit Testing

Integration Testing

Development

QA

Theoretical

Perform ongoing unit testing during development. This will involve testing individual components as they are developed. At this stage, individual components will be tested for correctness of calculations as well as interaction with other components.

General Testing Phase

Process Testing

Integration Testing

System Testing

QA

Development

Theoretical

Documentation

Validate process results for correctness. Track the occurrence and resolution of all anomalous issues. Test the performance of components when they are fully integrated.

Installation Testing Phase

Installation and Environmental Testing

QA

Development

Test the system internally/externally under various installation environments. For custom systems, work with the client to test the system at the deployment location.

Calculation Validation Phase

Calculation Validation

Theoretical

Validate a representative sample of system calculations by performing identical calculations independently and comparing the results.

Release Phase

Final Testing

QA

Development

Theoretical

Documentation

Unit, process, integration and functionality, usability and deployment testing.

Testing in each phase is cyclical utilizing a test-analyze-and-fix (TAAF) process. All issues after the initial development phase are communicated, documented and resolved utilizing CA Agile Central.


Issue Tracking and Resolution Process

CA Agile Central tracks all issues encountered for each product from the development phase and throughout its life cycle. This integrated system forms the basis of our QA procedures. All issues are logged into CA Agile Central as incidents and are tracked from creation to resolution. Each defect is logged for the appropriate team to address. Defects can include suggestions from customers, employees, bugs found during testing, etc. Once the defect has been addressed by development, it must be tested by the QA group to ensure that the resolution employed correctly addresses the issue and does not introduce any additional regression issues. CA Agile Central facilitates teamwork between developers, theoretical experts, technical writers, QA and management to create a reliable and robust application.

The next figure shows one of the defect reporting interfaces within CA Agile Central.

Defect Reporting Interface within CA Agile Central


Software Deployment

A software application is not deemed ready for deployment until the release candidate has been accepted by ReliaSoft's QA group. The QA group must be satisfied with the current state of the application based on the results of the testing.

Once a product is ready for deployment:

  • A master is built for subsequent duplication.
  • The source control system is updated to include the final configuration for the product and source code for the build and is then frozen.
  • Backups of the source code, build components and machine configuration used for the release build are made, and component versions are documented (including non-ReliaSoft components). Storage and location of redundant backups is done according to ReliaSoft’s Disaster Recovery Protocols.
  • The machine used for the build is then isolated and frozen and may not be used for any purpose other than subsequent builds (revisions) of the same version of the application.