Multiple Testing Methods for AV Devices
In the lab, we test devices in two methods: #1 as recommended by the vendor and #2 bench testing. This blog discusses both methods and why they are important in the overall assessment of devices.
Naturally, we test devices as intended by the manufacturer. The reasons for this are probably more obvious than bench testing. With each device, we like to perform an assessment on not just how the device performs, but we look at the installation and setup processes as well. We feel that having an understanding of the installation, in addition to management, end user workflows and performance metrics, help to paint an overall picture on devices. Just plugging in a device, using it for a couple of days and reporting on the subjective performance won't generally help end users or vendors who read our reviews.
Bench testing is utilized for several purposes, primarily to verify the correctness of the device's specifications and to use in benchmarking performance. In the lab, it is important because we’re able to accurately control the conditions to run exercises in a simulated environment. This helps with comparing one device against another or tracking changes with firmware updates.
Here are a couple of examples of our bench testing process:
Video Devices
If we simply install and point a camera into our office and assess the experience, we may not notice issues that will be apparent for other users in different rooms. Cameras are set up in a fashion where we can mathematically assess the device for determining the usable field of view or geometric issues like lens distortion. We are also able to perform a video quality analysis, color replication tests and look at how a device handles different lighting conditions.
Audio Devices
Bench testing audio devices allows us to maintain a controlled environment to objectively test metrics like frequency response, noise cancellation / suppression, max SPL and microphone capture quality. Performing audio testing in a "normal" conference room may put some devices at a disadvantage depending on the environment. If we test an audio device in a room with a loud HVAC system, for example, and report negatively on it due to the environment, we may be altering an opinion of someone looking to install the device in a normally quiet room. This why we deploy devices into a controlled environment so we can flesh out any audio anomalies that may or may not be present in a "normal" situation.
Read more about audio device testing here.
One can only learn so much with one type of test or the other. If we were testing only for our internal use, we very well may just plug the device in and start using it. However, the results we aim to gather aren't just for internal use and it would be a disservice to our clients if we treat our subjective experience as being completely accurate.