Visual Perception

Color Perception ¶

Informative · Practical

We can establish an understanding on color perception through studying its physical and perceptual meaning. This way, we can gather more information on its relation to technologies and devices including displays, cameras, sensors, communication devices, computers and computer graphics.

Color, a perceptual phenomenon, can be explained in a physical and visual perception capacity. In the physical sense, color is a quantity representing the response to wavelength of light. The human visual system can perceive colors within a certain range of the electromagnetic spectrum, from around 400 nanometers to 700 nanometers. For greater details on the electromagnetic spectrum and concept of wavelength, we recommend revisiting Light, Computation, and Computational Light section of our course. For the human visual system, color is a perceptual phenomenon created by our brain when specific wavelengths of light are emitted, reflected, or transmitted by objects. The perception of color originates from the absorption of light by photoreceptors in the eye. These photoreceptor cells convert the light into electrical signals to be interpreted by the brain¹. Here, you can see a close-up photograph of these photoreceptor cells found in the eye.

Image title — Micrograph of retinal photoreceptor cells, with rods and cones highlighted in green (top row). Image courtesy of NIH, licensed under CC PDM 1.0. View source.

The photoreceptors, where color perception originates, are called rods and cones². Here, we provide a sketch showing where these rods and cones are located inside the eye. By closely observing this sketch, you can also understand the basic average geometry of a human eye and its parts helping to redirect light from an actual scene towards retinal cells.

Rods, which are relatively more common in the periphery, help people see in low-light (scotopic) conditions. The current understanding is that the roids can only interpret in a greyscale manner. Cones, which are more dense in the fovea, are pivotal in color perception in brighter (photopic) environments. We highlight the distribution of these photoreceptor cells, rods and cones with changing eccentricities in the eye. Here, the word eccentricities refer to angles with respect to our gaze direction. For instance, if a person is not directly gazing at a location or an object in a given scene, that location or the object would have some angle to the gaze of that person. Thus, there would be at some angles, some eccentricity between the gaze of that person and that location or object in that scene.

In the above sketch, we introduced various parts on the retina, including fovea, parafovea, perifovea and peripheral vision. Note that these regions are defined by the angles, in other words eccentricities. Please also note that there is a region on our retina where there are no rods and cones are available. This region could be found in every human eye and known as the blind spot on the retina. Visual acuity and contrast sensitivity decreases progressively across these identified regions, with the most detail in the fovea, diminishing toward the periphery.

The cones are categorized into three types based on their sensitivity to specific wavelengths of light, corresponding to long (L), medium (M), and short (S) wavelength cones. These three types of cones³ allow us to better understand the trichromatic theory⁴, suggesting that human color perception stems from combining stimulations of the LMS cones. Scientists have tried to graphically represent how sensitive each type of cone is to different wavelengths of light, which is known as the spectral sensitivity function⁵. In practical applications such as display technologies and computational imaging, the LMS cone response can be replicated with the following formula:

\[ LMS = \sum_{i=1}^{3} \text{RGB}_i \cdot \text{Spectrum}_i \cdot \text{Sensitivity}_i \]

Where:

\(RGB_i\): The i-th color channel (Red, Green, or Blue) of the image.
\(Spectrum_i\): The spectral distribution of the corresponding primary
\(Sensitivity_i\): The sensitivity of the L, M, and S cones for each wavelength.

This formula gives us more insight on how we percieve colors from different digital and physical inputs.

Looking for more reading to expand your understanding on human visual system?

We recommend these papers, which we find it insightful:
- B. P. Schmidt, M. Neitz, and J. Neitz, "Neurobiological hypothesis of color appearance and hue perception," J. Opt. Soc. Am. A 31(4), A195–207 (2014)
- Biomimetic Eye Modeling & Deep Neuromuscular Oculomotor Control

The story of color perception only deepens with the concept of color opponency⁶. This theory reveals that our perception of color is not just a matter of additive combinations of primary colors but also involves a dynamic interplay of opposing colors: red versus green, blue versus yellow. This phenomenon is rooted in the neural pathways of the eye and brain, where certain cells are excited or inhibited by specific wavelengths, enhancing our ability to distinguish between subtle shades and contrasts. Below is a mathematical formulation for the color opponency model proposed by Schmidt et al.³

\[\begin{bmatrix} I_{(M+S)-L} \\ I_{(L+S)-M} \\ I_{(L+M+S)} \end{bmatrix} = \begin{bmatrix} (I_M + I_S) - I_L \\ (I_L + I_S) - I_M \\ (I_L, I_M, I_S) \end{bmatrix}\]

In this equation, \(I_L\), \(I_M\), and \(I_S\) represent the intensities received by the long, medium, and short cone cells, respectively. Opponent signals are represented by the differences between combinations of cone responses.

We could exercise on our understanding of trichromat sensation with LMS cones and the concept of color opponency by vising the functions available in our toolkit, odak. The utility function we will review is odak.learn.perception.display_color_hvs.primarier_to_lms() from odak.learn.perception. Let us use this test to demonstrate how we can obtain LMS sensation from the color primaries of an image.

test_learn_perception_display_color_hvs.py

import odak # (1)
import torch
import sys
from odak.learn.perception.color_conversion import display_color_hvs


def test(
         device = torch.device('cpu'),
         output_directory = 'test_output'
        ):
    odak.tools.check_directory(output_directory)
    torch.manual_seed(0)

    image_rgb = odak.learn.tools.load_image(
                                            'test/data/fruit_lady.png',
                                            normalizeby = 255.,
                                            torch_style = True
                                           ).unsqueeze(0).to(device) # (2)

    the_number_of_primaries = 3
    multi_spectrum = torch.zeros(
                                 the_number_of_primaries,
                                 301
                                ) # (3)
    multi_spectrum[0, 200:250] = 1.
    multi_spectrum[1, 130:145] = 1.
    multi_spectrum[2, 0:50] = 1.

    display_color = display_color_hvs(
                                      read_spectrum ='tensor',
                                      primaries_spectrum=multi_spectrum,
                                      device = device
                                     ) # (4)

    image_lms_second_stage = display_color.primaries_to_lms(image_rgb) # (5)
    image_lms_third_stage = display_color.second_to_third_stage(image_lms_second_stage) # (6)


    odak.learn.tools.save_image(
                                '{}/image_rgb.png'.format(output_directory),
                                image_rgb,
                                cmin = 0.,
                                cmax = image_rgb.max()
                               )


    odak.learn.tools.save_image(
                                '{}/image_lms_second_stage.png'.format(output_directory),
                                image_lms_second_stage,
                                cmin = 0.,
                                cmax = image_lms_second_stage.max()
                               )

    odak.learn.tools.save_image(
                                '{}/image_lms_third_stage.png'.format(output_directory),
                                image_lms_third_stage,
                                cmin = 0.,
                                cmax = image_lms_third_stage.max()
                               )


    image_rgb_noisy = image_rgb * 0.6 + torch.rand_like(image_rgb) * 0.4 # (7)
    loss_lms = display_color(image_rgb, image_rgb_noisy) # (8)
    print('The third stage LMS sensation difference between two input images is {:.10f}.'.format(loss_lms))
    assert True == True

if __name__ == "__main__":
    sys.exit(test())

Adding odak to our imports.
Loading an existing RGB image.
Defining the spectrum of our primaries of our imaginary display. These values are defined for each primary from 400 nm to 701 nm (301 elements).
Obtain LMS cone sensations for our primaries of our imaginary display.
Calculating the LMS sensation of our input RGB image at the second stage of color perception using our imaginary display.
Calculating the LMS sensation of our input RGB image at the third stage of color perception using our imaginary display.
We are intentionally adding some noise to the input RGB image here.
We calculate the perceptual loss/difference between the two input image (original RGB vs noisy RGB).
This a visualization of a randomly generated image and its' LMS cone sensation.

Our code above saves three different images. The very first saved image is the ground truth RGB image as depicted below.

We process this ground truth image by accounting human visual system's cones and display backlight spectrum. This way, we can calculate how our ground truth image is sensed by LMS cones. The LMS sensation, in other words, ground truth image in LMS color space is provided below. Note that each color here represent a different cone, for instance, green color channel of below image represents medium cone and blue channel represents short cones. Keep in mind that LMS sensation is also known as trichromat sensation in the literature.

Earlier, we discussed about the color oppenency theory. We follow this theory, and with our code, we utilize trichromat values to derive an image representation below.

Lab work: Observing the effect of display spectrum

We introduce our unit test, test_learn_perception_display_color_hvs.py, to provide an example on how to convert an RGB image to trichromat values as sensed by the retinal cone cells. Note that during this exercise, we define a variable named multi_spectrum to represent the wavelengths of our each color primary. These wavelength values are stored in a vector for each primary and provided the intensity of a corresponding wavelength from 400 nm to 701 nm. The trichromat values that we have derived from our original ground truth RGB image is highly correlated with these spectrum values. To observe this correlation, we encourage you to find spectrums of actual display types (e.g., OLEDs, LEDs, LCDs) and map the multi_spectrum to their spectrum to observe the difference in color perception in various display technologies. In addition, we believe that this will also give you a practical sandbox to examine the correlation between wavelengths and trichromat values.

Closing remarks¶

As we dive deeper into light and color perception, it becomes evident that the task of replicating the natural spectrum of colors in technology is still an evolving journey. This exploration into the nature of color sets the stage for a deeper examination of how our biological systems perceive color and how technology strives to emulate that perception.

Consider revisiting this chapter

Remember that you can always revisit this chapter as you progress with the course and as you need it. This chapter is vital for establishing a means to complete your assignments and could help formulate a suitable base to collaborate and work with my research group in the future or other experts in the field.

Reminder

We host a Slack group with more than 300 members. This Slack group focuses on the topics of rendering, perception, displays and cameras. The group is open to public and you can become a member by following this link. Readers can get in-touch with the wider community using this public group.

Jeremy Freeman and Eero P Simoncelli. Metamers of the ventral stream. Nature Neuroscience, 14:1195–1201, 2011. doi:10.1038/nn.2889. ↩
Trevor D Lamb. Why rods and cones? Eye, 30:179–185, 2015. doi:10.1038/eye.2015.236. ↩
Brian P Schmidt, Maureen Neitz, and Jay Neitz. Neurobiological hypothesis of color appearance and hue perception. Journal of the Optical Society of America A, 31(4):A195–A207, 2014. doi:10.1364/JOSAA.31.00A195. ↩↩
H. V. Walters. Some experiments on the trichromatic theory of vision. Proceedings of the Royal Society of London. Series B - Biological Sciences, 131:27–50, 1942. doi:10.1098/rspb.1942.0016. ↩
Andrew Stockman and Lindsay T Sharpe. The spectral sensitivities of the middle- and long-wavelength-sensitive cones derived from measurements in observers of known genotype. Vision Research, 40:1711–1737, 2000. doi:10.1016/S0042-6989(00)00021-3. ↩
Steven K Shevell and Paul R Martin. Color opponency: tutorial. Journal of the Optical Society of America A, 34(8):1099–1110, 2017. doi:10.1364/JOSAA.34.001099. ↩