1. INTRODUCTION

The increasing demand for pervasive connectivity also affects the aviation sector. More and more users want to use wireless services also during flights ^[1,2]. In order to increase passenger comfort during flights, airlines intend to provide, for example, information about on-board shopping, in-flight magazines, news, or information about the destination such as airport maps, hotels, rental cars, and public transportation. In addition, in-flight entertainment (IFE) is to be provided to the passengers ^[3]. Besides this general information, passenger-specific data and services, like direction information for connecting gates, departure time of connecting flights, and travel plan updates, are to be provided as well. This information is often not fully known before the passenger boards the aircraft, especially if the current flight is delayed. It is desirable that the passengers can download the information on their own mobile devices for easy and ubiquitous access. In order to provide a reasonable amount of information, several kilobytes of data have to be transferred. Typical file sizes are about 300–400 kB. As passenger-specific data is to be transmitted, aspects like security and privacy have to be considered. Interception and fraudulent access of the passengers’ data have to be prevented. Even more important, the transmission system has to comply with high safety requirements as potential access to sensitive on-board systems has to be prevented ^[4]. Additionally, the wireless transmission must not interfere with the on-board systems. Within the aircraft cabin, the wireless service has to be provided to all passengers. This results in several hundreds of users which might simultaneously use the service to download passenger-specific data. Consequently, the bandwidth has to be shared between all passengers if common wireless transmission techniques are applied. Conventional radio frequency (RF) communications cannot be easily employed in sensitive environments like aircraft cabins ^[5]. First, RF signals can interfere with electronic on-board systems. Second, as airplanes are operated globally, wireless in-flight RF communication is subject to stringent frequency regulation of local authorities ^[6,7].

A few airlines already provide wireless services to their passengers on some selective flights ^[1,2]. However, the airlines have to make large investments to provide the necessary infrastructure within the aircraft cabin and to upgrade existing airplanes. This is due to the fact that wireless communications requires the installation of complex and costly hardware equipment such as access points and transceivers. Moreover, these installations cause maintenance effort and increase the weight, and consequently the fuel consumption, of the aircraft. The majority of existing airplanes are still not equipped with wireless in-flight services. Instead of using dedicated RF-based transmission, optical wireless communication (OWC) can be applied for data transfer within the aircraft cabin. Optical signals are not subject to frequency regulation nor do they interfere with electronic devices. The application of OWC for aircraft intracabin passenger communication has already been studied in ^{[6–9" target="_self" style="display: inline;">–9]}, for instance. However, all these systems require additional complex hardware equipment, namely special optical transmitters and receivers which have to be installed into the aircraft cabin. This hardware equipment is not commercially available up to now. Moreover, this work ^{[6–9" target="_self" style="display: inline;">–9]} implies the division of the aircraft cabin into several individual communication cells. The available bandwidth has to be shared between all users (passengers), and elaborate medium access control schemes have to be implemented. Due to these reasons, a simple wireless transmission system is envisaged which is cost-efficient, safe, and easy to maintain. Moreover, a software-based solution which reuses available hardware is largely preferred as it can be readily integrated into existing airplanes.

In this paper, a simple optical wireless data transmission method for in-flight applications is proposed. The developed solution addresses the shortcomings of conventional wireless systems such as complex system implementations and additional hardware installations. In detail, the proposed application uses visual codes to transmit data. The data to be transmitted, such as text documents or images, is encoded into a sequence of several visual codes resulting in a “visual code video” which is displayed on the screen of the IFE system. This sequence is captured and decoded by a user device which acts as receiver. Conventional visual codes are only black-and-white. In this work, the black-and-white visual codes are extended to colored visual codes in order to increase the achievable data rate. The contributions of this paper are as follows: First, a novel approach for wireless in-flight data transmission is presented. The functionality of the proposed approach is demonstrated under realistic conditions within an aircraft cabin mock-up. Second, a temporal sequence-wise forward error correction (FEC) encoding scheme is implemented. This FEC encoding scheme is evaluated and compared to the existing FEC schemes of conventional visual codes. Typically, visual codes employ a frame-wise FEC encoding. Finally, a simple protocol and a synchronization approach are developed which enable the reconstruction of the transmitted data and the detection of frame transition artifacts.

The remainder of this paper is organized as follows: In Section 2, the concept of using visual codes for data transmission is shown, and related work is reviewed. Section 3 introduces the in-flight application scenario and overviews the proposed transmission system. It is shown how passenger-specific data can be transferred to a user device via the IFE system of an aircraft. Section 4 presents the visual encoding and decoding approach developed within this work. In Section 5, the FEC techniques of conventional visual codes, specifically of quick response (QR) codes, are introduced. The limitations of these techniques are analyzed, and a temporal sequence-wise FEC encoding approach is proposed to overcome the shortcomings. Section 6 shows the results of transmission experiments which have been conducted within an actual aircraft cabin mock-up under realistic application conditions. Finally, Section 7 concludes the paper.

2. RELATED WORK: VISUAL CODES FOR DATA TRANSMISSION

Visual codes are optical machine-readable representations of data which can be decoded by appropriate code readers. These so-called scanners are equipped with photosensors to capture the visual code. Visual codes have become popular due to their fast readability and large data storage capacity. The encoded information can be made up of any kind of data, such as binary or alphanumeric data. There exist several different codes like one-dimensional barcodes ^[10] or more sophisticated two-dimensional codes such as data matrix codes ^[11], maxi-codes ^[12], Aztec codes ^[13] and QR codes ^[14]. For instance, QR codes consist of black modules arranged in a square pattern on a white background as shown in Fig. 1. The upper left, upper right, and lower left corners of the code contain three identical position detection patterns. These patterns comprise three superimposed concentric squares. The capturing receiver identifies the location and the orientation of the code by detecting these patterns. Once the corners are found, the detected region in the image is rectified. The rectified image is binarized to generate black-and-white patterns. The inner part of the QR code contains various sections. Besides some meta information about size and encoding format, the code contains additional alignment patterns to assist the reading process in determining the module grid and maintaining its accuracy. A horizontal and a vertical timing pattern enable the symbol density and the code version to be determined. These patterns consist of a row and a column, respectively, of alternating black and white modules. The complete code is surrounded by a white “quiet zone.” The data section contains the actual payload and error correction codewords. These error correction codewords enable the QR code to compensate for module errors without loss of data. This redundancy allows the recovery of the encoded data even if parts of the QR code cannot be fully recovered, for instance due to bad binarization. According to ^[14], four different redundancy levels and 40 different sizes of QR codes, referred to as versions 1…40, are defined. Due to their specific shape, these two-dimensional visual codes are robust to visual distortion and enable fast readability. Commonly, visual codes comprise only black-and-white elements as they are mainly designed for print media and have to be robust to abrasion. However, there also exist special colored codes like Microsoft’s Tag ^[15]. Commonly, visual codes encode only simple information like uniform resource locators or the name of a product. As visual codes provide only low data density ^[16,17], extensions are required that mitigate the limitations in data capacity if the codes are to be applied for communications.

Fig. 1. Structure of a QR code.

下载图片 查看所有图片

In order to enable actual wireless data transmission by visual codes for the envisaged scenario, the following requirements have to be fulfilled: •

A reasonable data rate of several kilobit per second has to be provided.

There has been some work on using visual codes for wireless data transfer. In ^[18], the authors propose a system which can read seven-segment digits from displays. The capturing and reading is reported to require about 2 s. Thus, this system does not provide the required data rate. In ^[19,20], the author proposes a new visual code to determine the position of a user device relative to a screen which displays the proposed code. The code is used for a marker-based interaction with camera phones and not for actual data transmission. In ^[21], a circular two-dimensional visual code called Spotcode is proposed which is used to bypass Bluetooth device discovery. In ^[22], a three-dimensional visual code is proposed. In comparison to the two-dimensional black-and-white visual codes, this code uses color as a third dimension. However, the proposed code can only encode a small amount of data, thus not providing the required data rate. In ^[23], unsynchronized four-dimensional codes are used for wireless data transfer. Public displays, public web pages, advertisements, and commercials in movies are identified as potential applications for the transmission system. The authors combine three different two-dimensional data matrix codes in the complementary color channels red, green, and blue to one common frame. Several of these frames are displayed in a temporal sequence resulting in a colored visual code video. Thus, the authors count the time and color as additional dimensions. Moreover, the authors evaluate different aspects such as size of the codes and compression rate with respect to the data throughput. The goal of their work is to increase the robustness of the system by adding redundancy instead of increasing the transfer rate. To this end, the authors propose a redundancy concept which uses frame duplication to restore lost frames and to address synchronization issues. The authors report transmission experiments in which the visual code sequence is displayed on a screen and captured by a mobile phone which is manually aligned by the user. Due to the high redundancy of this simple concept, the reported system achieves a low throughput of only 23 characters per second with a success rate of 82%. The size of the transmitted text is 700 characters. This amount of data can be encoded into a single visual code of the application presented in this work. As shown in Section 6, the proposed application provides a data capacity of up to 1158 bytes per frame. In ^[24], another transmission system using colored visual codes is shown. The setup uses a current high-end smartphone (Samsung Galaxy SII ^[25]) for receiving the images. However, the system is reported to achieve a theoretical throughput of only 430 byte/s. Consequently, the transfer of 1 MB takes more than 20 min, which disallows a practical application. In Chap. 8 of ^[5] and ^[26] a line-of-sight image transfer system is described, which uses a pixelated display (a liquid crystal display) as transmitter and a digital camera as receiver. In order to transfer data, a series of two-dimensional intensity images is transmitted to the receiver array. The intensity images are generated by applying a two-dimensional orthogonal frequency division multiplexing (OFDM) transmission scheme. The authors investigate the capacity of this two-dimensional OFDM approach and report a low speed point-to-point transmission experiment. The authors consider a static setup and employ a high-end camera in their experiments. Frame loss is not taken into account by the authors. In ^[27], a similar system called PixNet is proposed. The encoding of the image is also based on the OFDM scheme, whereas the corner detection uses the data matrix detection algorithm. The authors motivate the use of OFDM and the image generation in the frequency domain by the fact that the encoding in the visual domain suffers from sensitivity to light changes. This sensitivity to light changes would require special light balancing techniques. However, current cameras have robust autobalancing for contrast and white. The authors report achieving up to 12 Mbit/s at a distance of 10 m. However, this performance is achieved by using static setups. Additionally, sophisticated high-definition cameras and a large 30 in. (76 cm) display is employed within the setup. A low-complexity implementation which can be applied to simple mobile devices is not within the scope of the authors. Moreover, effects caused by frame loss are not taken into account by the authors.

A detailed comparison of different visual codes and visual encoding techniques is beyond the scope of this work. Moreover, a simple encoding technique is envisaged which can be efficiently implemented on common mobile devices. Therefore, the widely used QR codes are employed in the following, whereas other visual codes might be used as well. Specifically, the corner detection, rectification, and binarization technique of these codes is used. As QR codes are well-established, they can be reliably received and decoded by nearly all current mobile devices. To the best of the authors’ knowledge, all work done so far does not specifically handle the aspect of encoding redundancy into the visual code stream to compensate for frame loss while providing reasonable data rate.

3. SYSTEM OVERVIEW: IN-FLIGHT TRANSMISSION OF PASSENGER-SPECIFIC DATA

In order to address the presented requirements for wireless in-flight applications and to overcome the shortcomings of common wireless techniques, a new approach for wireless data transmission within the aircraft cabin is proposed. This approach enables the download of passenger-specific data from the on-board IFE system onto a user device. The data is transferred by optical wireless transmission, specifically by a sequence of visual codes. Figure 2 illustrates the proposed transmission system. The central IFE server transfers the data to the IFE display at the passenger’s seat. In order to transfer the data to the user device, the data (file) is encoded into a sequence of several visual codes. The detailed operation of the data transmission via visual codes is displayed in Fig. 3. First, the file, for instance text documents or images, is compressed to reduce the amount of data which is to be transmitted. In addition, the file can be encrypted to provide security and privacy of the passenger-specific information. Second, the encrypted and compressed file is FEC encoded to compensate for potential transmission errors. In a third step, the FEC encoded data is segmented into several packets. Each of these packets is visually encoded by a visual code resulting in a sequence of several visual codes. Fourth, this visual code sequence is displayed in a continuous loop on the IFE screen like a common video film. The intended receiver, namely the user device, can start the capturing at any time without the need for an initial synchronization. This asynchronous transmission enables simple handling as the passenger has only to record a video of the IFE display using the built-in camera of the user device. In a fifth step, the captured visual codes are visually decoded. The visually decoded data packets are reassembled in the correct order by means of additionally encoded meta information providing the packet number. The reassembling reconstructs the originally FEC encoded data stream. In a sixth step, the FEC encoded data is decoded. Due to the induced FEC, errors which arise during the transmission and capturing process can be corrected. Finally, the FEC decoded data is decrypted and uncompressed and the reconstructed file is stored on the user device. As a result, the users can access the data on their own devices at any time and can edit the data (e.g., fill out documents) or zoom into maps, for instance. The passengers interact with the transmission system via the touch-screen of the IFE system ^{[2830" target="_self" style="display: inline;">–30]}. For instance, they can choose the desired data and apply settings for the data transfer via the touch-screen.

Fig. 2. Setup for the transmission of passenger-specific data.

下载图片 查看所有图片

Fig. 3. Data transmission via visual code videos.

下载图片 查看所有图片

As the IFE system can display the desired content on demand ^[3], the wireless data transfer can be individually and simultaneously used by each passenger via their respective IFE displays. The bandwidth does not have to be shared between the passengers. Moreover, the proposed application enables only the transmission of data to the passenger’s mobile device. This unidirectional communication provides a fundamental safety feature as there is no uplink connection from the user device to the sensitive on-board systems; that is, the user device cannot send data to the aircraft system. The major benefit of the proposed transmission system is that it reuses the existing IFE equipment without the need for any additional hardware installations. The system requires only a software update of the existing IFE system and an application to be installed on the user device. This software comprises the visual coding algorithms and a simple communication protocol to provide some meta information for the data transfer. Therefore, the proposed transmission system is a low-complexity and low-cost solution which is especially suitable for so-called retro-fits as it can be easily applied to existing airplanes.

4. VISUAL ENCODING AND DECODING

As the application is to be used within the aircraft cabin, specific conditions like illumination constraints or motion effects, for instance caused by turbulences, have to be considered. Moreover, the visual transmission is largely affected by the employed hardware equipment. The intended transmitter and receiver devices have specific characteristics. First, displays usually build up images line-wise. Thus, the display is not completely refreshed at once. Second, cameras typically have a so-called rolling shutter. The rolling shutter causes a vertically or horizontally scanning of the camera across an image. Additionally, low-cost built-in cameras have a small aperture. Therefore, these cameras require long exposure times, especially in low illumination environments.

Since there is no synchronization between the IFE display and the built-in camera of the user device, these hardware characteristics induce distortions if the displayed image changes during the capturing process. As a consequence, transition artifacts between two consecutive frames arise. The effect of frame overlay is illustrated in Fig. 4(c). Two consecutive frames partially overlap or superimpose completely. These transition artifacts are caused by the line-wise rendering of the display and by the rolling shutter of the camera. Additionally, cameras capture images by integrating the impinging light over a period of time (exposure time). If the displayed frame changes during this period of time, two consecutive frames are superimposed. If the hand-held camera is moved during the capturing process, motion blur occurs. The effect of motion blur is shown in Fig. 4(b). Due to the movement of the camera, the captured image represents an integration of the impinging light along the direction of movement, resulting in a blurred image.

Fig. 4. Illustration of distortions and transition artifacts induced by visual transmission: (a) frame without transition effects; (b) motion blur; (c) overlay artifacts.

下载图片 查看所有图片

If colored visual codes are considered, additional distortions occur, and further effects have to be taken into account. In Fig. 5, a captured colored visual code is decomposed into its three color channels: red, green, and blue (RGB). Compared to black-and-white codes, colored codes have a reduced optical contrast. The reduced contrast corresponds to a reduced distance of the signal constellation points which are encoded in the RGB color space. For instance, the color red has a value of (255,0,0) in the RGB color space, and the color blue has a value of (0,0,255). The color white has a value of (255,255,255), and the color black has a value of (0,0,0). Thus, the Euclidean distance between the colors red and blue is smaller than the distance between the colors white and black. Moreover, color channels suffer from cross talk resulting in “color mixing,” especially if low-cost image sensors are used. Cross talk occurs when photons impinging on one pixel are additionally sensed by other pixels of the image sensor. For instance, green colored light pointing to a green pixel additionally hits an adjacent blue pixel. As a consequence, the blue pixel shows a response and registers more blue colored light than is actually present in the captured image. This optical cross talk between the pixels reduces the color resolution and creates noisy images. As shown in Fig. 5, the red and blue channels undergo higher noise, and thus higher distortions, compared to the green channel. Nonetheless, the green channel suffers also from higher noise compared to pure black-and-white visual codes, which provide the highest optical contrast [see Fig. 4(a)]. All these effects and the resulting distortions have to be taken into account to enable a reliable data transfer via visual codes.

Fig. 5. Captured colored image and decomposed color channels for RGB: (a) captured colored image; (b) red channel; (c) green channel; (d) blue channel.

下载图片 查看所有图片

In the following, the employed visual encoding and decoding are described which address these effects. Figure 6 illustrates the proposed frame setup of the visual codes. As shown, the frames are composed of two individual visual codes: a meta code and a payload code. The payload code comprises the actual data to be transmitted. This code is displayed on the right hand side of the IFE screen. The payload code contains the conventional QR code detection patterns (concentric squares) used by the receiver for detection and rectification of the visual code. However, its actual data is encoded using the proposed sequence-wise FEC scheme to compensate for detection errors and frame loss caused by motion blur, for instance. Details about the FEC scheme are given in Section 5. The FEC encoded data bits are visually encoded by black-and-white modules or by colored modules, respectively. Since the data to be transmitted are encoded into a sequence of multiple visual codes, the captured frames have to be reassembled in the correct order. To this end, the receiver has to know the frame number, that is, the index of each visual code within the sequence. This meta information is crucial for the reconstruction of the transmitted data since receiver and transmitter are not synchronized and since frame loss may occur. Therefore, this information is separated from the actual payload. For a first implementation approach, the frame number is encoded into an additional meta code. A conventional QR code is used as meta code which is displayed on the left hand side of the IFE screen (see Fig. 6). As there is only a small amount of meta information to be transmitted, a small QR code having a low version number (e.g., version 1–2) is sufficient. Moreover, the meta code has the highest QR code redundancy level to guarantee its correct detection. As the IFE screen has a rectangular shape, there is enough space to enlarge the quadratic payload code to maximum size while still having space for simultaneously displaying the meta code on the left hand side. In order to detect frame transitions and transition artifacts, an additional frame synchronization pattern is included into the visual code. As shown in Fig. 6, a black-and-white pattern is added on the left hand side of each visual code. Specifically, two black-and-white patterns (called the even- and odd-pattern) are used, which alternate frame by frame. The patterns differ only in color: the even-pattern is a sequence of black-and-white modules, and the odd-pattern is the inverted sequence of white-and-black modules.

Fig. 6. Frame setup of visual codes. The proposed visual codes consist of three parts: payload code, meta code, and frame synchronization pattern.

下载图片 查看所有图片

For decoding, the receiver localizes the payload code and rectifies it. To this end, the QR code detection patterns are used. In detail, the localization and rectification process uses the pattern detector of the open source image processing implementation ZXing ^[31]. First, the receiver identifies the location and the orientation of the visual code by detecting the concentric squares. Once the corners of the visual code are found, a projective transformation, a so-called homography, is applied. Perspective distortions are rectified by aligning the patterns of the captured image with the known structure of the visual code. Second, the rectified visual code is binarized. To this end, a histogram of the intensity values of the captured gray-scale image is generated. The two maximum peaks of this histogram are considered as the white reference point and the black reference point. The value that optimally separates this two points in the histogram is considered as threshold for the binarization. Using this threshold, the decoder decides if a pixel is black or white, and if it is a binary one or zero, respectively.

In order to enhance the data rate, colored payload codes are used. As illustrated in Fig. 7, three independent single-colored payload codes, which correspond to three consecutive data packets, are combined to one common colored payload code. The first payload code is colored red, the second code is colored green, and the third code is colored blue. Each color corresponds to the RGB photoreceptors on the camera chip. Therefore, these colors represent three complementary color channels. The three color channels, and thus the three payload codes, are simultaneously displayed on the screen of the IFE system. Consequently, the throughput is increased by a factor of 3 compared to simple black-and-white payload codes. Depending on the data to be transmitted, the combination of the three visual codes results in one colored visual code which can have eight different colors: white, black, red, green, blue, yellow, cyan and magenta. As the three visual codes have the same basic shape, their detection patterns are identical. Therefore, the detection patterns of the three payload codes superimpose to the conventional black-and-white patterns as shown in Fig. 7. Thus, the receiver employs the conventional localization and rectification algorithm as presented above. However, each color channel is binarized individually. The captured colored payload code is converted into three separate black-and-white visual codes using three individual thresholds. As a result, color distortions caused by ambient light as well as by display and camera effects can be compensated.

Fig. 7. Merging of three payload codes into a single colored payload code.

下载图片 查看所有图片

As discussed above, the display and the camera are not synchronized. In order to ensure that the camera captures all displayed frames, the displaying rate of the visual codes is set to half the capturing rate of the camera, which is about 30 frames per second (fps). Thus, the displaying rate of the visual codes is set to 10–15 fps. The screen has an image build-up time of about 17 ms, that is, a response rate of about 60 Hz. The relation between displaying and capturing is shown in Fig. 8. As illustrated, the receiver performs an oversampling and captures a displayed frame several times. This oversampling ensures that each displayed frame is at least once captured correctly without transition effects. The other captured frame copies might undergo transition effects. For optimal processing and performance, the transition artifacts (illustrated by “X” in Fig. 8) have to be filtered out by the receiver because they are highly distorted. The receiver has to select the best captured images for the decoding process. The functionality of the selection process is as follows: For each captured frame, the frame synchronization pattern (see Fig. 6) is correlated with the original even- and odd-pattern. The frame is classified according to the pattern version which provides the highest correlation score and thus is classified as an even- or odd-frame. As long as the detected pattern version does not change compared to the previous one, the frame is added to a buffer. As soon as a new pattern version is detected, the frame with the highest correlation score is taken from the buffer and passed to the decoder. Consequently, the frame with the highest correlation score, and thus the frame with least transition effects, is chosen for further processing.

Fig. 8. Timing of displaying and capturing process. The upper time line shows how the screen displays the visual code sequence. The displaying is affected by the response rate of the screen and the time each visual code is shown. The lower time line shows the capturing times of the camera.

下载图片 查看所有图片

5. FEC CODING TECHNIQUES

Visual codes are primarily designed for print media. The codes have to be robust to abrasion and have to enable reliable decoding. Therefore, visual codes have inherent FEC capabilities to withstand damage without loss of data. In the following, the FEC coding techniques of standard visual codes are introduced using the example of QR codes. Commonly, QR codes employ a frame-wise FEC encoding; that is, error correction codewords are added to the actual data words within one visual code. According to ^[14], four different error correction levels can be employed. Table 1 shows these error correction levels and their particular recovery capacity, that is, what percentage of damaged codewords can be recovered. The error correction codewords are able to correct two types of erroneous codewords: (1) erasures, that is, erroneous codewords at known locations within the QR code; and (2) errors, that is, erroneous codewords at unknown locations. An erasure is caused by an unscanned or undecodable symbol character within the visual code. An error is caused by a misdecoded symbol character. For instance, erroneously converting a module from black to white or vice versa results in a misdecoding of the affected symbol character; that is, the decoder decides for an apparently valid but different codeword causing an error. QR codes use a Reed–Solomon FEC coding under a Galois field of GF(28) as they operate on byte characters. The number of erasures and errors which can be corrected by Reed–Solomon block-codes is given by ^[14]: where e is the number of erasures and t is the number of errors. The number of error correction codewords is denoted by q. Consequently, at least two error correction codewords are required to correct an error. The number of codewords for error detection is denoted by p. These codewords can detect (but not correct) additional errors. If the number of errors exceeds the error correction capacity, the data cannot be reconstructed correctly. Table 2 shows two error correction schemes as specified in ^[14]. The notation of the shown error correction code is (u,v,m), where u is the total number of words, v is the number of data words, and m describes the error correction capacity. For instance, a version 6 QR code contains a total number of 172 words (see Table 2). If the error correction level H is used, only 172(v/u)=172(15/43)=60 of the words are actual data words, and thus, 112 of the codewords are error correction codewords. The 112 error correction codewords are able to correct 172(m/u)=172(14/43)=56 misdecoded words which is (56/172)≈32.6% of the symbol capacity of the version 6 QR code. The actual data capacity of this QR code is (60/172)≈34.9% which corresponds to v/u=15/43.

As shown above, the FEC coding of standard visual codes is done on a per frame basis; that is, the data and the error correction codewords are contained in the same visual code. If a large amount of data, such as a text file, is distributed across several visual codes, multiple visual codes have to be decoded individually in order to reconstruct the data. However, if just one or even several of the visual codes are corrupted and cannot be reconstructed correctly, the complete information is lost. As shown in Section 6, frame loss can occur during the capturing process, even though the highest FEC level is employed. Therefore, a temporal FEC encoding of the complete visual code sequence is proposed. The loss of frames can be recovered and the data can be reconstructed correctly even if some frames are erroneously captured by the receiver.

The implemented application is based on the conventional QR code generation according to ^[14]. However, only the corner detection, the rectification, and the binarization algorithms of these codes (see Section 4) are employed. The per frame FEC encoding scheme is completely removed. Instead of this per frame encoding, the complete data is encoded resulting in a temporal per stream FEC encoding. This encoding approach is illustrated in Fig. 9. As shown, the data, for instance a text file, is passed to the encoder. The FEC encoder uses a (127,67,30) Reed–Solomon block-code. A Reed–Solomon block-code is chosen to be consistent to the standard QR code FEC coding. More powerful FEC coding techniques like Turbo Codes or soft-decision Viterbi decoding schemes can also be employed. However, these sophisticated coding techniques cannot efficiently be implemented on current mobile devices and may cause large processing delays, high power consumption, and low battery life times. The chosen Reed–Solomon code has a data capacity of v/u=(67/127)≈52.8% and a recovery capacity of m/u=(30/127)≈23.6% which is similar to the QR code error correction level Q. Higher or lower error correction levels can also be employed if higher robustness or higher throughput, respectively, is required. During a flight, higher robustness may be beneficial due to the swaying motion of the aircraft causing increased motion blur. During taxiing at the ground, a higher throughput may be required due to time constraints when reaching a connecting flight. The information about the currently employed error correction level can be incorporated into the meta code which contains information to reconstruct the data, for instance the frame number (see Section 4). In order to increase robustness, the FEC encoded data is interleaved. This means that the FEC encoded data is “randomly” distributed across all frames within the visual code sequence. Consequently, burst errors which arise from the loss of complete frames can be compensated by the receiver. Due to the chosen Reed–Solomon code, the data can be correctly reconstructed if up to about a quarter of the visual code sequence is completely missing. Thus, the temporal sequence FEC encoding is able to compensate for frame losses. The encoded and interleaved data stream is segmented into several parts of fixed size. Each segment is processed by the visual encoding method presented in Section 4.

Fig. 9. Illustration of temporal sequence-wise FEC encoding.

下载图片 查看所有图片

6. TRANSMISSION EXPERIMENTS

In the following, the results of some transmission experiments are presented. The aim of the experiments is threefold: first, the performance of the frame-wise FEC encoding of standard visual codes is evaluated. To this end, standard black-and-white QR codes and typical envisaged data sets (e.g., text files) are considered. Moreover, a low-cost user device is employed as receiver in order to perform a practical evaluation. Second, the performance of the proposed temporal per stream FEC encoding approach is evaluated. For means of comparison, black-and-white visual codes as well as the same low-cost user device are employed. Finally, the achievable data rate is maximized. To this end, a high-end user device and colored visual codes are employed to derive an upper performance bound. In order to evaluate the implemented transmission system under realistic conditions, the experiments have been conducted within an Airbus A330 aircraft cabin mock-up. Thus, the measurements have been taken within the aspired application scenario under realistic ambient light conditions and transmitter–receiver spacings. The cabin mock-up is equipped with an IFE system. The screen of the entertainment system is installed in the rear side of a passenger seat. Thus, each passenger can individually employ the respective IFE display in front of him or her. As the display is a touch-screen, the passenger can interact with the IFE system and can individually choose the content to be displayed on the screen. The IFE screen has a size of about 17cm×13cm (resulting in 8.4 in.) with a resolution of 1024×768 pixels. The passengers record the visual code sequence, which is displayed on the IFE screen, with the built-in camera of their user devices, for instance a smartphone. The distance between the transmitter (IFE screen) and the receiver (user device) is about 20–30 cm. As the passenger manually aligns the user device towards the IFE screen, there is no fixed alignment between the transmitter and the receiver. Moreover, as the passenger moves his or her hand during the capturing, such as due to turbulences during the flight, the alignment changes for each frame of the visual code sequence. Therefore, the receiver has to individually detect the transmitted visual code within each captured image. However, the movement causes additional effects like motion blur which affect the correct detection of a transmitted visual code. If one or several frames are lost, the transmitted information cannot be correctly decoded by the receiver. Thus, the whole data is lost and the user has to restart the transmission process. Consequently, the transmission and the frame detection have to be robust.

6.2 A. Performance of Conventional Visual Codes with Frame-wise FEC Coding

In the following, the performance of conventional black-and-white QR codes with per frame FEC coding is evaluated. The data to be transmitted is split into several segments. Each segment is encoded into one QR code. These QR codes are sequentially displayed on the IFE screen and are recorded with the user device. The user device is a low-cost Samsung Galaxy Ace smartphone ^[32]. The built-in camera of this smartphone has a resolution of 640×480 pixels in the video mode. In order to compensate for frame transition artifacts, the frame rate of the IFE display has to be downsampled to at least 10 fps. Experiments have shown that this is the largest frame rate which ensures that each frame is captured by this smartphone at least once without transition artifacts. Due to this oversampling, a sophisticated frame synchronization between transmitter and receiver is not required. Several experiments using different QR code versions and error correction levels have been conducted. For the experiments, the smartphone is manually aligned in front of the IFE screen, resulting in realistic motion effects. After a complete QR code sequence has been displayed on the screen, the captured frames are decoded. For performance evaluation, it is checked if the transmitted data is received correctly, and the symbol error rate is calculated. The aim is to transmit as much data as possible within a given time period, that is, to provide high data rate, while enabling the correct reconstruction of the transmitted data. This is due to the fact that capturing the screen mainly affects user comfort whereas the subsequent data processing is done off-line and therefore is considered uncritical.

Table 3 shows the effective symbol rate (throughput) which can be achieved per frame related to the QR code version and error correction level. The total rate is the sum of data words and error correction codewords per frame. As shown, level H codes have a lower data capacity compared to level L codes due to increased redundancy and error correction capability. Besides, the measured mean symbol error rate for a transmitted QR code sequence is shown. It can be seen that due to the lower error correction capability, level L codes provide a higher symbol error rate than level H codes. Level L codes enable a reliable data transmission; that is, the transmitted data is received correctly, up to version 9. However, level H codes enable a reliable transmission up to version 12. Using larger QR codes results in increasing misdetections and erroneously received data. Summing up, version 9 L codes achieve the highest symbol rate provided that the data is received without any errors. The symbol rate of this scenario is about 2300 data words/s (230data words/frame with 10fps). This symbol rate results in a bit rate of 18.4 kbit/s as one data word comprises 8 bits. Note however that this data rate is achieved without distinct motion effects which might occur during a flight due to turbulences. Thus, this data rate represents a best case result as additional motion effects induce increased frame loss which cannot be compensated by the per frame FEC coding of standard visual codes. Consequently, this per frame encoding scheme is not suitable for the proposed application which relies on the transmission of a temporal sequence of several visual codes.

6.3 B. Performance of Visual Codes with Temporal Sequence-Wise FEC Coding

In the following, the performance of the proposed temporal FEC coding is analyzed. Table 4 shows the performance of black-and-white visual codes using the FEC coding approach presented in Section 5. For means of comparison, the same setup scenario and user device are used as presented above. The results show that this FEC coding approach provides a more reliable data transmission in comparison to the per frame encoding technique. This is due to the fact that the temporal sequence FEC coding can compensate for the loss of complete frames. Thus, frames which are lost or misdetected can be reconstructed by the receiver if the error/loss rate is smaller than 23.6% (depending on the chosen coding rate as shown in Section 5). Consequently, the chosen FEC coding technique provides a reliable data transmission, that is, the transmitted data is correctly received, up to QR code version 13. As shown by these results, applying the FEC coding technique improves the reliability of data transmission and induces a drop of error rate. Given the chosen coding rate, this QR code version enables a symbol rate of 2800 data words/s (280data words/framewith10fps). This symbol rate results in a bit rate of 22.4 kbit/s. Thus, the achievable data rate is increased by 21.7% compared to the per frame FEC coding scheme.

6.4 C. Enhancement of Data Rate Using Colored Visual Codes

In the following, the performance of the proposed application is analyzed if a high-end user device is used instead of a low-cost device. Therefore, an Apple iPhone 4 smartphone ^[33] is used as receiver device. The built-in camera of this smartphone has a higher resolution of 1280×720 pixels in video mode. Moreover, for this user device, the frame rate of the IFE display can be increased to 13 fps. This transmission rate ensures that each frame is captured by this smartphone at least once without transition artifacts. Table 5 shows the performance using black-and-white visual codes with temporal sequence FEC coding for this user device. It can be seen that due to the superior built-in camera, this smartphone provides an increased performance as it enables a reliable transmission up to QR code version 19. Thus, a symbol rate of 6786 data words/s (522data words/framewith13fps) can be achieved. This symbol rate results in a bit rate of 54.29 kbit/s. This data rate exceeds the achievable data rate of the low-cost smartphone by a factor of about 2.42.

The data rate can be additionally increased by using colored visual codes. Figure 6 shows a colored visual code. As shown, the frame contains the colored payload code and the black-and-white meta code which encodes the frame number. For decoding, the colored payload code is first of all split into the complementary RGB color channels. This splitting generates three independent visual codes which are separately processed and decoded (see Fig. 5). By using these three independent color channels, the data rate can be increased by a factor of three. However, as shown in Section 4, the colored codes suffer from optical cross talk and color mixing. Therefore, colored codes are subject to increased error rates. Table 6 shows the performance of colored visual codes with temporal sequence FEC coding using an Apple iPhone 4 smartphone as the receiving device. It can be seen that the QR code version has to be reduced if colored visual codes are applied. As shown, the data can be correctly received up to QR code version 16. Nonetheless, a higher symbol rate of 15054 data words/s (1158 data words/framewith13fps) can be achieved by using colored codes. This symbol rate results in a bit rate of 120.43 kbit/s which is an increase by a factor of about 2.22 compared to the black-and-white codes.

Table 7 summarizes all performance bounds derived from the conducted transmission experiments.

7. SUMMARY AND CONCLUSIONS

A new approach for wireless data transmission within an aircraft cabin has been identified. An in-flight optical wireless transmission system has been proposed to transfer passenger-specific data to a user device. By displaying visual code sequences on the IFE screen inside an aircraft cabin, data can be securely and safely transmitted to a user device. As the application uses existing equipment, no additional hardware installations are required. The proposed approach enables a low-cost transmission system which requires only an additional software application to be installed on the user device and a software update of the IFE system. Therefore, the application is especially suitable for aircraft retro-fits.

The frames consist of a meta code, which contains supplementary information like the frame number, and the payload code, which contains the information to be transmitted. In a future implementation, these two codes might be combined to one common visual code. The meta information might be included into the payload code by inserting a meta data field. Furthermore, a selection algorithm has been implemented which uses a synchronization pattern to detect frame transitions. By applying this algorithm, the receiver is able to choose the captured frames with least frame transition effects for the decoding process. Experiments within an aircraft cabin mock-up have demonstrated the functionality of the proposed application under realistic conditions. The performance of the unidirectional transmission system has been evaluated. The application provides a throughput of up to 120 kbit/s with current smartphones. Tests have shown that 30 s is a tolerable time which is accepted by a user (passenger) to capture the IFE screen with a mobile device. This duration is merely derived from subjective perception. Longer capturing runs can also be employed. Considering a duration of 30 s, up to 450 kB can be transmitted by a single visual code sequence. This file size is sufficient to transfer the required amount of data for the proposed application. As the data processing is done off-line, the mobile device does not have to be placed in front of the screen for the processing. Therefore, the processing does not affect the user comfort as capturing the screen with the mobile device is the most crucial aspect from a user’s point of view. The actual duration of the data processing largely depends on the specific user device, namely, its performance, processing power, and current load. Experiments have shown that complete frames may be misdetected or lost due to occlusion and movement of the receiver. In order to compensate for misdetections and frame loss, a temporal sequence-wise FEC coding method has been implemented. This sequence-wise encoding has been shown to outperform the frame-wise FEC coding of conventional visual codes as it provides higher reliability and increased throughput. The performance has been evaluated for current low-cost and high-end smartphones. In order to increase data rate, the black-and-white visual codes have been enhanced to colored codes by using the three complementary RGB color channels. These colored codes provide a data rate which is more than two times larger compared to the black-and-white codes, even though the colored codes suffer from increased noise and cross talk.

Future work might evaluate the performance of visual codes with more than eight different colors. For instance, the intensity of the pixels might be varied to encode additional data and to increase throughput. Additionally, more sophisticated FEC coding schemes and visual calibration techniques might be evaluated to further enhance the performance of the aspired application. In this context, the impact of receiver field of view on the performance might be investigated in more detail as well as the impact of ambient light noise such as sunlight and illumination light.