Eduardo Cervantes Aguilar

Serious Games: Rehabilitation Fuzzy Grammar for Exercise and Therapy Compliance Victor Fernandez-Cervantes, Eleni Stroulia Luis E. Oliva, Francisco Gonzalez, and Claudio Castillo University of Alberta ITS Chapala, ITS Zapopan Edmonton AB. Canada Chapala and Zapopan, Jal. Me´xico Email: {vf, stroulia}@ualberta.ca Email: {luis.oliva, francisco.gonzalez, claudio.castillo}@itschapala.edu.mx Abstract—Serious Games (SG) are advocated as a technology and the modesty this privacy affords them [6]. Unfortunately, for engaging and motivating a variety of activities, such as the lack of perseverance and consistency in performing the learning and exercising. The motivating intuition is that infus- required exercises, and the potential for errors in the absence ing activities with game mechanics should make them more of expert observation and monitoring are major concerns interesting and entertaining, resulting in increased practice time, for therapy at home. These reasons make rehabilitation and and consequently, improved performance. In our work we are physiotherapy an excellent domain in which to deploy serious interested in the role that serious games can play in rehabilitation, games. Computer games, with aesthetically pleasing imagery, relying on affordable, accessible and increasingly precise bio- interesting graphics, and engaging storytelling, can mediate mechanic sensors, such as the KinectT M . In this paper, we the boring aspect of repetitive rehabilitation exercises, at a describe a KinectT M -based system that guides players through relatively low cost. their prescribed rehabilitation-exercise regimen at home, after a sports injury. The system is endowed with a grammar, in terms This is why we have developed an integrated system for of which the rehabilitation exercises can be precisely specified physical-therapy rehabilitation. The model is based but not lim- by physical therapists, and a fuzzy-logic-based component that ited to the KinectT M tracking skeleton2. The KinectT M sensor discerns in real time, whether the user “correctly” follows the is able to observe, with sufficient precision, the “player” prescribed regimen. To respect the privacy of the player, the movements. The recorded observations are analyzed, based system mimics the postures into an avatar. We demonstrate our on a fuzzy-logic method, to extract information about the system through the development of three different games. sequence of postures that the user moves through. The ob- served posture sequences are compared against a specification I. INTRODUCTION of the prescribed exercise, in terms of a posture grammar. This exercise-observation-analysis-assessment process takes place The term “Serious Games (SG)1” is used to denote “enter- with a game as the background, implemented in Unity 3D, taining games with no entertainment goals”, that is “games where specific prescribed postures (and their correct sequenc- that educate, train and inform” [1], as “a mental contest ing) correspond to game interactions advancing the game against a computer in accordance to specific rules, which objectives. Thus, the game reflects the rehabilitation objectives, use entertainment to improve corporate training, education, which is essential for reaping the benefits of serious gaming. health, etc” [2]. The core idea behind the SG paradigm is The complete record of the user’s exercise is archived on a to infuse simulation, learning, and training applications with cloud back-end, designed to support longitudinal analysis for game-playing interactions, in order to improve engagement the purpose of assessing the user’s progress and evaluating the and acquisition of new skills [3]. In particular “serious games effectiveness of the regimen. We have evaluated the expres- for health” is attracting increased interest and includes (at siveness of our physical-therapy rehabilitation grammar and least) three broad categories of applications [4]: (1) healthy- the effectiveness of our platform with the development of three habit games, focused on exercise, fitness, and nutrition; (2) different games. patient treatment, focused on assessment and personalized rehabilitation; and (3) medical training, focused on a spectrum The remainder of this paper is organized as follows: Section of skills training for health professionals, such as surgeon II reviews the technological background of this work and training, radiology operation, patient care and others. places it in the context of related research, Section II-C, describes the viability to implement rehabilitation exercises The second category is of particular interest, since the with KinectT M , in Section IV describes our fuzzy grammar rising cost of health care and the lack of adequate numbers exercise description proposal, then in Section V-A explain of trained individuals may result in patients being left behind, the game design, and finally in Section VI we present the suffering negative affects to their physical, social, and psy- conclusions. chological development. Physiotherapy and rehabilitation, in particular, require that the individual consistently and regularly II. BACKGROUND performs a repetitive physical-exercise regimen, over a long period of time. Ideally, this activity should be conducted under There has already been substantial research in the area of the direct supervision of a therapist, which is impractical, gamified rehabilitation. Some of this research uses special- however, as patients prefer the comfort of their own home 2https://msdn.microsoft.com/en-us/library/hh855352.aspx 1http://www.seriousgamesassociation.com/

purpose equipment. A head-motion monitoring system for IR, a RGB, and a depth image [12]. Even though the accuracy home rehabilitation was described in [8], involving a complex of the KinectT M in computing the joints’ positions is much set of algorithms at the back-end to process the collected data. lower than motion-capture systems[13], it has now reached They show an inexpensive technique for tracking and optimiz- the point of being “good enough” for recognizing how well ing “Head Angular Motion-Monitoring System (HAMMS)” patients follow their prescribed clinical rehabilitation-exercise rehabilitation exercise at home. This has the potential to regimens with minor errors (about 10cm) in the case where improve clinical outcomes and to reduce the duration of the the sensor is positioned in an ideal range (1m to 3m) and with treatment. More recently, a number of research projects have an effective field of view. In effect, the KinectT M is the most been pursuing the objective of gamified rehabilitation relying practical way to capture RGB images with depth map in real on commercial, off-the-shelf gaming platforms. time. In [7], the Wii Balance Board was evaluated as an inexpen- (a) (b) sive and precise force platform for personalized medical care. Subjects without lower limb pathologies performed single and Fig. 1. (a) Skeleton Joints Recognized by the KinectT M Sensor; (b) Joint double leg standing balance tests with open and closed eyes. Hierarchy Their results indicated that the device could be a valid tool for assessing standing balance, failing only one of the four tests The KinectT M SDK version 1.7 skeleton tracking estimates (double limp, eyes open). the position and orientation of 20 joints, as shown in Figure 1(a), providing the absolute player orientation in camera space The Xbox Kinect3 game console was used to monitor coordinates. Assuming that the system has its origin at the patients4, who typically perform therapeutic exercises at home hip-center joint, the y-axis is upright, the x-axis is to the left with or without the direct supervision of a therapist [9]. The and the z-axis faces the camera5. The calculation accuracy data captured by the sensors is compared to the recommended is affected by several environmental characteristics, such as, body posture for that specific patient and exercise. Thus, for example, light changes, body size, poses, noise or bulky immediate feedback can be given to the patient about the clothes. correctness of the movements. [10] describes a rehabilitation game for balance training of adults with neurological injury. The hierarchy is the correct sequence in which the skeleton In the game, the player moves through a mine shaft with eight joints are recognized by the KinectT M SDK 1.7, as shown in jewels scattered in it, glowing to indicate when they can be Figure 1(b). It is centered at the hip joint (CH), which extends gathered. The order is controlled by three pre-defined patterns to the left and right hips (LH, RH), knees (KL, KR), ankles (Sequential pattern, Simon pattern, or Sam pattern), that the (AL, AR) and feet (FL, FR). CH also extends to the spine therapist configures before each session. Our platform offers a (SP), all the way up to the head (H) through the shoulder general grammar for specifying posture sequences, expressive center (CS), which itself extends to the shoulders (SL, SR), enough to capture more exercises than what is possible with elbows (EL, ER), wrists (WL, WR) and hands (HL, HR). This these three patterns. hierarchy causes difficulties with several postures (including some of the postures involved in rehabilitation exercises in The sorcerer’s apprentice, presented in [11], is an innova- section II-C), due to occlusion or, more generally, lack of tive game design for a specific pathology “Subacromial Im- information regarding the root of a particular joint and its pingement Syndrome (SIS)”, with KinectT M . The prescribed hierarchical rotation, i.e., the amount of rotation in the 3D movements are specified in terms of boundaries within which space that the joint inherits from its parent joint. the patient has to move; these boundaries are configured by a therapist, and exceeding them triggers a warning and resets B. Fuzzy Logic the exercise. Their studies show that the game context allows The fuzzy-computing paradigm [14], has been used in the patient focus on playing and forget the exercise. However, this is not enough, since each rehabilitation exercise must be many different fields, including computer vision, decision adjusted to the needs of each user. 5https://msdn.microsoft.com/en-us/library/hh855348.aspx These experiences indicate that serious games may be a reasonable, low barrier, motivating and sustainable means to improve or, at least, delay the decline of selected social, sensory-motor, cognitive and emotional functions of people in need of physical rehabilitation. A. KinectT M The first version of the KinectT M sensor is composed of an IR pattern projector and an IR camera, which are used to triangulate points in space; effectively, the two components work together like a depth and color camera (RGB) that can be used to recognize the image content and the texture of the 3D points. For measurements, the device delivers three outputs: an 3http://www.xbox.com/KINECT 4http://doctorkinetic.nl/

making, and control systems, just to name a few [15]. The main Finally, in order to assess the feasibility of each exercise, reason to use fuzzy logic is because this paradigm considers that uncertainty is unavoidable when concepts are imprecisely we analyzed the movement of the joints at three different expressed in natural language. The main characteristics of the orientations [13], from frontal to side view in 45◦ increments rehabilitation exercise description, replacing “true” and “false” with continuous set membership values µs : U → [0, 1], and (i.e., the angle between camera optical axis and the sagittal associating each element with a real value between 0 and 1. plane was 0◦, 45◦, 90◦) at a distance between 3.5m to 5m. The representation of the membership function depends not only on the concept, but also on the context in which it is used. Our observations are described in table I, with the following Thus, it is possible to derive the output from the analysis of all possible conditions between the trapezoid-shaped membership explanation of the symbols: functions. This function is defined in equation 1 and illustrate ⊕EP: Exercise prescribed by the therapist in the rehabilitation. the membership function in figure 6. Typically it evaluates the variables in natural-language terms, i.e., , from “correct” EP: Exercise not prescribed by the therapist. to “incorrect” with a certainty value and It has the following ⊕KI: Implementation with KinectT M was achieved. properties: the upper side is not longer than the lower side α ≤ β ≤ γ ≤ δ and the value of each element in the upper KI: Exercise not considered implementable with KinectT M . side θ, is always equal to 1 [17]. KI: KinectT M initial position changes are necessary. ⊗KI: The tracking needs more than one KinectT M device. TABLE I. EXERCISES AVAILABLE FOR THE KINECT’S ACCURACY RANGE  0 when x < α and x > δ Rehab Flection when α ≤ x ≤ β Extension when β ≤ x ≤ γ Shoulder Rotation when γ ≤ x ≤ δ Elbow Abduction/ Forearm Adduction Wrist Supination Hip Deviation/ Knee Inclination Ankle  (α − x)θ Neck EP |KI EP |KI EP |KI EP |KI EP |KI EP |KI  ⊕|⊕ ⊕|⊕ ⊕| ⊕|⊕ | |  ⊕|⊕ ⊕|⊕ | | | | |  | | | | ⊕| ⊕| | | | | ⊕|  ⊕|⊗ | ⊕| ⊕|⊕ | | ⊕|⊗ | | | | |  α−β ⊕|⊗ | | | | | θ ⊕|⊗ ⊕|⊗ ⊕|⊗ | | f (x; α, β, γ, δ, θ) = (1) ⊕|⊕  (δ − x)θ      δ−γ Each trapezoidal function is composed by a set of if-then rules Once we evaluated the exercise together with the hardware as follows: limitation, we start with the simplest exercise rehabilitation: repetitive movements which involves only one angle in a IF((x < α) ∧ (x > δ))THENf (x) is 0 hinge movement, composed by simple instructions, and easy to use at invariant distance. Finally, for each of the prescribed IF((x ≥ α) ∧ (x ≤ β))THENf (x) is (α − x)θ exercises that we found implementable with a single device α−β and no position changes, we empirically configured the min- (2) imum angle change between the joints (at least five degrees IF((x ≥ β) ∧ (x ≤ γ))THENf (x) is θ difference) required in order to consider the angle change as exercise movement instead of noise from the KinectT M during IF((x ≥ γ) ∧ (x ≤ δ))THENf (x) is (δ − x)θ the tracking. δ−γ These rules assume the conditional statement form with the antecedent (IF) which has several preconditions and the con- sequent (THEN) that describes the action of the condition. C. Rehabilitation-Exercise Selection III. REHABILITATION EXERCISE GRAMMAR In order to design our posture-sequence grammar, we Our exercise-description grammar is composed of two studied a broad spectrum of rehabilitation exercises, in collab- elements: the body-posture grammar and the displacement oration with physiotherapy experts. Considering the various grammar. The former defines the position of the joints of different causes of injuries, namely ligament injury, surgical interest in each particular posture that the player has to hold. intervention, and post-traumatic pathologies, and the degree The displacement grammar defines how the movement of each to which physical assistance is required from a therapist, we joint should be tracked from one posture to the next, during the conclude that our rehabilitation games will be most useful in prescribed exercise. Figure 2 illustrates the elements of posture the last three stages of motor-control conditioning [18]: description, and Figure 3 shows the displacement-grammar definition. • spinal reflexes (reflex of the joint during abnormal stress condition); A. Body Postures • cognitive programming (ability to maintain postures The basic element of the body posture is the interior angle and balance of the body); and between three joints. Each joint is described in terms of its • brain-stem activity (cognitive awareness of voluntary movements). Cartesian coordinates in a three-dimensional space and its We then selected the specific rehabilitation exercises con- respective orientation. The rule p refers to three joints, based sidering the KinectT M accuracy limitations discussed in Sec- on the KinectT M skeleton shown in Figure 1. The joints J1 and tion II-A, and which we considered the coverage scope for our J3 form an interior angle with J2; the rule p also defines the KinectT M implementation. correct angles θ among these joints in the corresponding axes: x left-right, y down-up, and z front-back). p = [J1, J2, J3, θx, θy, θz] (3)

The validation of the angle between three joints as a p rule is not enough to describe a posture. The therapist has to define a set of rules P for each specific rehabilitation exercise, which together ensure the correct body posture as follows. P = {p1, p2, ..., pn} (4) For example, the posture grammar for rehabilitation of the right shoulder, through a sequence of abductions and adductions, is described in Figure 2 below. The angle between the three Fig. 3. Exercise Movement Grammar Fig. 2. The posture rules for right-shoulder rehabilitation. In effect, an exercise E is defined as a sequence of intermediate postures, corresponding to the n continuous changes in the joints is based on the initials defined in Figure 1. The first angle between the consecutive postures, defined as follows: three rules p1, p2 and p3, describe the posture of the shoulder aligned to the arm to prevent forward-backward or bending Et = {e0, e1, ..., en} (7) movement of the arm, relative to the body. p4 describes the correct posture between the shoulders. The rule p5 describes The initial displacement is accomplished when the player the correct back posture: SP and H make a 180y◦,z angle with reaches the proper initial posture; the transition between CS, to ensure that the head does not bend left-to-right (axis “not ready” and this “initial posture” corresponds to e0. The y), forward-backward (axis z), or up-down (axis z). maximum displacement en is reached in the transition from Pn−1 to Pn. Then, the system starts observing the player B. Exercise Movements move from the maximum displacement back to the initial position e0. Each ei element connects two postures: Pi and The exercise-movement grammar defines the sequence of Pi+1, this is exemplified in figure 3. -The initial position [I] (e0) is the starting posture specified postures describing the displacement of the angles between by the therapist to begin the exercise. -The maximum displacement [M] (en) refers to the maximum consecutive postures, as we define in equation 5 and illustrate angle aperture permitted during the exercise; it is typically reached and held at the final posture. in Figure 3. -The time step t (ti ≡ ei) defines the frequency (in “time steps”) in terms of which the player’s posture P should be E = [P0, P1, ..., Pn] (5) compared against the corresponding posture ei in the correct “time step”. The movement between each two subsequent postures -The normal displacement [N] is the minimum expected movement over the exercise movement range prescribed by in an exercise E, as defined in 5, is defined as AC = the therapist. In effect this is the acceptable end position, stressing the joint that is being rehabilitated. [J1, J2, J3, θxI , θxF , θyI , θyF , θzI , θzF , t]. The joints J1 and J3 form -The bonus displacement defines a recognizable effort, above an interior angle with the main joint J2 in three-dimensional and beyond the basic exercise movement (which is defined by the “normal displacement”) and less than the “maximum space. To define the overall movement between two postures, displacement”. -The bar-attachment point defines where the we define θI as the initial angle of the three relevant joints in guidance/feedback bar is placed into the screen, attached to a specific joint J2 (visually this position has to make sense to the original posture, and θF as the angle among the same joints guide the movement). -The user indicator visually indicates the movement that in the final posture, at the end of the movement. We then divide player has to follow, according to specific grammar through the elements in E shown in equation 7. the entire range of movement in time steps (recommended in section II-C as 5◦ at least), to balance the need for continuity in the movement tracking and accuracy. On every time step we create the posture description pi ∈ P rules; the final rule in the posture pn ∈ P will describe the correct displacement based on the θI and θyF . It is important to note here that the assumption underlying this grammar is that each exercise is, in effect, designed to expand the range of motion along this angle of interest. θF − θI en = t (6) Figure 4 illustrates the automaton that recognizes whether

or not the player performs the prescribed exercise. The states performance and the prescribed movement through the correct I, N , H and M represent the initial position, the normal posture sequence. The posture validation component relies on displacement, the bonus displacement, and the maximum two different sets of rules. The static postures, shown in Figure bonus displacement correspondingly. This automaton defines 2, have to be precisely recognized during the exercise. The dynamic posture rules are related to the guidance/feedback bar. Fig. 4. Pushdown automaton model for the guidance/feedback bar’s behavior. Intuitively, the postures defining the exercise 5, are prescribed by the therapist and have to be recognized or else the player the behavior of a special-purpose widget (in Unity 3D) cannot be said to perform the exercise in a valid form. The dy- we have developed for our rehabilitation environment, the namic movement between subsequent postures is evaluated to guidance/feedback bar (shown in Figure 5), which plays the assess whether the player’s speed and rhythm are appropriate. dual role of (a) guiding the player’s movement towards the prescribed exercise, and (b) providing feedback to the player The posture-comparison method relies on a metric for as to how exactly he is performing relative to the exercise specification. comparing the angles of the corresponding joints associated to the pd rule. Once the KinectT M captures the posture of the player P c, it is compared against the relevant rules from P E, to produce the difference between the two postures Pd once every time step ti as follows: ∀pi ∈ P E : ∃! pk ∈ P c ∴ pd =| pk − pi | pd = pd = [J1, J2, J3, θxc , θyc , θzc] − [J1, J2, J3, θxE , θyE , θzE ] pd = [J1, J2, J3, | θxc − θxE |, | θyc − θyE |, | θzc − θzE |] (8) Pd = [J1, J2, J3, θxd, θyd, θzd] k pdi ∴ Pd = {p1d, ..., pdk } i=1 therefore, the P d is the posture difference in terms of p rules. The two criteria are based in the hypothesis that static postures rules, are critical to validate the exercises, and Dynamic pos- ture rules, describes valid movements which in the worse case scenario are delayed, but belongs to the exercise. Otherwise the dynamic posture rule should be restricted by static posture rules. (a) (b) A. Static Posture Rules Fig. 5. (a) Parts of the Movement-Guiding Bar; (b) Description of the time The static rules define a comparison of the absolute value step components over the three angles between the joints of interest for each To illustrate the role of the guidance/feedback bar let us θcx, θcy, θcz as show in equation 8, then each P d is qualified review the shoulder adduction-abduction exercise, described with a fuzzy membership function as follows: below: f1 = f (x1; 0, 0, 5, 10, 1) = P erf ect (9) E = [RH, SR, W R, θ, θ, 0, 100, θ, θ, 20] f2 = f (x2; 5, 10, 15, 20, 1) = Inadequate f3 = f (x3; 15, 20, 180, 180, 1) = Incorrect The specification uses the right-hip and right-wrist joints to calculate the shoulder interior angle. The guidance bar Finally, a set of fuzzy functions qualify the overall quality goes from 0◦ to 100◦ in the y-axis, in 20 time steps. The of the player’s posture. The absolute value of θ is labeled therapist instruction indicates that the rehabilitation exercise is according to the rules below. composed by two phases: (a) the displacement to reach the maximum posture, and (b) the return to the initial posture, IF(θc ≤ 5) THEN f (x) is perf ect both of which are equally important. The bar indicates that IF((θc > 5) ∧ (θc ≤ 7.5))THENf (x) is “almostperf ect a typical correct execution of the exercise may involve some IF((θc > 7.5) ∧ (θc ≤ 10))THENf (x) is “goodenough time in the initial position I (three in the example 0◦-15◦). IF((θc > 10) ∧ (θc ≤ 15))THENf (x) is “inadequate The normal displacement N should be reached next (in five IF((θc > 15) ∧ (θc ≤ 17.5))THENf (x) is “borderline time steps in the example 15◦-75◦). The player should move IF((θc > 17.5) ∧ (θc ≤ 20))THENf (x) is “poor past the normal position to reach the bonus displacement H IF((θc > 20) ∧ (θc ≤ 180))THENf (x) is “incorrect (in the example two steps, and this covers angles between 75◦ to 90◦) and may next move to the one maximum displacement The player receives a number of cumulative reward points, M (two steps in the example:90◦-100◦). depending on whether he achieves a label between “almost perfect” and “poor”. If, however, at any point, the player’s IV. REHABILITATION ENVIRONMENT AND PROCESS posture is “incorrect”, the game pauses immediately, since the label “incorrect” is reserved to describe forbidden postures Having described the rehabilitation exercise, our system during the exercise, which can potentially hurt rather than help can now quantify the differences between the player’s actual in the rehabilitation process.

B. Dynamic Posture Rules (in)correctly, and the user-interface component through which to guide the player, we now focus on the process of designing The Dynamic posture rules are less demanding, the calcu- “games” as a motivating background story for the exercise. lation of the P d rules are based in the equation 8. Each angle In effect, the game mechanics are designed to align with the θxd, θyd, θzd can be described as either negative or positive values prescribed postures and movements, so that correct perfor- as follow: mance of the exercise is rewarded by the game rules. We have developed three such games; for elbow rehabilitation (the pd = [J1, J2, J3, θxc − θxE , θyc − θyE , θzc − θzE ] (10) “Fishing” game shown in Figure 8), shoulder rehabilitation (the “Spaceship Orbit” game shown in Figure 9) and knee so as to quantify the movement posture differences in both rehabilitation (the “Way of the Penguin” game, shown in relative directions forward-backward from the actual posture. Figure 7). To complete the movement guide shown in Figure 5, each element is associated with a semantic phase to validate the ex- A. Game ercise. The movement grammar is qualified with the following equation: M P = [Insuf f icient, Inadequate, P erf ect, (11) Exaggerated, Overmuch] the definition of the trapezoidal shaped membership functions are constructed following the equation 1, and composed by a set of if-then rules illustrated in equation 2, this is illustrated in figure 6. Fig. 8. Elbow rehabilitation exercise with a fisherman game Fig. 6. Fuzzy Membership Functions for movement grammar Fig. 9. Shoulder rehabilitation exercise with a spaceship orbiting the moon game The elements of the dynamic posture rules are listed with the respective values in f (M P ; αi, βi, γi, δi, θi) as follow: The key problem in designing these games is to come up with a proper metaphor that aligns with the movements in- I rules: f(Insufficient)=(-180, -180, -20, -15) volved in the rehabilitation exercise. An appropriate metaphor f(Inadequate)=(-20, -15, -10, -5) places the player in the context of a situation where the player’s f(Perfect)=(-10, -5, 5, 10) natural movements in response to the game story would align f(Exaggerated)=(5, 10, 15, 20) well with the rehabilitation-exercise movements. Then, the f(Overmuch)=(15, 20, 180, 180) movements defined, recognized and assessed based on the grammar of the prescribed exercise naturally advance the game N rules f(Insufficient)=(-180, -180, -16, -12) story. f(Inadequate)=(-16, -12, -8, -4) f(Perfect)=(-8, -4, 4, 8) The “Fishing” game for example adopts the metaphor of f(Exaggerated)=(4, 8, 12, 16) “fishing in a quiet lake” to engage and motivate people on f(Overmuch)=(12, 16, 180, 180) an elbow-rehabilitation regimen. The hypothesis is that this metaphor will make sense to players, especially if they fish, H rules: f(Insufficient)=(-180, -180, -10, -7.5) and will make the exercise aspect of the game transparent: f(Inadequate)=(-10, -7.5, -5, -2.5) since, during fishing the elbows have to move in a way f(Perfect)=(-5, -2.5, 2.5, 5) very similar to the rehabilitation exercise, the movement will f(Exaggerated)=(2.5, 5, 7.5, 10) f(Overmuch)=(7.5, 10, 180, 180) To exemplify this, we use the posture N01 in figure 3, pE = [CH, SR, W R, X, 90, 180] the interior angle in SR form a 90◦y, 180z◦, in this example the user posture pc = [90, 88, 176]. Due θx = X the difference is 0, the θyd = θyE − θyc and θzd = θzE − θzc, ∴ pd = [0, 2, 4]. Then, the rule is labeled as Perfect using the rules from item N. V. GAMES AS REHABILITATION CONTEXTS Having defined the prescribed exercise, the mechanism for assessing the degree to which the exercise is performed

Fig. 7. Knee rehabilitation exercise with the way of the penguin game become a natural aspect of interacting with the game (and will the player has to stay in that posture for five seconds. no longer be associated with the therapeutic exercise). The value of the each second is multiplied by the correct posture score. The “Spaceship Orbit” game is less obviously aligned with the shoulder rehabilitation exercise. The player’s movement Small penalties and big rewards should encourage the player controls the speed of the spaceship. The assumption is that the to continue the rehabilitation. The only exception to these player will try to move the spaceship away from the moon by rules are the “overmuch” and “insufficient”, these big penalties, controlling the position of his arm with respect to his shoulder. because the postures that fit into this category are out of the grammar scope. Thus, the assumption is that these are danger The “Way of the Penguin” for knee rehabilitation assumes postures (they could hurt rather than help). that the player will feel involved by helping the penguin complete a trip on a frozen path over the ocean. When the TABLE II. REWARDS AND PENALTIES exercise is executed according to the instructions the penguin smoothly surfs over the surface otherwise it falls and slips Insufficient straight into the path. Inadequate Perfect B. Scoring Rules Game Rule Exaggerated Overmuch The goal of any game player is to obtain the game rewards. In designing rehabilitation games, we have to ensure that the Fisherman I -100 -2 +5 -2 -100 game rewards represent the quality of the player’s execution of the exercise, or else, there would be no therapeutic value in Fisherman N -100 -3 +10 -3 -100 the game at all. Fisherman H -100 +2 +20 +2 -100 In our games, game rewards are controlled by two different measures: the correctness of each posture, and the correctness Spaceship I -100 -2 +5 -2 -100 of the movement based on how closely the player follows the guiding-feedback bar. In section IV, we described different Spaceship N -100 -3 +10 -3 -100 rules for each element of the grammar. Each one represents a different value in the final reward. Spaceship H -100 +2 +20 +2 -100 • The initial score is (100), and the game starts when Penguin I -100 -2 +5 -2 -100 the system recognizes that the player has achieved the initial perfect posture. Penguin N -100 -3 +10 -3 -100 • As the game advances the players body posture is Penguin H -100 +5 +20 +5 -100 compared with correct posture, as indicated by the guidance-feedback bar shown in Figure 5; the re- Finally, the game calculates the number of stars that the spective rewards or penalties are shown in Table II. player accumulates. On each less than perfect posture in the The penalties in each posture are divided by 100 to exercise, E, the initial score (100) points is diminished by one avert the player’s discouragement, and yet the rewards point for each rule pd that is not perfect on P d. Thus, at the end remain the same. of each exercise we calculate the proportion of the mistakes with the initial score and calculate the reward stars shown • The bonus postures are described in H HOLD; when below. While Table II qualifies each exercise repetition, table the grammar encounters these particular elements a countdown and a message is triggered to indicate that TABLE III. STAR REWARD FROM THE LEVELS OF THE GAMES Game Level 1 Level 2 Level 3 Level 4 Fisherman 3 stars 65% 2 stars 70% stars 75% stars 80% Spaceship 3 stars 65% 2 stars 70% stars 75% stars 80% Penguin 3 stars 50% 2 stars 55% stars 60% stars 65% III shows the qualification of the complete exercise session, and provides feedback and stimulates the player with reward sounds and pop up stars animated scene.

C. Feedback ACKNOWLEDGMENT The feedback ensures the patients awareness of his move- This work was supported in part by CONACYT, number ments. In the game the representation of the player is an avatar of the project 199769, Kaxan Media Group, and the UVM which imitates the movements of the player as accurately as therapy department, the Computer Science Department at the hardware allows. The guidance-feedback bar, attached to Superior Institute Technological of Chapala, and the Postgrad- the avatar at a specific joint, “fills” up to indicate the correct uate Computer science department of the Superior Institute movement and reflects the actual movement with a moving Technological of Zapopan. arrow. In effect, the comparison of the level of the bar’s filling and the arrow position is an indicator of the discrepancy be- REFERENCES tween the prescribed and the actual movement. The game also shows motivational messages (well done, excellent, marvelous) [1] David R Michael and Sandra L Chen. Serious games: Games that and instruction messages (hold, countdown, rest, fast, slow). In educate, train and inform. Muska and Lipman/Premier-Trade, 2005. addition, a number of distinct sounds are used as a feedback: an alert sound for inadequate postures, a reward sound to [2] Michael Zyda. From visual simulation to virtual reality to games. motivatea countdown sound when the player has reached the Computer, 38(9):2532, 2005. maximum posture/angle and should start moving back towards the initial position, and a hold sound, indicating that the player [3] B. Sawyer and P. Smith. Serious games taxonomy. Game Developer has reached a “hold” position and should maintain it. The Conference, 2008. therapist can see the grammar of the performed exercise in real time 6 through a web page. This is useful to be in-touch [4] WATTANASOONTORN, Voravika, et al. Serious games for health. with the patient and receive alerts of wrong postures during the Entertainment Computing, 2013, vol. 4, no 4, p. 231-247. exercise or just the confirmation that the patient has performed the exercise in the correct way. The therapist can review the [5] Lennart Nacke. Affective ludology: Scientific measurement of user exercise, change the configurations or correct postures through experience in interactive entertainment. 2009. the portal. The game requires an xml file with the descriptions of the postures and exercise prescribed to each patient. [6] M. Fraiwan, N. Khasawneh, A. Malkawi, M. Al-Jarrah, R. Alsadi, and S. Al-Momani, Therapy central: On the development of computer games VI. CONCLUSIONS for physiotherapy, in Innovations in Information Technology (IIT), 2013 9th International Conference on, 2013, pp. 2429. In this paper, we have described a grammar for specifying a family of rehabilitation exercises, and an environment that [7] R. A. Clark, A. L. Bryant, Y. Pua, P. McCrory, K. Bennell, and M. Hunt, uses fuzzy logic to recognize how well a player performs these Validity and reliability of the Nintendo Wii Balance Board for assessment exercises. of standing balance, Gait Posture, vol. 31, no. 3 pp. 307310, 2010. Our grammar defines the sequence of postures that one [8] Bhatti, Pamela T., et al. ”A prototype head-motion monitoring system has to move through, as well as the timing of the movement, for in-home vestibular rehabilitation therapy.” Journal of bioengineering including information on (a) whether a posture is transient or & biomedical science (2011). should be held, and (b) on how long the transition from one posture to the next should take. This information is readily [9] Cary, Francisco and Postolache, Octavian and Girao, Pedro Silva. Kinect available to rehabilitation therapists, which is a necessary Based System and Serious Game Motivating Approach for Physiotherapy prerequisite for it to be understandable and usable by these Assessment and Remote Session Monitoring. therapists. [10] Lange, Belinda, et al. ”Leveraging unencumbered full body control of Our environment includes the Kinect sensor, to recognize animated virtual characters for game-based rehabilitation.” Virtual and the joints of the person performing the exercise, and the Unity Mixed Reality-Systems and Applications. Springer Berlin Heidelberg, 3D environment to present a virtual game environment through 2011. 243-252. which to motivate and provide feedback. Our methods for recognizing the prescribed postures and the correctness of the [11] Fikar, Peter, Christian Schoenauer, and Hannes Kaufmann. ”The Sor- movement dynamics are based on fuzzy logic. In this environ- cerer’s Apprentice A serious game aiding rehabilitation in the context of ment, the exercise is performed with a game as the background, Subacromial Impingement Syndrome.” Pervasive Computing Technolo- where the game gestures align with the exercise posture and gies for Healthcare (PervasiveHealth), 2013 7th International Conference movements. Thus, the game mechanics and rewards provide a on. IEEE, 2013. mechanism for assessing exercise performance and motivating performance improvement. Our games treat the postures as [12] J. Smisek, M. Jancosek and T. Pajdla, ”3D with Kinect”, Computer being of the highest importance; care is taken to not allow the Vision Workshops (ICCV Workshops). 2011 IEEE International Confer- patient to continue the exercise when he reaches an incorrect ence on, pp. 1154-1160, 6-13 Nov. 2011. (and potentially dangerous) posture. The grammar is capable of recognizing both the correct postures and possibly dangerous [13] Obdrzalek, S., Kurillo, G., Ofli, F., Bajcsy, R., Seto, E., Jimison, postures. This overcomes the lack of feedback which is one H., & Pavel, M. (2012, August). Accuracy and robustness of Kinect of the main problems with rehabilitation at home. pose estimation in the context of coaching of elderly population. In Engineering in medicine and biology society (EMBC), 2012 annual 6http://transportepremier.com/rehab/pmlogin.php international conference of the IEEE (pp. 1188-1193). IEEE. [14] ZADEH, Lotfi A. Fuzzy sets. Information and control, 1965, vol. 8, no 3, p. 338-353. [15] GARCIA-GARCIA, Amanda; MENDEZ-VAZQUEZ, Andres. Learning fuzzy rules through ant optimization, LASSO and Dirichlet Mixtures. En Fuzzy Systems (FUZZ-IEEE), 2014 IEEE International Conference on. IEEE, 2014. p. 2558-2565. [16] AZIMI, Mehran. Skeletal Joint Smoothing White Paper. MSDN digital library, 2012. [17] GEORGE J, K. L. I. R.; BO, Yuan. Fuzzy sets and fuzzy logic, theory and applications. -, 2008. [18] LEPHART, Scott M., et al. The role of proprioception in the man- agement and rehabilitation of athletic injuries. The American journal of sports medicine, 1997, vol. 25, no 1, p. 130-137. [19] HOUGLUM, Peggy A. Therapeutic exercise for musculoskeletal in- juries. Human Kinetics Publishers, 2010.