class: center, middle, inverse, title-slide # Explaining Performance Ratings Using Proportional Odds Regression ### Keith McNulty --- class: left, middle, r-logo ## Note on source This document is a summary learning developed for the NY Strategic HR Analytics Meetup on 15 January 2021. It is based on material in the open source book *[Handbook of Regression Modeling in People Analytics](http://peopleanalytics-regression-book.org)*. Please consult this book for a deeper understanding/treatment. The code for this document is [here](https://github.com/keithmcnulty/peopleanalytics-regression-book/tree/master/presentations). ## Note on languages This document is coded in R. The model type used in this document (proportional odds logistic regression) is not currently well supported in Python. --- class: left, middle, r-logo ## Example data set: salespeople performance ratings Let's look at some data that relates to the performance ratings of sales people in an organization on a scale of 1 to 3 of increasing performance: ```r ## download data url <- "https://raw.githubusercontent.com/keithmcnulty/peopleanalytics-regression-book/master/presentations/employee_performance.csv" employee_performance <- read.csv(url) ## look at the data head(employee_performance) ``` ``` ## sales new_customers region gender rating ## 1 10.10 9 North M 2 ## 2 11.67 17 West F 3 ## 3 8.91 13 North M 3 ## 4 7.73 10 South F 2 ## 5 11.36 12 North M 2 ## 6 8.06 12 North M 2 ``` Sales is in $m. We are being asked which of the other four factors influence the performance rating and how do they influence it. --- class: left, middle, r-logo ## What does our data look like? The first data step in any modeling process is to make sure your data is in good shape. ```r ## look at the data types - are they appropriate? str(employee_performance) ``` ``` ## 'data.frame': 366 obs. of 5 variables: ## $ sales : num 10.1 11.67 8.91 7.73 11.36 ... ## $ new_customers: int 9 17 13 10 12 12 10 12 9 11 ... ## $ region : chr "North" "West" "North" "South" ... ## $ gender : chr "M" "F" "M" "F" ... ## $ rating : int 2 3 3 2 2 2 3 2 2 2 ... ``` --- class: left, middle, r-logo ## Adjusting data types for appropriate modeling We need to make sure that `region` and `gender` is understood as an **unordered** categorical column of data. We also need to make sure that the `rating` column is understood as an **ordered** categorical column. ```r ## unordered categories cat_columns <- c("region", "gender") employee_performance[cat_columns] <- lapply(employee_performance[cat_columns], as.factor) ## ordered categories employee_performance$rating <- ordered(employee_performance$rating, levels = 1:3) str(employee_performance) ``` ``` ## 'data.frame': 366 obs. of 5 variables: ## $ sales : num 10.1 11.67 8.91 7.73 11.36 ... ## $ new_customers: int 9 17 13 10 12 12 10 12 9 11 ... ## $ region : Factor w/ 4 levels "East","North",..: 2 4 2 3 2 2 2 2 1 2 ... ## $ gender : Factor w/ 2 levels "F","M": 2 1 2 1 2 2 2 1 2 2 ... ## $ rating : Ord.factor w/ 3 levels "1"<"2"<"3": 2 3 3 2 2 2 3 2 2 2 ... ``` --- class: left, middle, r-logo ## Quick visualizations of bivariate relationships in the data ```r library(GGally) ## create a pairplot GGally::ggpairs(employee_performance) ``` <img src="polr_files/figure-html/unnamed-chunk-4-1.png" style="display: block; margin: auto;" /> --- class: left, middle, r-logo ## We want to answer a few questions as precisely as we can 1. Which variables are meaningful in explaining employee performance? 2. To what extent does each meaningful variable influence performance? 3. How much of the entire variance of performance do these variables explain? --- class: left, middle, r-logo ## First we need to select the right type of model Through asking some questions about the data that we have, we can determine an appropriate model to use. **Question 1:** What type of outcome are we studying? *Ordered categories* **Question 2:** Can we assume that each input acts similarly on each level of the outcome? *Yes for now and we can check this later. This is called the proportional odds assumption*. Then we should use a proportional odds logistic regression model. This model will tell us how each input variable affects the *odds of someone having a higher performance rating*. See [Section 7.1](http://peopleanalytics-regression-book.org/ord-reg.html#when-to-use-it-2) for more details. --- class: left, middle, r-logo ## Run the model We can use the `polr()` function from the `MASS` package to run the model. ```r library(MASS) # formula for the model our_formula = "rating ~ ." # run model model <- polr(data = employee_performance, formula = our_formula) ``` Now we have model sitting ready to be viewed. --- class: left, middle, r-logo ## Viewing and clarifying the results I like to use the `broom` package to view model results in a tidy way and to easily add columns to the results. ```r library(dplyr) library(broom) (model_results <- tidy(model) %>% filter(coef.type == "coefficient")) ``` ``` ## # A tibble: 6 x 5 ## term estimate std.error statistic coef.type ## <chr> <dbl> <dbl> <dbl> <chr> ## 1 sales 0.385 0.0477 8.07 coefficient ## 2 new_customers 0.387 0.0413 9.37 coefficient ## 3 regionNorth 0.182 0.322 0.564 coefficient ## 4 regionSouth 0.0944 0.335 0.281 coefficient ## 5 regionWest 0.303 0.321 0.944 coefficient ## 6 genderM -0.0222 0.230 -0.0965 coefficient ``` See [Section 10.1.1](http://peopleanalytics-regression-book.org/alt-approaches.html#the-broom-package) for more information on tidy output and `broom`. --- class: left, middle, r-logo ## Are variables significant in explaining performance? To determine this we need to convert our `statistic` into a p-value, and determine if the p-value is less than an alpha which is usually 0.05. See [Section 3.3](http://peopleanalytics-regression-book.org/found-stats.html#hyp-tests). ```r ## add p-value (model_results <- model_results %>% dplyr::mutate( pval = (1 - pnorm(abs(statistic), 0, 1))*2 )) ``` ``` ## # A tibble: 6 x 6 ## term estimate std.error statistic coef.type pval ## <chr> <dbl> <dbl> <dbl> <chr> <dbl> ## 1 sales 0.385 0.0477 8.07 coefficient 6.66e-16 ## 2 new_customers 0.387 0.0413 9.37 coefficient 0. ## 3 regionNorth 0.182 0.322 0.564 coefficient 5.73e- 1 ## 4 regionSouth 0.0944 0.335 0.281 coefficient 7.78e- 1 ## 5 regionWest 0.303 0.321 0.944 coefficient 3.45e- 1 ## 6 genderM -0.0222 0.230 -0.0965 coefficient 9.23e- 1 ``` We can safely drop everything except `sales` and `new_customers`, because `region` and `gender` have no significant effect. --- class: left, middle, r-logo ## Simplify the model ```r simpler_formula <- "rating ~ sales + new_customers" ## create simpler model simpler_model <- polr(data = employee_performance, formula = simpler_formula) ## generate tidy results (simpler_results <- tidy(simpler_model) %>% filter(coef.type == "coefficient")) ``` ``` ## # A tibble: 2 x 5 ## term estimate std.error statistic coef.type ## <chr> <dbl> <dbl> <dbl> <chr> ## 1 sales 0.380 0.0471 8.07 coefficient ## 2 new_customers 0.389 0.0411 9.48 coefficient ``` --- class: left, middle, r-logo ## How does each variable affect performance? We need to take exponents of the `estimate` to get an interpretable *odds ratio*. See [Section 7.2](http://peopleanalytics-regression-book.org/ord-reg.html#modeling-ordinal-outcomes-under-the-assumption-of-proportional-odds). ```r ## create odds ratio column (simpler_results <- simpler_results %>% dplyr::mutate( odds_ratio = exp(estimate) )) ``` ``` ## # A tibble: 2 x 6 ## term estimate std.error statistic coef.type odds_ratio ## <chr> <dbl> <dbl> <dbl> <chr> <dbl> ## 1 sales 0.380 0.0471 8.07 coefficient 1.46 ## 2 new_customers 0.389 0.0411 9.48 coefficient 1.48 ``` The odds ratio is the multiplier of the odds of a higher category associated with a one unit increase in the input variable **assuming all other variables are equal**. Here's how we interpret this: 1. For individuals with similar new customers, each additional $1m of sales is associated with ~46% increase in the odds of higher performance. 2. For individuals with similar sales, each additional new customer is associated with ~48% increase in the odds of higher performance. --- class: left, middle, r-logo ## How much of the overall variance in performance is explained by this model? In a linear model we would use an `\(R^2\)` value to get a good point of view on this. In a logistic model there are several variants of this called *pseudo-* `\(R^2\)`. Let's look at a few popular variants. See [Section 5.3.2](http://peopleanalytics-regression-book.org/bin-log-reg.html#logistic-gof) for more information on these metrics. ```r library(DescTools) PseudoR2(simpler_model, which = c("CoxSnell", "Nagelkerke")) ``` ``` ## CoxSnell Nagelkerke ## 0.4956424 0.5600221 ``` This statistic should be interpreted and referenced with care. For example: >"It is estimated that sales and new customer acquisition explain somewhere around half of the variance in performance" --- class: left, middle, r-logo ## Graphing the model If you want to you can visualize this model because it has only 3 dimensions. Different surfaces can represent the probability of each performance level. See appendix for code. <div id="htmlwidget-3bdbb391848053b078f7" style="width:504px;height:504px;" class="plotly html-widget"></div> <script type="application/json" data-for="htmlwidget-3bdbb391848053b078f7">{"x":{"visdat":{"7498233e1442":["function () ","plotlyVisDat"]},"cur_data":"7498233e1442","attrs":{"7498233e1442":{"alpha_stroke":1,"sizes":[10,100],"spans":[1,20],"z":[0.0715620171520615,0.00188053224068062,0.0249031460820219,0.113958936977739,0.0146294604259406,0.0495006668415434,0.0172952496831005,0.101982063670543,0.0553701372711276,0.0286353533737232,0.0897430593852119,0.381040857785334,0.0152657317223527,0.103192867204601,0.104932407014721,0.0296427527901067,0.266247575270719,0.103769859708575,0.509314006107794,0.0266298238846291,0.282699629286711,0.654097432649424,0.162766243075019,0.150398894711399,0.259949957040793,0.212251593715925,0.372196391220142,0.057448960709763,0.202588500306648,0.437730305398845,0.0926351335838112,0.971609520758326,0.187294790632085,0.974117738287028,0.432715618656491,0.0702855963806175,0.188085992352521,0.0301540409612246,0.118107077704224,0.164375933598205,0.365115470004526,0.592009311110432,0.0259744668639446,0.11706715719689,0.141147454318196,0.235043983281054,0.46933332572311,0.823030496993827,0.184781363472043,0.00117432583547274,0.0299922694848059,0.159218757837724,0.585227558659841,0.832781399825857,0.0129571987832193,0.144317393460453,0.0317080426743303,0.0637205355674919,0.206299657969592,0.16767784643528,0.383163972562349,0.0806168816473305,0.0809522419543165,0.82796067007032,0.945483807352317,0.0272917848677388,0.944892747299785,0.228093942626926,0.117173655705196,0.11671140031201,0.317749021769596,0.0234757055595617,0.065581743257764,0.12546461319408,0.427462596615017,0.18803154487944,0.241572811245993,0.937314180898495,0.347590946258749,0.0267285770181804,0.552596419471839,0.630190129631175,0.457129805811976,0.50611592429787,0.40954318019814,0.704554263849527,0.502570723282779,0.018335403591733,0.0250294733761619,0.0275315244069706,0.843153996637047,0.0136560788113715,0.0775957514283955,0.199148411781054,0.032611787577169,0.152087183260045,0.619166143991322,0.0846798108800506,0.0404967321904806,0.0541176876712188,0.00309110899023074,0.128104719999777,0.91530839506666,0.418519384352381,0.0484380504286214,0.189250248341756,0.0907349540724051,0.470714857742446,0.0335853724549986,0.288340185983455,0.00597360473995239,0.750911136372912,0.0292083144983733,0.395160361503275,0.0935112221135267,0.225184710059878,0.134170139276031,0.0498109881373884,0.00260349683929211,0.0429853043592763,0.0972083986964295,0.0409688244745139,0.0242732692092338,0.361276545204006,0.120908149378535,0.31282317235566,0.919933852003429,0.016268683133182,0.0821419807409361,0.396486686906368,0.0144117860405348,0.0210972488583772,0.149167976562879,0.715022800158296,0.741815391648284,0.134289830293269,0.937314180898495,0.693057410306995,0.633489090461147,0.125351649052351,0.915602731792271,0.0673837713653499,0.0798014958327017,0.291545330742977,0.200529311059746,0.0350636667170784,0.298374497867956,0.0536069389050481,0.470971465730771,0.10415734281117,0.0789417787832669,0.00421750065856323,0.0194633194705809,0.0834021216524624,0.213639195253906,0.145131363813344,0.424246912468536,0.450874718216202,0.268754750132823,0.0348072340340365,0.182400430382472,0.18525326507341,0.00285452343896592,0.387009925462548,0.0138765861951083,0.364711504117751,0.909223065398229,0.35483913348207,0.119629805058195,0.057187825009484,0.285438809278953,0.0800812065728254,0.184835084043952,0.393098194166737,0.209719289613815,0.274570497562349,0.448306356627404,0.412890525885763,0.0302956994599348,0.71368094776635,0.0297523374480466,0.408040066270551,0.321738396067994,0.0220085372985732,0.620715541222944,0.0193186933300921,0.0230981889407189,0.650332438588845,0.0796997387770664,0.0120526103272924,0.015302100055115,0.00633950018045808,0.126301541054564,0.0607750529166081,0.941637269098563,0.176745505083995,0.0877568118738541,0.720695179079588,0.258686891064983,0.655583767234494,0.0960912594705447,0.0467149916034184,0.954779216763067,0.127451303591525,0.00285057961238908,0.0148857667693838,0.681839974925992,0.230291416161221,0.606042965799454,0.59167437719593,0.790544267189389,0.749031803188682,0.128414770390349,0.29076046379802,0.244751367557521,0.00376061483463486,0.0796735882225479,0.000140762151444735,0.0462252604578848,0.754325982225105,0.176693626038829,0.30639644459887,0.908306710658174,0.0748211075107219,0.0951355562654837,0.0173539224167515,0.833359842711622,0.516825516730261,0.82796067007032,0.15414702914493,0.0249118064250547,0.200751685926953,0.972436809230613,0.560700890500691,0.147028880267873,0.261346983178466,0.318127018627113,0.786740954277183,0.030643261010614,0.133289025168132,0.957798421418131,0.947549121456539,0.704125306070138,0.711343975918176,0.861662932825296,0.657218773891169,0.185618268253795,0.0907938088092116,0.802640829573628,0.225067247636981,0.64084296884816,0.969546980278544,0.971609520758326,0.833359842711622,0.493499469417927,0.0476319669210478,0.355392521312583,0.673533597921548,0.0281674950314162,0.763493877184007,0.326581016714932,0.347357423220818,0.00981012759419847,0.801597061630728,0.280671718143332,0.137170917669706,0.14925345651385,0.416332590493122,0.483393683115403,0.349948307719647,0.288055756172901,0.97233469320778,0.480891288280193,0.552508258735281,0.679963624706993,0.911814617693576,0.00694113620177551,0.112812017860033,0.202255939819293,0.0113919307149696,0.0687431332207902,0.706496606905883,0.697087498300635,0.934979601927662,0.582792703052768,0.82182138140185,0.674751924892024,0.41322667455138,0.497995339016597,0.292831594066261,0.218618836386294,0.835412610922479,0.659864404176161,0.754259895044401,0.345241030929678,0.173398702289149,0.472610231270027,0.918883033229126,0.918806231888176,0.236915233019941,0.262549389849083,0.824537142071744,0.698691129199146,0.0217643048080081,0.307939218438506,0.251655111975596,0.320753667704921,0.0086305219690546,0.601408570219266,0.0240611999688403,0.683487675483081,0.102585883289257,0.762117714907537,0.486411483244692,0.816184347956868,0.524250426075629,0.753620541423587,0.286571348957285,0.0581457505079596,0.378763826566289,0.467183063057757,0.0473810101119802,0.0197826308514016,0.605134623621325,0.95702310453966,0.00795145074556548,0.717550608284139,0.339870334160006,0.50110503526713,0.200751685926953,0.118036960862054,0.175790687295365,0.0108892709923423,0.0301540409612246,0.0283764494036284,0.0774965701076092,0.0293061961109773,0.723675627415955,0.0808226519032513,0.913221835948682,0.0181311768086587,0.784586267881161,0.0474597487932119,0.891292754906538,0.897400108280222,0.0276335264461476,0.312082799136738,0.000412978502671148,0.480891288280193,0.0242732692092338,0.547802419459438,0.054241516633076,0.184209181734962,0.73046725671358,0.700073709270469,0.367441845989725,0.854677631819004,0.969546980278544,0.340724080962962,0.27449947869881,0.816754211512021,0.164326960594434,0.0363372875848702,0.827322118759869,0.00522767644783282,0.745049360289867],"x":{},"y":{},"type":"mesh3d","name":"P(Low Performance)","inherit":true},"7498233e1442.1":{"alpha_stroke":1,"sizes":[10,100],"spans":[1,20],"z":[0.537214631134719,0.0347623079129929,0.315290653972141,0.60799746931415,0.215980816453447,0.463024372286208,0.244866126283177,0.594313419176758,0.486620524256723,0.344462931844991,0.575853541977183,0.544489770509453,0.223101994845477,0.595876642114914,0.598047625498132,0.351822248394902,0.613640097915885,0.596606414649074,0.445138065839599,0.329174702764859,0.605650906446027,0.320376837069843,0.634181174370076,0.630965653045564,0.616462090849283,0.632459258775151,0.550695682324172,0.494220625999114,0.63425373258297,0.502449808619474,0.580684028948291,0.0269452082257663,0.635795629541399,0.0245678965664929,0.506306736607161,0.533867809901534,0.635758815325911,0.355478953738409,0.611894970130886,0.634468750139473,0.555583533924591,0.374981083457699,0.323986327111744,0.610954940537897,0.627256308492553,0.626128947529709,0.477630721964152,0.166431025518718,0.635879147865216,0.0220110746447185,0.354327586620597,0.633445370810035,0.380857056566649,0.157370513627511,0.196541475894971,0.628664778675416,0.366280962177704,0.515044267909757,0.633634160147168,0.634971677710453,0.5429840817586,0.55840402194819,0.55910975661384,0.161851834434861,0.0516682628733198,0.334317439204853,0.0522267392516439,0.62834304266929,0.611052327211738,0.610627756792825,0.586120291330892,0.30327424817462,0.52066844326624,0.617883334863223,0.510321008958794,0.635761508008541,0.623844854918608,0.0593840684554119,0.567346326566668,0.329948096020891,0.408845884534212,0.341563697034217,0.487314416496829,0.44777664261845,0.523802625447391,0.275097284573658,0.450694069880453,0.255489176936853,0.316330132573545,0.336156161352734,0.147715817662761,0.20479624133611,0.5518036396606,0.634746487154517,0.372354531129893,0.631515857220636,0.351267972346882,0.566610383131003,0.419567895273731,0.481858752773401,0.0558194688028633,0.619766477374344,0.0801293088954236,0.51709164875875,0.458384735558442,0.635695649555374,0.577545485759687,0.476527041290524,0.37873280044864,0.602723545637041,0.102222465972443,0.232923541942358,0.348673983960851,0.534366249435609,0.582086782974454,0.62919936100543,0.623607203706253,0.464356878061257,0.047456293463459,0.432572071857912,0.58771200334937,0.422098509371976,0.310050195836222,0.558202077909253,0.614308590278562,0.589045158520495,0.0757735469464129,0.234033955057391,0.561571495685125,0.533402365203719,0.213510528528195,0.282087555644842,0.630540833506806,0.265617596159157,0.24123703023887,0.623676507793894,0.0593840684554119,0.285476144945466,0.338651413059854,0.617799757076601,0.0798522107733268,0.525891621391548,0.556666036874186,0.601020442056944,0.634558331563335,0.388103623836445,0.59730003254471,0.479875999207172,0.476321880238191,0.597091082038272,0.554799149917646,0.0745527396128295,0.266625455675536,0.564108988945432,0.632154591019879,0.629002697630039,0.512764900282843,0.492230533900112,0.612478651573978,0.386504566144364,0.635910224325572,0.635867390922408,0.0517813332584109,0.540241372715747,0.207361388634211,0.555860138735197,0.0858558700690433,0.562541382708114,0.613228170326455,0.493286142777649,0.604240639903501,0.557265786265816,0.635877901919213,0.535860605225541,0.632985675221382,0.609704983293968,0.494239415629286,0.521310238597958,0.356482938101427,0.266834209058164,0.35261037531024,0.52491777624927,0.583712838197142,0.290383796570785,0.349906669441313,0.265219161563743,0.300010932263345,0.323725692393422,0.556446925809889,0.185567403889851,0.223504606945075,0.107765059705828,0.618494792806688,0.50563925198024,0.0553020563519955,0.6357880665316,0.572351031209493,0.260469833825939,0.617011119317076,0.319053565633803,0.586061323385994,0.450600826260845,0.0428802548546497,0.619312792591129,0.0517137066402835,0.218867130724321,0.2955690001866,0.627668186796221,0.362762501506068,0.3752717311419,0.19650185008465,0.234642703723188,0.619978950177659,0.601440089658341,0.622665270522715,0.0670506914254735,0.556390536406708,0.00269335621509898,0.448327667424974,0.229797825197547,0.635785623950408,0.59277854689911,0.0867178060747995,0.545338298021762,0.584615848373178,0.245474643609622,0.156832548343454,0.43891595839268,0.161851834434861,0.632137520376415,0.315362037314777,0.634526813856214,0.0261611674354282,0.401940552711368,0.629753887706228,0.615848130575043,0.585893644976805,0.200010146957082,0.358929614656848,0.623089506420615,0.0400238475976348,0.0497165018664564,0.275485139530781,0.268952012618361,0.130447419790999,0.31759714336358,0.635857044314997,0.577644707496092,0.185326141865705,0.62923303039635,0.332146244443617,0.02889962148596,0.0269452082257663,0.156832548343454,0.458121869454658,0.454784144154876,0.56217096151298,0.303020233916593,0.340973701069142,0.221394184859636,0.580745694973511,0.567499658586966,0.156866044372622,0.18629148139635,0.6066808829829,0.625272608292432,0.630570988744457,0.518734583438196,0.466331207925244,0.565793258556453,0.602873314478499,0.0262579489766488,0.468352896056623,0.408920824240166,0.297253876418864,0.083417659880702,0.116719489783158,0.606848429568155,0.634304848984865,0.177338493024321,0.52969310818727,0.273340475457932,0.281841862209512,0.0615875820490644,0.382961989024916,0.167553458758376,0.301928557069627,0.521059267378901,0.454447298673924,0.600329117355135,0.630969839941507,0.154923000491283,0.315238745937692,0.229858340127931,0.568884990971602,0.635579289922975,0.475010481472408,0.0767633215368974,0.0768356561513132,0.625493900824501,0.615314133288464,0.165032058283031,0.280394546219839,0.288182619443384,0.59189109519701,0.619959138659677,0.584310235172967,0.140851381221561,0.366806118452009,0.308263974295262,0.294088635324364,0.595098415865871,0.222656725213582,0.463887180759846,0.172783182958621,0.432732557048351,0.230443738389223,0.603651270856407,0.496686395978447,0.546097852768983,0.479345586313665,0.453648601955142,0.269708232569414,0.363555707437458,0.0407574518640477,0.131325823285513,0.263324530488918,0.572365066983301,0.45189786735072,0.634526813856214,0.611832349347268,0.635739150576997,0.170952871671612,0.355478953738409,0.342537742088606,0.551579350018581,0.349386643872708,0.257761961251119,0.558837669079737,0.0820933617928807,0.253430667809734,0.201996508123193,0.45400563515033,0.102702165924319,0.0969686429777862,0.3369346933481,0.589480097599851,0.00785887845570342,0.468352896056623,0.310050195836222,0.412915312607599,0.482335856307421,0.635890938254131,0.251583701829765,0.279146198959403,0.553985837260853,0.136970489433876,0.02889962148596,0.571815316276847,0.609739507270422,0.172254735575037,0.634460417196882,0.39588622940381,0.162445148730356,0.0906895759371897,0.238283236022745],"x":{},"y":{},"type":"mesh3d","name":"P(Middle Performance)","inherit":true},"7498233e1442.2":{"alpha_stroke":1,"sizes":[10,100],"spans":[1,20],"z":[0.391223351713219,0.963357159846327,0.659806199945837,0.278043593708111,0.769389723120612,0.487474960872248,0.737838624033722,0.303704517152699,0.458009338472149,0.626901714781286,0.334403398637605,0.0744693717052123,0.761632273432171,0.300930490680485,0.297019967487147,0.618534998814991,0.120112326813397,0.299623725642351,0.0455479280526074,0.644195473350512,0.111649464267262,0.0255257302807329,0.203052582554905,0.218635452243037,0.123587952109925,0.155289147508924,0.0771079264556858,0.448330413291123,0.163157767110382,0.0598198859816813,0.326680837467897,0.0014452710159073,0.176909579826516,0.00131436514647887,0.0609776447363476,0.395846593717848,0.176155192321568,0.614367005300367,0.26999795216489,0.201155316262322,0.0793009960708829,0.0330096054318693,0.650039206024311,0.271977902265213,0.231596237189251,0.138827069189237,0.0530359523127381,0.0105384774874553,0.179339488662741,0.976814599519809,0.615680143894597,0.207335871352241,0.0339153847735102,0.00984808654663194,0.790501325321809,0.227017827864131,0.602010995147966,0.421235196522751,0.160066181883239,0.197350475854267,0.0738519456790507,0.360979096404479,0.359938001431843,0.0101874954948185,0.00284792977436366,0.638390775927408,0.00288051344857077,0.143563014703785,0.271774017083065,0.272660842895165,0.0961306868995122,0.673250046265818,0.413749813475996,0.256652051942697,0.0622163944261889,0.176206947112019,0.134582333835398,0.00330175064609273,0.0850627271745827,0.643323326960929,0.0385576959939488,0.0282461733346083,0.055555777691195,0.0461074330836797,0.0666541943544695,0.0203484515768146,0.0467352068367672,0.726175419471415,0.658640394050293,0.636312314240295,0.0091301857001912,0.781547679852518,0.370600608911005,0.166105101064429,0.595033681292938,0.216396959519319,0.0295658836617961,0.348709805988946,0.539935372535788,0.46402355955538,0.941089422206906,0.252128802625879,0.00456229603791647,0.0643889668888682,0.493177214012937,0.17505410210287,0.331719560167908,0.0527581009670302,0.587681827096361,0.108936268379504,0.891803929287605,0.0161653216847299,0.622117701540776,0.0704733890611164,0.324401994912019,0.145615928934692,0.242222657017716,0.485832133801354,0.949940209697249,0.524442623782812,0.3150795979542,0.53693266615351,0.665676534954544,0.0805213768867409,0.264783260342903,0.098131669123845,0.00429260105015761,0.749697361809427,0.356286523573939,0.0701109478899133,0.77207768543127,0.696815195496781,0.220291189930315,0.0193596036825467,0.0169475781128462,0.242033661912837,0.00330175064609273,0.0214664447475389,0.0278594964789998,0.256848593871048,0.00454505743440226,0.406724607243102,0.363532467293113,0.107434227200078,0.16491235737692,0.576832709446476,0.104325469587334,0.46651706188778,0.0527066540310379,0.298751575150559,0.366259071299087,0.921229759728607,0.713911224853883,0.352488889402106,0.154206213726215,0.225865938556617,0.0629881872486211,0.0568947478836864,0.118766598293199,0.5786881998216,0.181689345291957,0.178879344004182,0.945364143302623,0.072748701821705,0.778762025170681,0.0794283571470519,0.00492106453272756,0.0826194838098162,0.26714202461535,0.449526032212867,0.110320550817546,0.362653007161359,0.179287014036835,0.0710412006077223,0.157295035164803,0.115724519143684,0.0574542277433103,0.065799235516279,0.613221362438638,0.0194848431754865,0.617637287241713,0.0670421574801796,0.0945487657348641,0.687607666130642,0.0293777893357426,0.715462145106165,0.676890878795936,0.025941869017733,0.363853335413044,0.802379985782857,0.76119329299981,0.885895440113713,0.255203666138748,0.433585695103152,0.00306067454944192,0.187466428384405,0.339892156916653,0.0188349870944727,0.124301989617941,0.0253626671317031,0.317847417143461,0.502684182135736,0.00234052838228282,0.253235903817346,0.945435713747327,0.766247102506295,0.0225910248874086,0.142040397042557,0.0311945326944784,0.0330538916621703,0.0129538827259611,0.0163254930881296,0.251606279431992,0.107799446543639,0.132583361919764,0.929188693739892,0.363935875370744,0.997165881633456,0.505447072117141,0.0158761925773481,0.187520750010763,0.10082500850202,0.00497548326702657,0.379840594467516,0.320248595361338,0.737171433973626,0.00980760894492383,0.0442585248770587,0.0101874954948185,0.213715450478655,0.659726156260168,0.164721500216833,0.00140202333395878,0.0373585567879412,0.223217232025899,0.122804886246492,0.0959793363960822,0.0132488987657343,0.610427124332539,0.243621468411253,0.00217773098423391,0.00273437667700505,0.020389554399081,0.0197040114634623,0.00788964738370468,0.0251840827452502,0.178524687431208,0.331561483694696,0.0120330285606661,0.145699721966668,0.0270107867082232,0.00155339823549627,0.0014452710159073,0.00980760894492383,0.0483786611274148,0.497583888924076,0.0824365171744373,0.0234461681618586,0.630858803899442,0.0151119379563578,0.0926732883115571,0.0851429181922154,0.83332382803318,0.0121114569729223,0.112647398873768,0.237556474037862,0.220175554741692,0.0649328260686824,0.0502751089593526,0.0842584337238999,0.1090709293486,0.00140735781557122,0.0507558156631843,0.0385709170245532,0.0227824988741426,0.00476772242572221,0.876339374015066,0.280339552571812,0.163439211195842,0.811269576260709,0.40156375859194,0.0201629176361854,0.0210706394898523,0.00343281602327339,0.0342453079223158,0.0106251598397741,0.023319518038349,0.065714058069719,0.0475573623094794,0.106839288578604,0.150411323672199,0.00966438858623775,0.0248968498861475,0.0158817648276672,0.08587397809872,0.191022007787876,0.0523792872575647,0.00435364523397674,0.00435811196051095,0.137590866155558,0.122136476862453,0.0104307996452251,0.0209143245810144,0.690053075748608,0.100169686364483,0.128385749364727,0.0949360971221124,0.850518096809385,0.0317853113287248,0.667674825735898,0.022423689192555,0.302315700844872,0.0152255598788811,0.0497013359954624,0.0110324690845114,0.0430170168760197,0.0159357201871901,0.109777380186308,0.445167853513593,0.0751383206647281,0.053471350628578,0.498970387932878,0.710509136579184,0.0313096689412173,0.00221944359629189,0.860722725968921,0.0191248612269433,0.0877645988566927,0.0469970973821499,0.164721500216833,0.270130689790679,0.188470162127638,0.818157857336046,0.614367005300367,0.629085808507766,0.37092407987381,0.621307160016315,0.018562411332926,0.360339679017012,0.00468480225843726,0.728438155381607,0.0134172239956457,0.498534616056459,0.00600507916914261,0.00563124874199172,0.635431780205753,0.0984371032634108,0.991728143041625,0.0507558156631843,0.665676534954544,0.0392822679329639,0.463422627059503,0.179899880010907,0.0179490414566551,0.0207800917701284,0.0785723167494217,0.00835187874711973,0.00155339823549627,0.087460602760191,0.115761014030768,0.0109910529129426,0.201212622208684,0.56777648301132,0.0102327325097752,0.904082747614977,0.0166674036873886],"x":{},"y":{},"type":"mesh3d","name":"P(High Performance)","inherit":true}},"layout":{"margin":{"b":40,"l":60,"t":25,"r":10},"scene":{"xaxis":{"title":"sales"},"yaxis":{"title":"new_customers"},"zaxis":{"title":"Probability"},"aspectmode":"cube"},"hovermode":"closest","showlegend":true,"legend":{"yanchor":"top","y":0.25}},"source":"A","config":{"showSendToCloud":false},"data":[{"colorbar":{"title":"","ticklen":2,"len":0.25,"lenmode":"fraction","y":1,"yanchor":"top"},"colorscale":[["0","rgba(68,1,84,1)"],["0.0416666666666667","rgba(70,19,97,1)"],["0.0833333333333333","rgba(72,32,111,1)"],["0.125","rgba(71,45,122,1)"],["0.166666666666667","rgba(68,58,128,1)"],["0.208333333333333","rgba(64,70,135,1)"],["0.25","rgba(60,82,138,1)"],["0.291666666666667","rgba(56,93,140,1)"],["0.333333333333333","rgba(49,104,142,1)"],["0.375","rgba(46,114,142,1)"],["0.416666666666667","rgba(42,123,142,1)"],["0.458333333333333","rgba(38,133,141,1)"],["0.5","rgba(37,144,140,1)"],["0.541666666666667","rgba(33,154,138,1)"],["0.583333333333333","rgba(39,164,133,1)"],["0.625","rgba(47,174,127,1)"],["0.666666666666667","rgba(53,183,121,1)"],["0.708333333333333","rgba(79,191,110,1)"],["0.75","rgba(98,199,98,1)"],["0.791666666666667","rgba(119,207,85,1)"],["0.833333333333333","rgba(147,214,70,1)"],["0.875","rgba(172,220,52,1)"],["0.916666666666667","rgba(199,225,42,1)"],["0.958333333333333","rgba(226,228,40,1)"],["1","rgba(253,231,37,1)"]],"showscale":true,"z":[0.0715620171520615,0.00188053224068062,0.0249031460820219,0.113958936977739,0.0146294604259406,0.0495006668415434,0.0172952496831005,0.101982063670543,0.0553701372711276,0.0286353533737232,0.0897430593852119,0.381040857785334,0.0152657317223527,0.103192867204601,0.104932407014721,0.0296427527901067,0.266247575270719,0.103769859708575,0.509314006107794,0.0266298238846291,0.282699629286711,0.654097432649424,0.162766243075019,0.150398894711399,0.259949957040793,0.212251593715925,0.372196391220142,0.057448960709763,0.202588500306648,0.437730305398845,0.0926351335838112,0.971609520758326,0.187294790632085,0.974117738287028,0.432715618656491,0.0702855963806175,0.188085992352521,0.0301540409612246,0.118107077704224,0.164375933598205,0.365115470004526,0.592009311110432,0.0259744668639446,0.11706715719689,0.141147454318196,0.235043983281054,0.46933332572311,0.823030496993827,0.184781363472043,0.00117432583547274,0.0299922694848059,0.159218757837724,0.585227558659841,0.832781399825857,0.0129571987832193,0.144317393460453,0.0317080426743303,0.0637205355674919,0.206299657969592,0.16767784643528,0.383163972562349,0.0806168816473305,0.0809522419543165,0.82796067007032,0.945483807352317,0.0272917848677388,0.944892747299785,0.228093942626926,0.117173655705196,0.11671140031201,0.317749021769596,0.0234757055595617,0.065581743257764,0.12546461319408,0.427462596615017,0.18803154487944,0.241572811245993,0.937314180898495,0.347590946258749,0.0267285770181804,0.552596419471839,0.630190129631175,0.457129805811976,0.50611592429787,0.40954318019814,0.704554263849527,0.502570723282779,0.018335403591733,0.0250294733761619,0.0275315244069706,0.843153996637047,0.0136560788113715,0.0775957514283955,0.199148411781054,0.032611787577169,0.152087183260045,0.619166143991322,0.0846798108800506,0.0404967321904806,0.0541176876712188,0.00309110899023074,0.128104719999777,0.91530839506666,0.418519384352381,0.0484380504286214,0.189250248341756,0.0907349540724051,0.470714857742446,0.0335853724549986,0.288340185983455,0.00597360473995239,0.750911136372912,0.0292083144983733,0.395160361503275,0.0935112221135267,0.225184710059878,0.134170139276031,0.0498109881373884,0.00260349683929211,0.0429853043592763,0.0972083986964295,0.0409688244745139,0.0242732692092338,0.361276545204006,0.120908149378535,0.31282317235566,0.919933852003429,0.016268683133182,0.0821419807409361,0.396486686906368,0.0144117860405348,0.0210972488583772,0.149167976562879,0.715022800158296,0.741815391648284,0.134289830293269,0.937314180898495,0.693057410306995,0.633489090461147,0.125351649052351,0.915602731792271,0.0673837713653499,0.0798014958327017,0.291545330742977,0.200529311059746,0.0350636667170784,0.298374497867956,0.0536069389050481,0.470971465730771,0.10415734281117,0.0789417787832669,0.00421750065856323,0.0194633194705809,0.0834021216524624,0.213639195253906,0.145131363813344,0.424246912468536,0.450874718216202,0.268754750132823,0.0348072340340365,0.182400430382472,0.18525326507341,0.00285452343896592,0.387009925462548,0.0138765861951083,0.364711504117751,0.909223065398229,0.35483913348207,0.119629805058195,0.057187825009484,0.285438809278953,0.0800812065728254,0.184835084043952,0.393098194166737,0.209719289613815,0.274570497562349,0.448306356627404,0.412890525885763,0.0302956994599348,0.71368094776635,0.0297523374480466,0.408040066270551,0.321738396067994,0.0220085372985732,0.620715541222944,0.0193186933300921,0.0230981889407189,0.650332438588845,0.0796997387770664,0.0120526103272924,0.015302100055115,0.00633950018045808,0.126301541054564,0.0607750529166081,0.941637269098563,0.176745505083995,0.0877568118738541,0.720695179079588,0.258686891064983,0.655583767234494,0.0960912594705447,0.0467149916034184,0.954779216763067,0.127451303591525,0.00285057961238908,0.0148857667693838,0.681839974925992,0.230291416161221,0.606042965799454,0.59167437719593,0.790544267189389,0.749031803188682,0.128414770390349,0.29076046379802,0.244751367557521,0.00376061483463486,0.0796735882225479,0.000140762151444735,0.0462252604578848,0.754325982225105,0.176693626038829,0.30639644459887,0.908306710658174,0.0748211075107219,0.0951355562654837,0.0173539224167515,0.833359842711622,0.516825516730261,0.82796067007032,0.15414702914493,0.0249118064250547,0.200751685926953,0.972436809230613,0.560700890500691,0.147028880267873,0.261346983178466,0.318127018627113,0.786740954277183,0.030643261010614,0.133289025168132,0.957798421418131,0.947549121456539,0.704125306070138,0.711343975918176,0.861662932825296,0.657218773891169,0.185618268253795,0.0907938088092116,0.802640829573628,0.225067247636981,0.64084296884816,0.969546980278544,0.971609520758326,0.833359842711622,0.493499469417927,0.0476319669210478,0.355392521312583,0.673533597921548,0.0281674950314162,0.763493877184007,0.326581016714932,0.347357423220818,0.00981012759419847,0.801597061630728,0.280671718143332,0.137170917669706,0.14925345651385,0.416332590493122,0.483393683115403,0.349948307719647,0.288055756172901,0.97233469320778,0.480891288280193,0.552508258735281,0.679963624706993,0.911814617693576,0.00694113620177551,0.112812017860033,0.202255939819293,0.0113919307149696,0.0687431332207902,0.706496606905883,0.697087498300635,0.934979601927662,0.582792703052768,0.82182138140185,0.674751924892024,0.41322667455138,0.497995339016597,0.292831594066261,0.218618836386294,0.835412610922479,0.659864404176161,0.754259895044401,0.345241030929678,0.173398702289149,0.472610231270027,0.918883033229126,0.918806231888176,0.236915233019941,0.262549389849083,0.824537142071744,0.698691129199146,0.0217643048080081,0.307939218438506,0.251655111975596,0.320753667704921,0.0086305219690546,0.601408570219266,0.0240611999688403,0.683487675483081,0.102585883289257,0.762117714907537,0.486411483244692,0.816184347956868,0.524250426075629,0.753620541423587,0.286571348957285,0.0581457505079596,0.378763826566289,0.467183063057757,0.0473810101119802,0.0197826308514016,0.605134623621325,0.95702310453966,0.00795145074556548,0.717550608284139,0.339870334160006,0.50110503526713,0.200751685926953,0.118036960862054,0.175790687295365,0.0108892709923423,0.0301540409612246,0.0283764494036284,0.0774965701076092,0.0293061961109773,0.723675627415955,0.0808226519032513,0.913221835948682,0.0181311768086587,0.784586267881161,0.0474597487932119,0.891292754906538,0.897400108280222,0.0276335264461476,0.312082799136738,0.000412978502671148,0.480891288280193,0.0242732692092338,0.547802419459438,0.054241516633076,0.184209181734962,0.73046725671358,0.700073709270469,0.367441845989725,0.854677631819004,0.969546980278544,0.340724080962962,0.27449947869881,0.816754211512021,0.164326960594434,0.0363372875848702,0.827322118759869,0.00522767644783282,0.745049360289867],"x":[10.1,11.67,8.91,7.73,11.36,8.06,12.96,6.01,10.82,10.58,11.5,5.66,12.27,10.07,6.95,11.51,7.05,9.03,5.31,11.8,8.88,5.78,5.62,6.89,11.23,9.88,8.83,8.67,5.94,7.09,6.29,2.26,7.22,2.01,6.12,7.08,8.23,10.44,6.6,8.66,8.91,7.5,9.82,7.65,11.18,5.44,2.66,2.39,6.24,12.91,6.36,8.76,6.55,2.21,10.66,4.97,9.28,8.38,5.88,6.55,6.66,6.69,12.82,2.3,2,10.71,2.03,7.59,5.6,10.73,5.37,12.14,11.37,9.49,7.2,5.16,9.44,2.39,6.04,11.79,6.9,5.03,5.86,4.32,6.37,5.17,2.31,11.78,9.92,11.71,5.08,10.52,9.87,7.02,8.18,10.95,4.13,9.62,10.66,9.86,10.36,7.38,2.22,8.32,8.12,8.21,6.35,6.74,8.1,6.76,8.62,3.53,11.55,4.48,8.31,6.61,7.24,10.09,12.86,11.52,6.15,5.51,12.05,7.93,6.53,5.43,5.13,12.1,7.66,8.56,11.4,10.38,11.01,6.06,4.68,5.19,2.39,4.29,7.04,11.54,2.21,10.27,9.79,9.79,4.95,10.03,8.68,10.91,4.69,12.09,6.75,9.54,11.62,8.64,4.74,7,3.14,6.95,8.04,10.05,11.4,11.35,10.57,4.57,8.43,4.82,2.42,6.98,11.68,10.73,4.75,9.78,9.31,6.55,9.92,10.01,4.93,5.31,8.38,4.03,11.5,7.41,10.44,11.29,6.16,11.64,11.16,4.8,8.77,12.9,11.24,10.51,9.47,10.56,2.19,8.43,7.47,7.01,9.2,7.81,5.16,8.22,3.53,11.49,9.55,10.29,2.38,5.51,8.37,6.48,6.01,4.58,9.42,9.8,5.3,11.89,5.7,15.42,11.32,8.6,5.36,8.58,5.52,8.95,8.26,9.88,5.27,2.16,2.3,5.79,11.98,5.97,2.18,5.79,6.96,8.14,9.46,6.07,11.42,7.26,3.34,3.94,9.27,4.06,6.74,4.72,10.32,12.49,4.79,11.73,7.98,2.45,2.26,5.27,6.5,9.19,5.95,2.48,6.53,3.35,4.24,8.09,11.4,2.76,9.93,9.22,5.89,7.32,7.63,8.06,5.74,2.19,8.68,3.83,4.45,3.36,10.27,7.76,10.04,13.05,9.19,9.24,4.24,4.54,7.6,7.53,6.56,6.33,8.5,8.75,6.71,2.16,7.76,5.53,4.02,5.42,6.72,3.12,5.17,7.46,7.1,4.41,4.22,11.32,5.49,7.25,4.31,11.74,5.35,11.05,2.36,8.04,3.37,2.48,7.63,7.2,8.61,8.83,9.66,9.78,4.73,5.11,12.6,8.38,3.39,9.91,3.98,6.13,6.42,5.97,11.72,6.4,13.17,10.44,6.51,8.85,8.47,3.9,8.73,2.29,11.81,5.08,9.2,3.97,3.8,11.7,8.51,15.66,8.68,12.05,3.88,8.83,6.25,3.81,6.25,7.86,3.82,2.45,6.12,6.94,7.62,5.59,11.98,7.43,11.02,8.73],"y":[9,17,13,10,12,12,10,12,9,11,7,8,11,8,11,10,8,9,7,10,6,5,11,10,4,6,5,11,10,6,12,1,9,1,7,12,8,11,11,8,5,4,12,10,6,10,10,6,10,17,15,8,5,6,13,12,12,11,10,10,7,12,6,6,3,11,3,8,12,7,9,10,8,8,6,11,6,3,8,10,5,6,7,8,7,5,10,11,12,10,3,13,9,9,13,6,7,9,10,10,17,10,4,5,12,8,12,6,13,8,17,6,10,9,10,9,10,10,15,9,12,15,10,6,11,9,1,11,11,5,12,12,6,4,5,12,3,6,4,6,4,9,9,5,11,11,6,9,8,6,12,17,11,10,11,10,10,6,7,11,5,5,17,9,15,9,4,7,6,9,10,9,7,7,6,5,8,8,13,6,10,6,4,11,5,11,11,6,10,11,12,15,8,9,3,8,11,3,6,3,13,12,1,6,18,13,8,10,3,5,3,5,8,5,10,15,13,20,9,1,11,6,1,10,10,13,3,10,6,11,10,10,1,6,10,7,5,3,10,10,1,1,1,6,1,6,6,6,4,4,3,1,1,3,6,11,8,8,15,6,10,6,13,6,5,8,11,6,5,6,9,1,4,8,6,3,15,10,6,11,10,1,6,1,4,1,4,7,4,6,9,6,3,4,10,11,6,3,1,8,8,4,6,11,9,8,10,13,6,11,8,10,6,10,1,5,1,6,10,4,8,15,10,3,1,15,6,8,6,10,6,10,11,11,15,10,13,6,10,4,11,4,11,3,3,10,6,17,4,10,8,11,10,6,4,6,4,1,8,8,1,11,9,1,15,1],"type":"mesh3d","name":"P(Low Performance)","frame":null},{"colorbar":{"title":"","ticklen":2,"len":0.25,"lenmode":"fraction","y":0.75,"yanchor":"top"},"colorscale":[["0","rgba(68,1,84,1)"],["0.0416666666666667","rgba(70,19,97,1)"],["0.0833333333333333","rgba(72,32,111,1)"],["0.125","rgba(71,45,122,1)"],["0.166666666666667","rgba(68,58,128,1)"],["0.208333333333333","rgba(64,70,135,1)"],["0.25","rgba(60,82,138,1)"],["0.291666666666667","rgba(56,93,140,1)"],["0.333333333333333","rgba(49,104,142,1)"],["0.375","rgba(46,114,142,1)"],["0.416666666666667","rgba(42,123,142,1)"],["0.458333333333333","rgba(38,133,141,1)"],["0.5","rgba(37,144,140,1)"],["0.541666666666667","rgba(33,154,138,1)"],["0.583333333333333","rgba(39,164,133,1)"],["0.625","rgba(47,174,127,1)"],["0.666666666666667","rgba(53,183,121,1)"],["0.708333333333333","rgba(79,191,110,1)"],["0.75","rgba(98,199,98,1)"],["0.791666666666667","rgba(119,207,85,1)"],["0.833333333333333","rgba(147,214,70,1)"],["0.875","rgba(172,220,52,1)"],["0.916666666666667","rgba(199,225,42,1)"],["0.958333333333333","rgba(226,228,40,1)"],["1","rgba(253,231,37,1)"]],"showscale":true,"z":[0.537214631134719,0.0347623079129929,0.315290653972141,0.60799746931415,0.215980816453447,0.463024372286208,0.244866126283177,0.594313419176758,0.486620524256723,0.344462931844991,0.575853541977183,0.544489770509453,0.223101994845477,0.595876642114914,0.598047625498132,0.351822248394902,0.613640097915885,0.596606414649074,0.445138065839599,0.329174702764859,0.605650906446027,0.320376837069843,0.634181174370076,0.630965653045564,0.616462090849283,0.632459258775151,0.550695682324172,0.494220625999114,0.63425373258297,0.502449808619474,0.580684028948291,0.0269452082257663,0.635795629541399,0.0245678965664929,0.506306736607161,0.533867809901534,0.635758815325911,0.355478953738409,0.611894970130886,0.634468750139473,0.555583533924591,0.374981083457699,0.323986327111744,0.610954940537897,0.627256308492553,0.626128947529709,0.477630721964152,0.166431025518718,0.635879147865216,0.0220110746447185,0.354327586620597,0.633445370810035,0.380857056566649,0.157370513627511,0.196541475894971,0.628664778675416,0.366280962177704,0.515044267909757,0.633634160147168,0.634971677710453,0.5429840817586,0.55840402194819,0.55910975661384,0.161851834434861,0.0516682628733198,0.334317439204853,0.0522267392516439,0.62834304266929,0.611052327211738,0.610627756792825,0.586120291330892,0.30327424817462,0.52066844326624,0.617883334863223,0.510321008958794,0.635761508008541,0.623844854918608,0.0593840684554119,0.567346326566668,0.329948096020891,0.408845884534212,0.341563697034217,0.487314416496829,0.44777664261845,0.523802625447391,0.275097284573658,0.450694069880453,0.255489176936853,0.316330132573545,0.336156161352734,0.147715817662761,0.20479624133611,0.5518036396606,0.634746487154517,0.372354531129893,0.631515857220636,0.351267972346882,0.566610383131003,0.419567895273731,0.481858752773401,0.0558194688028633,0.619766477374344,0.0801293088954236,0.51709164875875,0.458384735558442,0.635695649555374,0.577545485759687,0.476527041290524,0.37873280044864,0.602723545637041,0.102222465972443,0.232923541942358,0.348673983960851,0.534366249435609,0.582086782974454,0.62919936100543,0.623607203706253,0.464356878061257,0.047456293463459,0.432572071857912,0.58771200334937,0.422098509371976,0.310050195836222,0.558202077909253,0.614308590278562,0.589045158520495,0.0757735469464129,0.234033955057391,0.561571495685125,0.533402365203719,0.213510528528195,0.282087555644842,0.630540833506806,0.265617596159157,0.24123703023887,0.623676507793894,0.0593840684554119,0.285476144945466,0.338651413059854,0.617799757076601,0.0798522107733268,0.525891621391548,0.556666036874186,0.601020442056944,0.634558331563335,0.388103623836445,0.59730003254471,0.479875999207172,0.476321880238191,0.597091082038272,0.554799149917646,0.0745527396128295,0.266625455675536,0.564108988945432,0.632154591019879,0.629002697630039,0.512764900282843,0.492230533900112,0.612478651573978,0.386504566144364,0.635910224325572,0.635867390922408,0.0517813332584109,0.540241372715747,0.207361388634211,0.555860138735197,0.0858558700690433,0.562541382708114,0.613228170326455,0.493286142777649,0.604240639903501,0.557265786265816,0.635877901919213,0.535860605225541,0.632985675221382,0.609704983293968,0.494239415629286,0.521310238597958,0.356482938101427,0.266834209058164,0.35261037531024,0.52491777624927,0.583712838197142,0.290383796570785,0.349906669441313,0.265219161563743,0.300010932263345,0.323725692393422,0.556446925809889,0.185567403889851,0.223504606945075,0.107765059705828,0.618494792806688,0.50563925198024,0.0553020563519955,0.6357880665316,0.572351031209493,0.260469833825939,0.617011119317076,0.319053565633803,0.586061323385994,0.450600826260845,0.0428802548546497,0.619312792591129,0.0517137066402835,0.218867130724321,0.2955690001866,0.627668186796221,0.362762501506068,0.3752717311419,0.19650185008465,0.234642703723188,0.619978950177659,0.601440089658341,0.622665270522715,0.0670506914254735,0.556390536406708,0.00269335621509898,0.448327667424974,0.229797825197547,0.635785623950408,0.59277854689911,0.0867178060747995,0.545338298021762,0.584615848373178,0.245474643609622,0.156832548343454,0.43891595839268,0.161851834434861,0.632137520376415,0.315362037314777,0.634526813856214,0.0261611674354282,0.401940552711368,0.629753887706228,0.615848130575043,0.585893644976805,0.200010146957082,0.358929614656848,0.623089506420615,0.0400238475976348,0.0497165018664564,0.275485139530781,0.268952012618361,0.130447419790999,0.31759714336358,0.635857044314997,0.577644707496092,0.185326141865705,0.62923303039635,0.332146244443617,0.02889962148596,0.0269452082257663,0.156832548343454,0.458121869454658,0.454784144154876,0.56217096151298,0.303020233916593,0.340973701069142,0.221394184859636,0.580745694973511,0.567499658586966,0.156866044372622,0.18629148139635,0.6066808829829,0.625272608292432,0.630570988744457,0.518734583438196,0.466331207925244,0.565793258556453,0.602873314478499,0.0262579489766488,0.468352896056623,0.408920824240166,0.297253876418864,0.083417659880702,0.116719489783158,0.606848429568155,0.634304848984865,0.177338493024321,0.52969310818727,0.273340475457932,0.281841862209512,0.0615875820490644,0.382961989024916,0.167553458758376,0.301928557069627,0.521059267378901,0.454447298673924,0.600329117355135,0.630969839941507,0.154923000491283,0.315238745937692,0.229858340127931,0.568884990971602,0.635579289922975,0.475010481472408,0.0767633215368974,0.0768356561513132,0.625493900824501,0.615314133288464,0.165032058283031,0.280394546219839,0.288182619443384,0.59189109519701,0.619959138659677,0.584310235172967,0.140851381221561,0.366806118452009,0.308263974295262,0.294088635324364,0.595098415865871,0.222656725213582,0.463887180759846,0.172783182958621,0.432732557048351,0.230443738389223,0.603651270856407,0.496686395978447,0.546097852768983,0.479345586313665,0.453648601955142,0.269708232569414,0.363555707437458,0.0407574518640477,0.131325823285513,0.263324530488918,0.572365066983301,0.45189786735072,0.634526813856214,0.611832349347268,0.635739150576997,0.170952871671612,0.355478953738409,0.342537742088606,0.551579350018581,0.349386643872708,0.257761961251119,0.558837669079737,0.0820933617928807,0.253430667809734,0.201996508123193,0.45400563515033,0.102702165924319,0.0969686429777862,0.3369346933481,0.589480097599851,0.00785887845570342,0.468352896056623,0.310050195836222,0.412915312607599,0.482335856307421,0.635890938254131,0.251583701829765,0.279146198959403,0.553985837260853,0.136970489433876,0.02889962148596,0.571815316276847,0.609739507270422,0.172254735575037,0.634460417196882,0.39588622940381,0.162445148730356,0.0906895759371897,0.238283236022745],"x":[10.1,11.67,8.91,7.73,11.36,8.06,12.96,6.01,10.82,10.58,11.5,5.66,12.27,10.07,6.95,11.51,7.05,9.03,5.31,11.8,8.88,5.78,5.62,6.89,11.23,9.88,8.83,8.67,5.94,7.09,6.29,2.26,7.22,2.01,6.12,7.08,8.23,10.44,6.6,8.66,8.91,7.5,9.82,7.65,11.18,5.44,2.66,2.39,6.24,12.91,6.36,8.76,6.55,2.21,10.66,4.97,9.28,8.38,5.88,6.55,6.66,6.69,12.82,2.3,2,10.71,2.03,7.59,5.6,10.73,5.37,12.14,11.37,9.49,7.2,5.16,9.44,2.39,6.04,11.79,6.9,5.03,5.86,4.32,6.37,5.17,2.31,11.78,9.92,11.71,5.08,10.52,9.87,7.02,8.18,10.95,4.13,9.62,10.66,9.86,10.36,7.38,2.22,8.32,8.12,8.21,6.35,6.74,8.1,6.76,8.62,3.53,11.55,4.48,8.31,6.61,7.24,10.09,12.86,11.52,6.15,5.51,12.05,7.93,6.53,5.43,5.13,12.1,7.66,8.56,11.4,10.38,11.01,6.06,4.68,5.19,2.39,4.29,7.04,11.54,2.21,10.27,9.79,9.79,4.95,10.03,8.68,10.91,4.69,12.09,6.75,9.54,11.62,8.64,4.74,7,3.14,6.95,8.04,10.05,11.4,11.35,10.57,4.57,8.43,4.82,2.42,6.98,11.68,10.73,4.75,9.78,9.31,6.55,9.92,10.01,4.93,5.31,8.38,4.03,11.5,7.41,10.44,11.29,6.16,11.64,11.16,4.8,8.77,12.9,11.24,10.51,9.47,10.56,2.19,8.43,7.47,7.01,9.2,7.81,5.16,8.22,3.53,11.49,9.55,10.29,2.38,5.51,8.37,6.48,6.01,4.58,9.42,9.8,5.3,11.89,5.7,15.42,11.32,8.6,5.36,8.58,5.52,8.95,8.26,9.88,5.27,2.16,2.3,5.79,11.98,5.97,2.18,5.79,6.96,8.14,9.46,6.07,11.42,7.26,3.34,3.94,9.27,4.06,6.74,4.72,10.32,12.49,4.79,11.73,7.98,2.45,2.26,5.27,6.5,9.19,5.95,2.48,6.53,3.35,4.24,8.09,11.4,2.76,9.93,9.22,5.89,7.32,7.63,8.06,5.74,2.19,8.68,3.83,4.45,3.36,10.27,7.76,10.04,13.05,9.19,9.24,4.24,4.54,7.6,7.53,6.56,6.33,8.5,8.75,6.71,2.16,7.76,5.53,4.02,5.42,6.72,3.12,5.17,7.46,7.1,4.41,4.22,11.32,5.49,7.25,4.31,11.74,5.35,11.05,2.36,8.04,3.37,2.48,7.63,7.2,8.61,8.83,9.66,9.78,4.73,5.11,12.6,8.38,3.39,9.91,3.98,6.13,6.42,5.97,11.72,6.4,13.17,10.44,6.51,8.85,8.47,3.9,8.73,2.29,11.81,5.08,9.2,3.97,3.8,11.7,8.51,15.66,8.68,12.05,3.88,8.83,6.25,3.81,6.25,7.86,3.82,2.45,6.12,6.94,7.62,5.59,11.98,7.43,11.02,8.73],"y":[9,17,13,10,12,12,10,12,9,11,7,8,11,8,11,10,8,9,7,10,6,5,11,10,4,6,5,11,10,6,12,1,9,1,7,12,8,11,11,8,5,4,12,10,6,10,10,6,10,17,15,8,5,6,13,12,12,11,10,10,7,12,6,6,3,11,3,8,12,7,9,10,8,8,6,11,6,3,8,10,5,6,7,8,7,5,10,11,12,10,3,13,9,9,13,6,7,9,10,10,17,10,4,5,12,8,12,6,13,8,17,6,10,9,10,9,10,10,15,9,12,15,10,6,11,9,1,11,11,5,12,12,6,4,5,12,3,6,4,6,4,9,9,5,11,11,6,9,8,6,12,17,11,10,11,10,10,6,7,11,5,5,17,9,15,9,4,7,6,9,10,9,7,7,6,5,8,8,13,6,10,6,4,11,5,11,11,6,10,11,12,15,8,9,3,8,11,3,6,3,13,12,1,6,18,13,8,10,3,5,3,5,8,5,10,15,13,20,9,1,11,6,1,10,10,13,3,10,6,11,10,10,1,6,10,7,5,3,10,10,1,1,1,6,1,6,6,6,4,4,3,1,1,3,6,11,8,8,15,6,10,6,13,6,5,8,11,6,5,6,9,1,4,8,6,3,15,10,6,11,10,1,6,1,4,1,4,7,4,6,9,6,3,4,10,11,6,3,1,8,8,4,6,11,9,8,10,13,6,11,8,10,6,10,1,5,1,6,10,4,8,15,10,3,1,15,6,8,6,10,6,10,11,11,15,10,13,6,10,4,11,4,11,3,3,10,6,17,4,10,8,11,10,6,4,6,4,1,8,8,1,11,9,1,15,1],"type":"mesh3d","name":"P(Middle Performance)","frame":null},{"colorbar":{"title":"","ticklen":2,"len":0.25,"lenmode":"fraction","y":0.5,"yanchor":"top"},"colorscale":[["0","rgba(68,1,84,1)"],["0.0416666666666667","rgba(70,19,97,1)"],["0.0833333333333333","rgba(72,32,111,1)"],["0.125","rgba(71,45,122,1)"],["0.166666666666667","rgba(68,58,128,1)"],["0.208333333333333","rgba(64,70,135,1)"],["0.25","rgba(60,82,138,1)"],["0.291666666666667","rgba(56,93,140,1)"],["0.333333333333333","rgba(49,104,142,1)"],["0.375","rgba(46,114,142,1)"],["0.416666666666667","rgba(42,123,142,1)"],["0.458333333333333","rgba(38,133,141,1)"],["0.5","rgba(37,144,140,1)"],["0.541666666666667","rgba(33,154,138,1)"],["0.583333333333333","rgba(39,164,133,1)"],["0.625","rgba(47,174,127,1)"],["0.666666666666667","rgba(53,183,121,1)"],["0.708333333333333","rgba(79,191,110,1)"],["0.75","rgba(98,199,98,1)"],["0.791666666666667","rgba(119,207,85,1)"],["0.833333333333333","rgba(147,214,70,1)"],["0.875","rgba(172,220,52,1)"],["0.916666666666667","rgba(199,225,42,1)"],["0.958333333333333","rgba(226,228,40,1)"],["1","rgba(253,231,37,1)"]],"showscale":true,"z":[0.391223351713219,0.963357159846327,0.659806199945837,0.278043593708111,0.769389723120612,0.487474960872248,0.737838624033722,0.303704517152699,0.458009338472149,0.626901714781286,0.334403398637605,0.0744693717052123,0.761632273432171,0.300930490680485,0.297019967487147,0.618534998814991,0.120112326813397,0.299623725642351,0.0455479280526074,0.644195473350512,0.111649464267262,0.0255257302807329,0.203052582554905,0.218635452243037,0.123587952109925,0.155289147508924,0.0771079264556858,0.448330413291123,0.163157767110382,0.0598198859816813,0.326680837467897,0.0014452710159073,0.176909579826516,0.00131436514647887,0.0609776447363476,0.395846593717848,0.176155192321568,0.614367005300367,0.26999795216489,0.201155316262322,0.0793009960708829,0.0330096054318693,0.650039206024311,0.271977902265213,0.231596237189251,0.138827069189237,0.0530359523127381,0.0105384774874553,0.179339488662741,0.976814599519809,0.615680143894597,0.207335871352241,0.0339153847735102,0.00984808654663194,0.790501325321809,0.227017827864131,0.602010995147966,0.421235196522751,0.160066181883239,0.197350475854267,0.0738519456790507,0.360979096404479,0.359938001431843,0.0101874954948185,0.00284792977436366,0.638390775927408,0.00288051344857077,0.143563014703785,0.271774017083065,0.272660842895165,0.0961306868995122,0.673250046265818,0.413749813475996,0.256652051942697,0.0622163944261889,0.176206947112019,0.134582333835398,0.00330175064609273,0.0850627271745827,0.643323326960929,0.0385576959939488,0.0282461733346083,0.055555777691195,0.0461074330836797,0.0666541943544695,0.0203484515768146,0.0467352068367672,0.726175419471415,0.658640394050293,0.636312314240295,0.0091301857001912,0.781547679852518,0.370600608911005,0.166105101064429,0.595033681292938,0.216396959519319,0.0295658836617961,0.348709805988946,0.539935372535788,0.46402355955538,0.941089422206906,0.252128802625879,0.00456229603791647,0.0643889668888682,0.493177214012937,0.17505410210287,0.331719560167908,0.0527581009670302,0.587681827096361,0.108936268379504,0.891803929287605,0.0161653216847299,0.622117701540776,0.0704733890611164,0.324401994912019,0.145615928934692,0.242222657017716,0.485832133801354,0.949940209697249,0.524442623782812,0.3150795979542,0.53693266615351,0.665676534954544,0.0805213768867409,0.264783260342903,0.098131669123845,0.00429260105015761,0.749697361809427,0.356286523573939,0.0701109478899133,0.77207768543127,0.696815195496781,0.220291189930315,0.0193596036825467,0.0169475781128462,0.242033661912837,0.00330175064609273,0.0214664447475389,0.0278594964789998,0.256848593871048,0.00454505743440226,0.406724607243102,0.363532467293113,0.107434227200078,0.16491235737692,0.576832709446476,0.104325469587334,0.46651706188778,0.0527066540310379,0.298751575150559,0.366259071299087,0.921229759728607,0.713911224853883,0.352488889402106,0.154206213726215,0.225865938556617,0.0629881872486211,0.0568947478836864,0.118766598293199,0.5786881998216,0.181689345291957,0.178879344004182,0.945364143302623,0.072748701821705,0.778762025170681,0.0794283571470519,0.00492106453272756,0.0826194838098162,0.26714202461535,0.449526032212867,0.110320550817546,0.362653007161359,0.179287014036835,0.0710412006077223,0.157295035164803,0.115724519143684,0.0574542277433103,0.065799235516279,0.613221362438638,0.0194848431754865,0.617637287241713,0.0670421574801796,0.0945487657348641,0.687607666130642,0.0293777893357426,0.715462145106165,0.676890878795936,0.025941869017733,0.363853335413044,0.802379985782857,0.76119329299981,0.885895440113713,0.255203666138748,0.433585695103152,0.00306067454944192,0.187466428384405,0.339892156916653,0.0188349870944727,0.124301989617941,0.0253626671317031,0.317847417143461,0.502684182135736,0.00234052838228282,0.253235903817346,0.945435713747327,0.766247102506295,0.0225910248874086,0.142040397042557,0.0311945326944784,0.0330538916621703,0.0129538827259611,0.0163254930881296,0.251606279431992,0.107799446543639,0.132583361919764,0.929188693739892,0.363935875370744,0.997165881633456,0.505447072117141,0.0158761925773481,0.187520750010763,0.10082500850202,0.00497548326702657,0.379840594467516,0.320248595361338,0.737171433973626,0.00980760894492383,0.0442585248770587,0.0101874954948185,0.213715450478655,0.659726156260168,0.164721500216833,0.00140202333395878,0.0373585567879412,0.223217232025899,0.122804886246492,0.0959793363960822,0.0132488987657343,0.610427124332539,0.243621468411253,0.00217773098423391,0.00273437667700505,0.020389554399081,0.0197040114634623,0.00788964738370468,0.0251840827452502,0.178524687431208,0.331561483694696,0.0120330285606661,0.145699721966668,0.0270107867082232,0.00155339823549627,0.0014452710159073,0.00980760894492383,0.0483786611274148,0.497583888924076,0.0824365171744373,0.0234461681618586,0.630858803899442,0.0151119379563578,0.0926732883115571,0.0851429181922154,0.83332382803318,0.0121114569729223,0.112647398873768,0.237556474037862,0.220175554741692,0.0649328260686824,0.0502751089593526,0.0842584337238999,0.1090709293486,0.00140735781557122,0.0507558156631843,0.0385709170245532,0.0227824988741426,0.00476772242572221,0.876339374015066,0.280339552571812,0.163439211195842,0.811269576260709,0.40156375859194,0.0201629176361854,0.0210706394898523,0.00343281602327339,0.0342453079223158,0.0106251598397741,0.023319518038349,0.065714058069719,0.0475573623094794,0.106839288578604,0.150411323672199,0.00966438858623775,0.0248968498861475,0.0158817648276672,0.08587397809872,0.191022007787876,0.0523792872575647,0.00435364523397674,0.00435811196051095,0.137590866155558,0.122136476862453,0.0104307996452251,0.0209143245810144,0.690053075748608,0.100169686364483,0.128385749364727,0.0949360971221124,0.850518096809385,0.0317853113287248,0.667674825735898,0.022423689192555,0.302315700844872,0.0152255598788811,0.0497013359954624,0.0110324690845114,0.0430170168760197,0.0159357201871901,0.109777380186308,0.445167853513593,0.0751383206647281,0.053471350628578,0.498970387932878,0.710509136579184,0.0313096689412173,0.00221944359629189,0.860722725968921,0.0191248612269433,0.0877645988566927,0.0469970973821499,0.164721500216833,0.270130689790679,0.188470162127638,0.818157857336046,0.614367005300367,0.629085808507766,0.37092407987381,0.621307160016315,0.018562411332926,0.360339679017012,0.00468480225843726,0.728438155381607,0.0134172239956457,0.498534616056459,0.00600507916914261,0.00563124874199172,0.635431780205753,0.0984371032634108,0.991728143041625,0.0507558156631843,0.665676534954544,0.0392822679329639,0.463422627059503,0.179899880010907,0.0179490414566551,0.0207800917701284,0.0785723167494217,0.00835187874711973,0.00155339823549627,0.087460602760191,0.115761014030768,0.0109910529129426,0.201212622208684,0.56777648301132,0.0102327325097752,0.904082747614977,0.0166674036873886],"x":[10.1,11.67,8.91,7.73,11.36,8.06,12.96,6.01,10.82,10.58,11.5,5.66,12.27,10.07,6.95,11.51,7.05,9.03,5.31,11.8,8.88,5.78,5.62,6.89,11.23,9.88,8.83,8.67,5.94,7.09,6.29,2.26,7.22,2.01,6.12,7.08,8.23,10.44,6.6,8.66,8.91,7.5,9.82,7.65,11.18,5.44,2.66,2.39,6.24,12.91,6.36,8.76,6.55,2.21,10.66,4.97,9.28,8.38,5.88,6.55,6.66,6.69,12.82,2.3,2,10.71,2.03,7.59,5.6,10.73,5.37,12.14,11.37,9.49,7.2,5.16,9.44,2.39,6.04,11.79,6.9,5.03,5.86,4.32,6.37,5.17,2.31,11.78,9.92,11.71,5.08,10.52,9.87,7.02,8.18,10.95,4.13,9.62,10.66,9.86,10.36,7.38,2.22,8.32,8.12,8.21,6.35,6.74,8.1,6.76,8.62,3.53,11.55,4.48,8.31,6.61,7.24,10.09,12.86,11.52,6.15,5.51,12.05,7.93,6.53,5.43,5.13,12.1,7.66,8.56,11.4,10.38,11.01,6.06,4.68,5.19,2.39,4.29,7.04,11.54,2.21,10.27,9.79,9.79,4.95,10.03,8.68,10.91,4.69,12.09,6.75,9.54,11.62,8.64,4.74,7,3.14,6.95,8.04,10.05,11.4,11.35,10.57,4.57,8.43,4.82,2.42,6.98,11.68,10.73,4.75,9.78,9.31,6.55,9.92,10.01,4.93,5.31,8.38,4.03,11.5,7.41,10.44,11.29,6.16,11.64,11.16,4.8,8.77,12.9,11.24,10.51,9.47,10.56,2.19,8.43,7.47,7.01,9.2,7.81,5.16,8.22,3.53,11.49,9.55,10.29,2.38,5.51,8.37,6.48,6.01,4.58,9.42,9.8,5.3,11.89,5.7,15.42,11.32,8.6,5.36,8.58,5.52,8.95,8.26,9.88,5.27,2.16,2.3,5.79,11.98,5.97,2.18,5.79,6.96,8.14,9.46,6.07,11.42,7.26,3.34,3.94,9.27,4.06,6.74,4.72,10.32,12.49,4.79,11.73,7.98,2.45,2.26,5.27,6.5,9.19,5.95,2.48,6.53,3.35,4.24,8.09,11.4,2.76,9.93,9.22,5.89,7.32,7.63,8.06,5.74,2.19,8.68,3.83,4.45,3.36,10.27,7.76,10.04,13.05,9.19,9.24,4.24,4.54,7.6,7.53,6.56,6.33,8.5,8.75,6.71,2.16,7.76,5.53,4.02,5.42,6.72,3.12,5.17,7.46,7.1,4.41,4.22,11.32,5.49,7.25,4.31,11.74,5.35,11.05,2.36,8.04,3.37,2.48,7.63,7.2,8.61,8.83,9.66,9.78,4.73,5.11,12.6,8.38,3.39,9.91,3.98,6.13,6.42,5.97,11.72,6.4,13.17,10.44,6.51,8.85,8.47,3.9,8.73,2.29,11.81,5.08,9.2,3.97,3.8,11.7,8.51,15.66,8.68,12.05,3.88,8.83,6.25,3.81,6.25,7.86,3.82,2.45,6.12,6.94,7.62,5.59,11.98,7.43,11.02,8.73],"y":[9,17,13,10,12,12,10,12,9,11,7,8,11,8,11,10,8,9,7,10,6,5,11,10,4,6,5,11,10,6,12,1,9,1,7,12,8,11,11,8,5,4,12,10,6,10,10,6,10,17,15,8,5,6,13,12,12,11,10,10,7,12,6,6,3,11,3,8,12,7,9,10,8,8,6,11,6,3,8,10,5,6,7,8,7,5,10,11,12,10,3,13,9,9,13,6,7,9,10,10,17,10,4,5,12,8,12,6,13,8,17,6,10,9,10,9,10,10,15,9,12,15,10,6,11,9,1,11,11,5,12,12,6,4,5,12,3,6,4,6,4,9,9,5,11,11,6,9,8,6,12,17,11,10,11,10,10,6,7,11,5,5,17,9,15,9,4,7,6,9,10,9,7,7,6,5,8,8,13,6,10,6,4,11,5,11,11,6,10,11,12,15,8,9,3,8,11,3,6,3,13,12,1,6,18,13,8,10,3,5,3,5,8,5,10,15,13,20,9,1,11,6,1,10,10,13,3,10,6,11,10,10,1,6,10,7,5,3,10,10,1,1,1,6,1,6,6,6,4,4,3,1,1,3,6,11,8,8,15,6,10,6,13,6,5,8,11,6,5,6,9,1,4,8,6,3,15,10,6,11,10,1,6,1,4,1,4,7,4,6,9,6,3,4,10,11,6,3,1,8,8,4,6,11,9,8,10,13,6,11,8,10,6,10,1,5,1,6,10,4,8,15,10,3,1,15,6,8,6,10,6,10,11,11,15,10,13,6,10,4,11,4,11,3,3,10,6,17,4,10,8,11,10,6,4,6,4,1,8,8,1,11,9,1,15,1],"type":"mesh3d","name":"P(High Performance)","frame":null}],"highlight":{"on":"plotly_click","persistent":false,"dynamic":false,"selectize":false,"opacityDim":0.2,"selected":{"opacity":1},"debounce":0},"shinyEvents":["plotly_hover","plotly_click","plotly_selected","plotly_relayout","plotly_brushed","plotly_brushing","plotly_clickannotation","plotly_doubleclick","plotly_deselect","plotly_afterplot","plotly_sunburstclick"],"base_url":"https://plot.ly"},"evals":[],"jsHooks":[]}</script> --- class: left, middle, r-logo ## Which chapters of the book should I read to fully understand this model? 1. Chapter 3 on foundational statistics 2. Chapter 4 on the foundations of regression 3. Chapter 5 on logistic regression 4. Chapter 7 on proportional odds --- class: center, middle, r-logo # Thank you! Questions? --- class: left, middle, r-logo ## Appendix: Checking our assumption of proportional odds See [Section 7.3](http://peopleanalytics-regression-book.org/ord-reg.html#testing-the-proportional-odds-assumption) for more info on how to test your proportional odds assumption. The quickest way is to do a Brant-Wald test on your model, looking for p-values less than 0.05 to indicate failure of the test. ```r library(brant) brant(simpler_model) ``` ``` ## -------------------------------------------- ## Test for X2 df probability ## -------------------------------------------- ## Omnibus 5.52 2 0.06 ## sales 3.87 1 0.05 ## new_customers 2.29 1 0.13 ## -------------------------------------------- ## ## H0: Parallel Regression Assumption holds ``` Here we can see that we pass the Brant test, but only just. If the Brant test is failed, this indicates that you would need to model each level of the rating separately using different binomial models. --- class: left, middle, r-logo ## Appendix: Testing goodness of fit of the model The goodness of fit test is an indicator of whether we have used an appropriate model to explain the data. For a proportional odds model, this test fails if the p-value is less than 0.05. Here is an example which confirms a good fit for our model. See [Section 7.3.2](http://peopleanalytics-regression-book.org/ord-reg.html#model-diagnostics) for further information. ```r library(generalhoslem) lipsitz.test(simpler_model) ``` ``` ## ## Lipsitz goodness of fit test for ordinal response models ## ## data: formula: rating ~ sales + new_customers ## LR statistic = 12.366, df = 9, p-value = 0.1935 ``` --- class: left, middle, r-logo ## Appendix: Code for 3D-plot ```r library(plotly) plot_ly(data = employee_performance) %>% add_trace(z = simpler_model$fitted.values[ , 1], x = ~sales, y = ~new_customers, type = "mesh3d", name = "P(Low Performance)") %>% add_trace(z = simpler_model$fitted.values[ , 2], x = ~sales, y = ~new_customers, type = "mesh3d", name = "P(Middle Performance)") %>% add_trace(z = simpler_model$fitted.values[ , 3], x = ~sales, y = ~new_customers, type = "mesh3d", name = "P(High Performance)") %>% layout(scene = list(xaxis = list(title = 'sales'), yaxis = list(title = 'new_customers'), zaxis = list(title = 'Probability'), aspectmode='cube')) ```