Beyond Tells - Poker Research Discussion, Limitations, Etc. Part 1
In every good form of research there is a discussion of the limitations and potential problems associated with the study and its data. This video discusses these things for Beyond Tells in a more casual way. I am going to go over some of the key limitations of Beyond Tells and then discuss how we dealt with them. This will be a multi-part video series to keep things short and manageable.
1. Dual Nature of the Study
The goal of the Beyond Tells Study was extremely practical. We intended to use the video data and insights to create a product. In fact the primary purpose of this pursuit was to create a highly lean and efficient method of approach for understanding nonverbal behavior at the poker table. Which we successfully did.
In the traditional sense this is definitely a conflict of interest but what we are doing isn’t so traditional. Beyond Tells didn’t have a hypothesis that we could prove or disprove. We used a method of approach very similar to what is called Grounding Theory, where you construct theories through coding, categorizing, and then analyzing data.
When it comes to reading human behavior in any domain, improvement doesn’t lie in x means y statements. It lies in the formation of an approach. The Beyond Tells training is essentially a systematic way of thinking about behavior at the poker table. I am not saying that an increase in blink rate is correlated with card strength. I am saying that for a specific player an increase in blink rate is correlated with strength primarily because they are applying less regulation than normal due to confidence and they tend to be over regulating in most of the hands that they play. These are two completely different statements. We create approaches, we don’t formulate conclusions that can be applied to everyone because quite frankly that is ridiculous.
However, people may question the conflict of interest of doing a study and profiting from it. Because of this, we currently offer a semi-open data set. We make our entire data set open to all members of the Beyond Tells Training. Creating Beyond Tells cost an incredible amount of money so for now that is the best that I can do. As our datasets grow through strategic partnerships we plan on making all of our video data public. Also, when I refer to data I am primarily referring to video data. In certain public videos we discuss insights from counting 60,000+ blinks and things like that. We will be publishing the raw data we used to reach those conclusions on the right side of the video.
I am trying to be as open of a book as possible without completely giving away what we worked so hard on. Finally, I am open to exchanging our datasets with any academic institutions or people doing thesis work because I am pretty sure it would be impossible to gather any of this data unless you somehow drugged your IRB.
2. Observation Bias for Players
A case definitely can be made for an observation bias. We were recording games with multiple cameras and players knew they were being recorded at all times. This can definitely create changes in behavior. For example, we had this player who was really trying to standardize his behavior as much as possible. Holding his hands in the same fashion, betting in the same style, and he was successful for about 45 minutes. After that this process of standardization seemed to become too much of a burden and he loosened up and started to display much more information.
You can also make an argument that a player’s anxiety is a facet of being watched, not card strength. A player has aces and is afraid of not playing them correctly which increases the amount of anxiety. If the player wasn’t being recorded there wouldn’t be that much anxiety. Even though this seems like a bad thing, it’s actually great for our purposes. The reason observation bias is great for us is that it creates changes in behavior. My goal is to collect as many changes in behavior as humanly possible so I can show people that there are so many different reasons behavior changes. If we were measuring the frequency and duration of anxiety at a poker table then this wouldn’t be good at all, but we are not. We are looking to understand the different sources of anxiety at the poker table and the different ways players can display that anxiety. This is also good because if you ever final table an event you have a mental schema where you can say, “Hmmm maybe the anxiety this player is displaying is due to the cameras and the stress of playing a final event.”
The more situations we create the more accurate of a mental model you have for interpreting behavior. Also, I personally believe that the behavioral information and tells found in this study are much fewer than the average due to the nature of the study. I think you find significantly more tells and useable information at an average game then the Beyond Tells study.
3. Sample Issues
Third is sampling issues. The goal of this study wasn’t to produce statistically significant “x means y” insights that are representative of the entire poker body. In fact that is the absolute worst way to approach human behavior because it is too complex to conform to this type of insight. The goal was to eventually create a systematic way of thinking about human behavior and tells at the table, and our sample had more than enough data for us to complete that goal. Also, our data sets are always expanding as we aspire to conduct more studies and form strategic relationships.
Still, there are a few issues that need to be addressed.
First, we don’t have longitudinal data on our players. It would be nice to see if specific tells hold up over the course of two or three years. Players and people do change and sometimes patterns don’t last forever but we would need to have the same players come in later to see. This is something that we will be doing but currently we don’t have the data.
Second, is financial comfort zones. Higher stakes would have produced more of an emotional response in certain players. For example, next year I want to take 4, $25/$50 regulars. and 4 people who only play $1/$2 and make like $30,000 dollars a year and give them $10k to sit down and play and see how that money shapes behavior.
Third, is size and diversity. I am sure we would have seen more insights with a larger and more culturally diverse sample, which is something we will do when we eventually start to study more variations of poker including mixed games.