Welcome back friends! It is time to take a deep dive back into our meta-analysis of combat robots! For this herculean numbers battle we would like to introduce our newest team member Luke Kaiser! Like the fumbling crew of brave Ulysses, we were led to victory over the data by his quick wit and tenacity. Without further ado, we present the analysis!
BACKGROUND
Greetings, Denkbots members and followers! There have already been some preliminary analyses conducted on the state of the battle robot metagame, but today we will focus on some specific questions:
- How do robot archetypes perform against each other?
- How do matchup trends evolve over time?
- How has the competitive landscape changed over time?
- Which robot archetypes are longest-lasting?
- What is “the best” robot archetype from a competitive standpoint?
The answers to these questions may be helpful in determining how effective the proposed Denkbot will be. Let’s get started!
MODEL DEVELOPMENT
As was done in the last competitive robot analysis, robot match and event data since 1994 was collected and tabulated. Three views of the data were employed:
- Matchups by event, showing winners, losers, and participation by year
- Event enrollment, showing durability and longevity of individual robots
- Robot-level analysis, breaking down wins and losses to produce weighted win rates
The integrity of these views depends on the reliability of the databases used to assemble them: proper tournament results and robot archetypes, in particular, are essential.
ANALYSIS
Win Rate
For the initial set of win rate analyses, box and whisker plots were constructed using increasingly specific cross sections. These results can be found below.
- Unfiltered Average Win Rate
As a first pass, the win rates for all robots in the match database were collected and grouped by archetype.
Figure 1. Average win percentage of all robot archetypes, for all robots on record
This result is not very helpful, as the lower quartile of many archetype win rates includes the absolute minimum value, and several archetypes range from 0-100%. This necessitates a refinement of the data.
- Filtered Average Win Rate
The average win rate for all robots on record is a 31%, which indicates that a notable portion of competing robots are stepping stones for superior robots. The quantity of “one and done” robots, who compete exactly once and lose, rapidly drops over time. If we aim to only consider “legitimate” competitors, we can set a minimum number of matches required to be included in the win percentage chart. Ideally, the win rate should be about 50%, which requires that the minimum number of matches be set at four. The resulting chart now appears:
Figure 2. Filtered average win percentage of all robot archetypes
This is better than the first pass, and now shows some trends. However, since we are only averaging win rates, robots with 4-0 records would have the same weight as robots with 40-1 records. So, we should employ a weighting system to ensure that averages are properly considered.
- Weighted & Filtered Average Win Rate
Although complicated weighted win metrics such as ELO produce excellent results with larger datasets, they cannot be employed here; these systems slowly converge towards their final value, and most robots do not have the match count to allow this to happen. Instead, we will employ a simplified model that damps out win or loss streaks while allowing robots with larger bodies of work to maintain their win percentages:
The constant, C, is determined through trial and error. With a value of 5, we see an acceptable dampening of win and loss streaks for robots with low match counts while keeping the integrity of robots with larger match counts. Implementing this formula with the average win percentage discovered in the prior chart, we achieve the following final box and whisker plot:
Figure 3. Weighted, filtered average win percentage of all robot archetypes
As expected, weighting the data in this way drives volatile win percentages towards the mean, allowing more representative data points (robots with higher match counts) to account for most of the differences between archetypes.
From this view, some initial conclusions can be drawn:
- The “other” archetype is the least successful, with the lowest median and maximum win percentages
- The “lift” archetype is the most successful by a slight margin, having the highest median and maximum win percentages
- Most archetypes have similar average win rates, suggesting that they have a similar number of advantageous and disadvantageous matchups
With these conclusions in mind, we can focus on the performance of archetypes against each other, rather than the pure win percentage of each archetype.
- Matchup Charts
From the same data used to assemble the above plots, a table of archetype performance against other archetypes was constructed.
Figure 4. Archetype win percentage against other archetypes
For a second view, we can apply the same minimum match count (4 matches) to the above table and produce a new set of values for comparison.
Figure 5. Filtered archetype win percentage against other archetypes
Increases or decreases in win percentages from the first to second table indicate that less-experienced robots tend to lose or win, respectively. The largest change takes place for the “grip” archetype, whose win rate leaps from a middling 53% to a tie for the lead. Remaining in the lead, the “lift” archetype is the only one with positive win percentage against the entire field.
Competitive Landscape
A table of percentage shares of the competitive landscape was created, grouped by archetype and year range.
Figure 6. Archetype percentage share of competitive landscape by year
From this chart, it is clear that “spin” robots have comprised most of the field since 1999. In fact, from 2003 to 2006, more than half of the field were spinners! Other consistent trends include the consistent shares owned by “lift” robots and the plummeting usage of “strike” robots. Although data for 2015 and 2016 is relatively sparse in comparison to prior years, there have been some major shifts: “flip” robots have suddenly disappeared from competition, the 2011-2014 “static” robot fad ended, and “grip” robots soared in popularity.
Overall Archetype Effectiveness
Using these percentages and the archetype matchup data in Figure 4, we can sum products to create an “effectiveness” metric for each archetype over the year-groups above.
Figure 7. Archetype effectiveness by year group
As was observed from the box and whisker plots, the “lift” archetype has a sizeable lead in terms of average effectiveness against the field since the inception of televised battle robots. All other archetypes, with the exception of the “other” archetype, are fairly even. Some conclusions emerge:
- If choosing a single functionality for your battle robot, the “lift” archetype is the most successful against the field
- “Spin” robots are over-represented in the field, considering their overall average performance against other archetypes
- Robot designers have correctly shied away from the “other” archetype after poor showings in the first few years of competitive robot battles
- The rise of “grip” robots as a percentage of the field may be due to the simultaneous decline of “strike” robots, which are “grip” robots’ weakest matchup
Durability and Longevity
One potential concern for robot designers the longevity of their design. If unable to face several matches while maintaining optimal functionality, or if major repairs are required after every tournament, the continued success of the design will be impossible.
Match and event counts were averaged by archetype in an effort to compare expected longevity.
Figure 8. Archetype effectiveness by year group
The average number of matches entered by each archetype may be easily confounded by high win rates, so event counts are also included to provide clarity. As could be expected, designs with more complicated moving parts show generally poorer durability. Interestingly, “lift” archetypes show greater longevity than “static” archetypes, but this could be due to the dominance of the “lift” archetype against the field; consistently ending bouts quickly could be the greatest path towards long-lasting robots.
SUMMARY
As always, feel free to join the conversation on our Facebook or Twitter or Instagram with your questions, thoughts, and feedback on these articles! Also, please feel free to leave your additions, corrections, and musings in the comments section below!