NCAA Rule Change on Tournament Selection Criteria

[Kuiper]

MASSIVE change recommended by the NCAA Division III Championships Committee on NCAA Tournament Selection Criteria for all team sports.  This is kind of a "throw out everything you think you knew about how teams are seeded and Pool C bids are handed out" rule.  It could dramatically influence how coaches construct their schedules in the future, but because it is supposed to go into effect for Fall 2024 under the recommendation, there will be teams advantaged or disadvantaged by the change right now if it is approved.

Here is the Recommendation:

(1) Recommendation. That the NCAA Power Index (NPI) be adopted as the selection metric
for selecting non-automatic qualifiers for the championship field for Division III team
sports.
(2) Effective date. September 1, 2024.

What does that mean?

Effectively, they will replace human selection committees with an algorithm.

What changes and what is in the algorithm (quotes from the FAQ section in the December release)?

They are basically recommending using the PairWise system in place already for Hockey to ensure more consistency in selection criteria. 

Most of the current selection criteria will no longer be used: head-to-head competition;
results versus common Division III opponents; results versus ranked Division III teams; and
secondary criteria. Winning percentage and strength of schedule will remain.

The selection criteria database will continue to use winning percentage and strength of
schedule along with a home/away multiplier, quality win bonus (QWB) and overtime results
(in applicable sports).

The home/away multiplier can be used to increase the weight of away wins and home losses
while decreasing the weight of home wins and away losses. The QWB provides a bonus for
wins against top-level teams, the degree of which is determined by the sport committee.

Soccer could adopt different weights on these criteria than in other sports

The weight of winning percentage versus strength of schedule is often referred to as one of the
"dials" that a sport committee can adjust, placing any subjectivity within the criteria
themselves. For example, a sport committee could set the dials to 70/30; 75/25, etc. Other
"dials" a sport committee can also choose to apply are home and away multiplier, quality win
bonus (QWB), and overtime win/loss weights.

What are the key differences in this change?

1.  Regional Advisory Committees less important

RACs lose almost all of their power.  They basically make sure there aren't mistakes (like North Central (MN)'s results being assigned to North Central (IL) or vice versa.

2.  Regional Rankings are just for fun

They will still be published, but the rankings will be based upon the algorithm and not the committee.  There is no effect in playing regionally ranked opponents other than possible SoS benefits

3.  Wins against lesser opponents will not hurt a strong team - the key difference, as I will illustrate in a subsequent post

Another benefit of the database is that teams are not penalized for winning a contest that it
should win. For example, in the current system if a strong team plays a weaker team its
strength-of-schedule will be negatively impacted even though it won the contest – as it should.
In the selection criteria database, the team that wins a contest against a lesser opponent will
not be statistically impacted by winning a contest it should win. This will greatly benefit
stronger teams in weaker conferences.

[Kuiper]

How does the new selection criteria impact soccer?

Denison Men's Soccer Head Coach Brandon Bianco, who has twice been chair of the NCAA Men's Soccer Committee and is considered a NCAA Soccer rules guru, has a twitter thread up on the impact.  I will reproduce a few key posts

1.  Bianco calls this "one of the most impactful/biggest changes" to DIII in a long time or maybe ever

At a high level, this is among one of the most impactful/biggest changes to
@NCAADIII we have seen in a long time (maybe ever). And unfortunately, despite the narrative out there, there has been nothing shared with broader membership, no opportunity for comment periods, etc.

2.  Bianco agrees that the biggest deal is the removal of the "bad" effect of beating weaker teams, through what he calls a process by which the algorithm "adjusts" the schedule to remove those bad games

High level, what does the Pairwise algorithm do?
1. It ranks every team in every sport #1 through whatever the sport membership is (like RPI)
2. It factors in WL%, SOS, Quality Wins
3. MOST IMPORTANT- it "adjusts" everyone's schedule and removes games against "bad" opponents.

3.  Here's the kicker:  According to Bianco, W&L, a 2023 Final Four team, would not have qualified for the NCAA Tournament under the new selection criteria

Had Pairwise been used last fall in soccer, W&L (Final Four team) who played the 10th-hardest schedule in the country (out of 415), won 70% of their games, and had 5 RvR would not have made the NCAA's. In fact, they would have needed to jump 5 teams to just be the last team in!!!

Who would have gotten in over them? A team who played 53% of their schedule against teams #217+ (5 against teams in the 300's!), with only 2 Top 100 wins (#45, 68). All other wins (13!) were #'s 179-365! This is just 1 example. There were more from last year just like this team.

4.  This means, according to Bianco, coaches will not have an incentive to construct a difficult schedule.  I'm not exactly sure why this has such a high magnitude effect, since committees can weight W-L and SoS differently, but presumably a higher W-L is more important and it is better to get a win against a weak opponent (which won't be counted against you for SoS purposes) than a loss against a strong opponent, which doesn't help you enough in SoS to offset the effect on W-L.

THERE IS LESS INCENTIVE TO PLAY A COMPETITIVE SCHEDULE- "bad" games get dropped. My team plays Newport and Stevens this year for example- moving forward, there's less incentive to play these.
Coaches are smart. Goalposts are moved. We'll schedule to manipulate the data!

If you play a great schedule and win a ton of games, you'll always be fine. It's that next "tier" that most likely will be affected most. The 12-3-3 high SOS team won't be as competitive as the higher WL lower SOS. Again, more incentive to schedule wins moving forward.

[Kuiper]

Just to provide a little balance to Bianco's thread, here are a few benefits I could see from this selection criteria change:

1.  More transparent, at least eventually, and less susceptible to bias and influence

The selection factors used before were public (albeit complicated), but the selections were always subject to debate and controversy because they were behind closed doors.  This should end that.  There won't even need to be selection shows anymore.  The rankings should tell you exactly where your team stands each week, just like a Massey rankings system.  You can still scoreboard watch to see if Western Connecticut wins or loses, but the debates on the board about whether they will get screwed by a pro-NESCAC committee in favor of Williams and its double digit ties won't happen anymore.  Bad for traffic on the Boards, but good for the student-athletes who can resolve matters on the field.

2.  Middling teams in top conferences will no longer get a huge bump

If it's better to win a lot than to play a really strong schedule, then there is an advantage for a dominant team in a weak conference over the 6th place NESCAC team.  Plus, UAA teams will no longer get a bump from being in different regions with different regional rankings.  That might give a bump to strong second place teams in weaker conferences over middling teams from stronger conferences.  More equitable distribution of tournament access seems like a DIII value.

3.  More geographical equity and less competitive pressure to spend a ton of money to travel for higher ranked competition

My perspective may be colored from being out in Region X, but there is a huge systemic bias on schools in DIII deserts because they need to travel to other areas of their region or to leave their region altogether in order to get higher SoS and to beat regionally ranked opponents from elsewhere.  Some poorer schools simply can't afford to do that and even wealthy schools don't really want to because it is inconsistent with a core value of DIII to prioritize travel and missing classes in the quest to make the NCAA tournament.  Strength of Schedule will still matter, but at least the regional ranking thing won't indirectly make it more costly to be in DIII for these schools and require more missed classes etc.

Moreover, the algorithm approach will affect hosting because the higher seed may no longer be concentrated in a few regions due to access to certain opponents.  That also can provide more geographical equity.

4.  Tournament success will still depend upon the same criteria as has been used in the past

Presumably, teams that make the tournament on the back of a weak schedule will still be at a disadvantage against teams who have been battle-hardened by strong competition in their regular season schedule.  So, in theory, coaches who over-correct their schedules to make them too weak will be punished and some kind of equilibrium or optimal balance will develop.

[Kuiper]

On the flip side, here's what Bianco either didn't say or undersold about what is bad about this change

1.  Not enough input from coaches

There is plenty about this in the twitter thread, so I won't add much.  I will push back slightly though - if some of the concerns are about travel, missed classes etc, then I could see how coach input would be somewhat less critical than AD/President input.  Either way, input is good and by making this an all team sports change, it definitely washes out the impacts on individual sports.  They do give the sport committees the ability to change the dials somewhat, but I'm not sure it does enough.

2.  No transition period

As Bianco mentioned, they are slating this to take effect this fall, but fall sports like soccer have already set their schedules for the most part.  I have a whole thread on this board full of schools that have published their schedules.  Conference schedules are often locked in for years in advance, but non-conference schedules is the variable coaches can control on SoS and this short effective date eliminates some of the benefits of the proposal for next year (e.g., reducing travel) for those schools with locked schedules and gives an unintentional advantage to others who aren't done yet.

There is a provision allowing sport committees to change the dials every few years if they want to emphasize SoS more etc, but they can't change it weekly if it's not working out this year.

This also impacts students who chose a school in their recruiting process because of the better chance of qualifying for the NCAA tournament and now may be worse off.  I'm not sure I'm that sympathetic (you should choose a school because of the fit of the school qua school under DIII principles), but it is a reality that some students will feel hard done by this change.

3.  Preferences non-competitive schedules

If it's true that W-L matters that much more, then we end up with a lot of non-competitive games.  I've seen tons of those in high school games.  It doesn't look like that much fun for either side and doesn't feel like what an athletic organization should aim to achieve, within limits.

[EnmoreCat]

Thanks Kuiper, whatever we miss on debating who makes it will be overwhelmed by this discussion for quite some time.  I don't suppose the NCAA indicated what the 2023 final 64 would have looked like for comparison purposes, other than what it meant for W&L?  Wondering how close Massey's algo will be to the new formula.

[stlawus]

Pairwise works in hockey because there aren't nearly as many teams.  There are a ton of common opponent results and just about every team will be no more than a single team/game removed from any opponent they haven't played. 

I do think there needs to be some balance struck when it comes to teams getting in solely due to SOS, this is mostly on the basketball sides of things but it does happen in soccer (my thoughts on teams like Rochester are well known in this regard).  But I don't see why there can't just be some tweaks made to the championship manual to give the committees more guidance.

I'm not going to immediately denounce this since I've literally only just learned about it.  It could be a good thing, could be bad.   I guess we'll have to wait and see. 

[Kuiper]

Thanks Kuiper, whatever we miss on debating who makes it will be overwhelmed by this discussion for quite some time.  I don't suppose the NCAA indicated what the 2024 final 64 would have looked like for comparison purposes, other than what it meant for W&L?  Wondering how close Massey's algo will be to the new formula.

According to the FAQ, they tested the algorithm in selected sports to compare the results against the selections made by the human committees and they didn't find all that many differences.  I don't know if men's soccer was one of the sports though, but I suspect Bianco got his information about W&L from that test. 

Q: How has this been tested?
A: The Division III Championships Committee has reviewed selected sport data from the 2018-19
season (a full pre-COVID season) and the 2021-22 season (a nearly full season post-COVID).
Staff compared the selected championship field against what would have been selected by the
database and saw only slight variances between the two fields, almost regardless of the dial
settings which have not yet been reviewed by sport committees. The Championships
Committee and staff are in the process of compiling and reviewing data for all team sports
from the 2022-23 season.

[Kuiper]

Pairwise works in hockey because there aren't nearly as many teams.  There are a ton of common opponent results and just about every team will be no more than a single team/game removed from any opponent they haven't played. 

I do think there needs to be some balance struck when it comes to teams getting in solely due to SOS, this is mostly on the basketball sides of things but it does happen in soccer (my thoughts on teams like Rochester are well known in this regard).  But I don't see why there can't just be some tweaks made to the championship manual to give the committees more guidance.

I'm not going to immediately denounce this since I've literally only just learned about it.  It could be a good thing, could be bad.   I guess we'll have to wait and see. 

Interestingly, Bianco later tweeted that the Pairwise algorithm they would use for all team sports is the not the same as the one they currently use for hockey.  I don't know if he means that individual sports can change the dials on the weight given to individual factors or something else.

Another big misunderstanding. The pairwise algorithm hockey has used will not be the same algorithm that will be used for all team sports.

[stlawus]

Reading Bianco's thread, I do wonder if this is partially if not predominantly financial based. Many teams (not all) that benefit from high SOS usually get that gaudy SOS number from a pretty eclectic mix of travel opportunities which in turn has resulted in the "have nots" of division III being disadvantaged for bids since they don't have the financial resources to craft hard schedules.  Now with the looming enrollment cliff along with schools already beginning to close and face dire financial situations, athletic department budgets are going to be squeezed even tighter. This will result in schools staying in region more and not traveling as much, which is likely by design as a way to save everyone money. 

[Kuiper]

On the flip side, here's what Bianco either didn't say or undersold about what is bad about this change

1.  Not enough input from coaches

There is plenty about this in the twitter thread, so I won't add much.  I will push back slightly though - if some of the concerns are about travel, missed classes etc, then I could see how coach input would be somewhat less critical than AD/President input.  Either way, input is good and by making this an all team sports change, it definitely washes out the impacts on individual sports.  They do give the sport committees the ability to change the dials somewhat, but I'm not sure it does enough.

2.  No transition period

As Bianco mentioned, they are slating this to take effect this fall, but fall sports like soccer have already set their schedules for the most part.  I have a whole thread on this board full of schools that have published their schedules.  Conference schedules are often locked in for years in advance, but non-conference schedules is the variable coaches can control on SoS and this short effective date eliminates some of the benefits of the proposal for next year (e.g., reducing travel) for those schools with locked schedules and gives an unintentional advantage to others who aren't done yet.

There is a provision allowing sport committees to change the dials every few years if they want to emphasize SoS more etc, but they can't change it weekly if it's not working out this year.

This also impacts students who chose a school in their recruiting process because of the better chance of qualifying for the NCAA tournament and now may be worse off.  I'm not sure I'm that sympathetic (you should choose a school because of the fit of the school qua school under DIII principles), but it is a reality that some students will feel hard done by this change.

3.  Preferences non-competitive schedules

If it's true that W-L matters that much more, then we end up with a lot of non-competitive games.  I've seen tons of those in high school games.  It doesn't look like that much fun for either side and doesn't feel like what an athletic organization should aim to achieve, within limits.

I should have added one more negative about this that Bianco alluded to, but only indirectly.  Maybe he's exaggerating, but he's suggesting that this change has the potential to get coaches fired, presumably because it may take a couple of years to sort itself out and in the meantime a coach with a couple years of not making the tournament might be let go.

This new selection criteria- something that will affect the livelihood of coaches- will go into effect this fall! And no one has seen it let alone know how the algorithm is calculated. What was intended to create more objectivity and transparency will do the opposite.

I hope that isn't the case and that schools are a bit more understanding than that, but it's noteworthy that the following D3 soccer coaches have all re-tweeted or commented about Bianco's thread and tons more have liked the post.

Travis Wall - Kenyon
Ryan Souders - Calvin
Greg Holker - Augsburg
Dejan Mladenovic - John Carroll
Justin Serpone - Amherst
Ben Federici - Transylvania
Steve McCarthy - Depauw
Justin Oliver - St. Olaf

Whether it affects their jobs or the jobs of their friends, it's pretty clear that the coaches are hot about the recommended change. 

I'm not sure of the next steps.  One D3 football guy tweeted the following:

I believe next steps would be a vote in the management council to adopt. May need to go to President's Council since it's a bylaw change? Not sure.

 

[SierraFD3soccer]

Hmmm, so many thoughts. All probably not answerable at least by me.

1. Will the higher ups rubberstamp this? I'm guessing so.

2. Talks about travel and trying to reduce costs. Does this favor teams that have many D3 soccer teams around them like Pennsylvania/Maryland?
For example almost all Cent. Conf. teams are in PA (7 in PA and 3 in MD - 2 are not that remote) and there are lots of D3 soccer teams they can play with very short drives and maybe not very good or marginal competition.

3. Will this reduce the quality of D3 soccer? Many more 5-0 to 10-1 games or more? Not sure, but it may reduce interest as there will be few not very interesting non-conf. games (no Messiah v. UMW etc). Guess most of us will be only looking at NESCAC regularly. Also will this mean that underdog teams will be more incentivized to park the bus and work for the tie especially with no OT?

4. How will this affect NESCAC? Fewer teams making the NCAAs or more?

5. If W&L would not make it in under this system, will teams like W&L will have to win their tournament to get in which would put many conferences like ODAC back 10 years (only one qualifier) for 2024. However, in 2025, W&L, Lynchburg, VA Wes and Roanoke will just play all, or many more,  ODAC teams as there are plenty of teams that are not very good thus increasing their W-L.

[Ejay]

Maybe we should go back the good 'ol days of 32 team brackets so we don't need to worry about which 2nd place conference team is in/out. 

[Hopkins92]

I'm a big college hockey fan and I will say the transparency of that process is really refreshing. (With that said, there are tons of issues with the actual tournament, but that's so off topic I wont bore everyone with it.)

I know that it's being stated that soccer would use a different Pairwise formula, but I will say, in college hockey, the better/bigger conferences gobble up more spots and the lesser conferences almost always just get their AQ in. So, Hockey East and the Big Ten get 3 and 4 teams in, and Denver's (NCHC) conference usually gets 2 or 3... In a 16 team field, you're talking about a lot of non-AQs going fast.)

[Newenglander]

I've heard this isn't just a recommendation - its already been approved.

[Pat Coleman]

I've heard this isn't just a recommendation - its already been approved.

The management council has to sign off on it but the championships committee is in favor and that carries HUGE weight. Then the presidents council sees it as well but they basically rubber stamp the management council.