Ratings Replacement

From US Go Wiki
Revision as of 21:55, 30 November 2010 by Vash3g (Talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Current status of replacement ratings system

Recruitment Needs

  • AGAGD Integration (possible? desirable?) -- Jonathan Bresler?
  • Database programming (SQL queries necessary to perform ratings updates)
  • Statistics expert--Brad Jones
    • Correct calculation of player sigma?
    • Statistical test of equivalence of online/face-to-face games (Kolmogorov-Smirnov?)
    • Rank certificates--Monte Carlo technique adequate or is there a better way?

Need-to-know

  • RDE and sigma_px parameters for various game settings--no difference between Ing and AGA rules, depends on komi and handicap. Extracted
  • Sigma assignment for new players entering system
    • Parameters extracted for ratings corresponding to rank midpoints
    • Spline fit between 9d and 50k, fixed values outside that range
  • Sigma assignment for out-of-date ratings
    • Extraction possible, test calculation performed
    • Looks to be linear increment of variance with time.
    • 2010-01-28: confirmed. sigma(t) = sqrt{ sigma(0)^2 + alpha^2 * t^2 }, where alpha = 0.0005 day^-1.
  • How to handle self promotions
    • Self promotion claims require at least one win to trigger promotion code
    • Rating promotion greater than 3.0: reseed
    • Rating promotion less than 3.0: seed_new = seed_old + 0.024746 + 0.32127 * deltaR; sigma_new = sqrt(sigma_old^2 + 0.256 * deltaR^1.9475)

General ratings update procedure

    • Find oldest game that needs to be (re)rated. Date of game is TournamentCascadeDate
    • Pull last valid TDList prior to TournamentCascadeDate
    • Move forward in time, rating games group by tournaments
    • Push updated ratings into database.


Important Considerations from Sam Zimmerman

  1. Who is going to generate the TDLISTS that Paul (Matthews) has been putting on the ratings page of the web site?
    • Jonathan Bresler: The primary information in the TDListA, appears to be, the player's ID, name and ratings information. The Membership database (http://usgo.org/mdb/) would be authoratative source for the name and AGA ID. The go database http://agagd.usgo.org/) would be the source for the ratings. The AGA ID links the two together...and indispensible element.

      The Go Database can(?) generate the TDListA....please see http://agagd.usgo.org/GetTDListA.php. Please bring forward any needed corrections. We can add features to this very plain version as needed. <p> Strangely, the TDListA from the http://usgo.org/ratings/RatingsQuery.php?Unrated=1 seems to round all ratings and sigma so that the last three decimals are zeros. I have mimiced that behavior.

  2. How am I going to get the ratings to put into the membership database? At present. I download the TDLISTA from the web site.
    • Phil Waldron: We will still generate the TDLists at the end of each ratings update.  With regards to getting new ratings into the membership database, I'd say keep going with what works unless it poses too great an inconvenience for you.  When the new membership database goes online I would expect that we would just inject the new ratings into the membership database directly as part of the update process.
    • Jonathan Bresler: Once the membership database is online, would suggest that the ratings be computed using the new ratings program and loaded directly into the go database. The membership database would pick us the ratings from the go database each day...automated database to database replication. Till then, download the TDListA from the go datbase ?
  3. Paul insisted that I send him the latest list of players each time that I sent him a .fix file. I sent it in the form of a TDLISTA. Should I send such a list to you or Jonathan or both or neither?
    • Phil Waldron: If you could send Jonathan a new TDList with the .fix files, I think that would be best until the membership system goes online.  At that point we could directly query the membership database for names.
  4. I now receive all tournament reports that are sent to ratings@usgo.org. Should I forward them to Jonathan or will he get them directly when they come to ratings@usgo.org?
    • Phil Waldron: Jonathan has a subscription to ratings@usgo.org, so he receives all the tournament reports that come in.
    • Jonathan Bresler: yes, please send the fix files to agagd@usgo.org so that its an organizational address, rather than me personally.
  5. Should I send the .fix file to jmb@bresler.org or will he have a special email address for this operation?
    • Phil Waldron: I'd suggest sending the .fix files to agagd@usgo.org rather than clogging up his personal email address.  It will make it easier for him to flag incoming report.
  6. Has anyone given any thought as to how we are going to handle all of this when the new web-hosted membership file becomes operationsl?
    • Phil Waldron: We haven't given a lot of thought yet to what happens when the membership stuff goes online, mostly because we don't have a good sense yet of what the final database will look like.  We do know it's coming, though, and we chatted in fairly broad strokes about how things would likely go. <p> My feeling is that once the new membership system goes online, the only thing that will be necessary is providing the .fix files with the new player ID numbers.  Everything else should be directly available from either the ratings or membership databases.
  7. When I first started working with Paul, I corrected the reports to insert new AGA numbers and correct erroneous numbers directly into the report. I would also check to be sure that all of the AGA numbers in the game report matched the player numbers. If these numbers did not match, I would coontact the tournametn director and get things corrected. I would also delete any names whose numbers did not appear the game records. I would then forward the correted reports on to Paul. Paul later asked me to not maodify the report but generate and send only the .fix file. Is there any part of this that you would like me to pck up again?
    • Phil Waldron: I'll defer Jonathan on the question of how much data correction to do to the tournament files.  The real crux of the matter is getting the AGA numbers for new players; only you can do that.
    • Jonathan Bresler: We have script that does a fair amount of validation of tournament results files. would like to create a webpage where TDs can submit their results. Should the file fail to pass muster they have to fix it in order to submit. Should the file include new players...well, looking for suggestions as to how that should be handled.

Ratings update cycle

It looks like we need three new flag fields:

  • TABLE games: Flag_Quarantined--indicates that the tournament is not suitable for ratings. Could be because of suspect data in the tournament report or some other reason. Regardless, data is presented on the AGAGD, but is not included in any ratings sweeps.
  • TABLE games: Flag_Online--indicates that the game was played online.
    • Jonathan: perhaps online is a characteristic of a game rather than a tournament. We rate games. Those games are usually the result of a tournament, but could be rated club games (for instance, the Baltimore Go Club every February). A tournament may consist, in the future, of a combination of facet-to-face and online games. Imagine High School championships...some players can make it to the venue, some can not...all play...some games are online, some are not. All would counted toward determining the High School champion. Only the face-to-face would be rated.
    • Phil: A convincing argument. Altered the flag to associate the online flag with the games table. Intuitively it makes more sense, although that means that the quarantined flag would also have to be associated with individual games rather than tournaments.
  • TABLE tournaments: Flag_Rerate--indicates that the ratings for this tournament (and consequently all subsequent tournaments) should be recalculated on the next ratings sweep.
    • Jonathan: Once we move the online flag to the games table, do we need a re-rate flag or can we clear the rated flag instead? The alternate ratings process will pick up all unrated games. Perhaps there is not a difference between unrated games and games that need to be re-rated.
    • Phil: I didn't realize that we had a rated flag already. I don't think there is a difference between 'need to rate' and 'needs to be re-rated'. What matters is being able to identify the oldest game that needs to be run through the rating system. Everything after that becomes invalid and needs to be (re)rated.

Rating Process

  1. Find the earliest tournament date that has a tournament that needs to be (re)rated and whose quarantined and online flags are not set.
    • SELECT Tournament_Code, Tournament_Date FROM tournaments WHERE flag_rerate=TRUE and flag_quarantined=FALSE ORDER BY Tournament_Date LIMIT 1;
  2. Get the historical TDList for the day before the date in question
    • TODO: need SQL here
  3. Get all tournaments that occured on or after the date in step 1 that have valid quarantined and online flags.
    • SELECT Tournament_Code from tournaments WHERE Tournament_Date > '$date' and flag_quarantined=false ORDER BY Tournament_Date
  4. For each tournament code in step 3.
    • Get games for the curent tournament
      • SELECT Pin_Player_1, Rank_1, Color_1, Pin_Player_2, Rank_2, Color_2, Handicap, Komi, Result FROM games WHERE Tournament_Code='$tournament_code' AND flag_online=FALSE
    • Rate the tournament
    • Push calculated ratings back into database.
      • TODO: Need SQL here
    • Set Flag_Rerate to false for this tournament
      • UPDATE tournaments SET Flag_Rerate=FALSE WHERE Tournament_Code='$tournament_code'

Alternate Rating Process

  1. Find the earliest eligible non-rated game date. SELECT Game_Date FROM games WHERE rated = 0 ORDER BY Game_Date ASC LIMIT 1;
  2. Get the TDListA applicable for that date. Currently using a mix of SQL and Perl. The SQL gets all the ratings prior to that date. The Perl selects the chronologically first for each player. SELECT p.Last_Name, p.Name, p.Pin_Player, p.MType, r.Rating, p.MExp, p.Club, p.State_Code, r.Sigma, r.Elab_Date FROM players p, ratings r WHERE p.Pin_Player != 0 AND p.Pin_Player = r.Pin_Player AND r.Elab_Date < 'last ratings run' ORDER BY Last_Name, Name, Elab_Date DESC;. Use Perl to thin the list down to the most recent rating for each player.
  3. Process games in batches....these batchees are called "tournaments"
# Get the list of batches in chronological order
foreach $tournament in `SELECT Tournament_Code from tournaments WHERE Tournament_Date > '$first_unrated_game_date';`{
    # Get the eligible games associated with that tournament
    SELECT Pin_Player_1, Rank_1, Color_1, Pin_Player_2, Rank_2, Color_2, Handicap, Komi, Result FROM games WHERE Tournament_Code='$tournament_code';
    Rate these games
    Update the <tt>ratings</tt> table and the <tt>players</tt> table with the new ratings
    Mark these games as rated in the database    
}

Personal tools