Usgo migration

From US Go Wiki
Revision as of 16:32, 25 February 2012 by Vash3g (Talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

American Go Association Go Database | Congress | Membership | Membership DB | Ratings | Software And IT Operations | Staff and Volunteers | Tournaments And Rules | Usgo_migration


Outstanding Tasks and Issues


  • Assigned: Jonathan Bresler (Josh Simmons will install prereqs)
  • Timeline: week of october 16th
  • Progress:
    • Copied over database

Collect content authors

  • Progress:
    • Volunteers that have agreed to help with this: (don't put emails here please, unless they've consented)
      • Mike Malveaux
      • Justin Hall
      • Roger Schrag
      • Roy Laird
      • Clark Wierda
  • email Josh if you know who is interested in this, especially if they have experience

Content copy over/creation

  • Progress: sent out first test assignments, let's see how it goes
  • Timeline: ?

Links and books system/content type

  • Assigned: Joshua Simmons
  • Progress: created taxonomies, mostly. content type is started, needs work.
  • Timeline: unknown, maybe a week or so

  • Assigned: Steve
  • Progress: contacted principal users
  • Timeline: soonish
    • Contacted owners of current hosted web sites. Explained situation to them and what steps they will need to take next. Will give everyone access to google apps with their own email address and site hosting.
    • Testing google sites. made login for UMD go club and a basic site. Will not display till we migrate the server over. Will have to find someone to make a proper site and test the sites functionality proper like.

  • Assigned: Steve
  • Progress: none
  • Timeline: done
    • Will do this almost last since we're heavily using this at the current point.
    • Moved over the database and updated from 1.12.0 to 1.18.1
    • This was a lame for a while but i made it my biotch - Steve
    • Changed permissions on the wiki so only trusted users could edit/create pages.


  • Assigned: Steve
  • Progress: uploaded site as of 10/24/2011. DNS moved to Linode 11/17/2011.
  • Timeline:
    • Should be a copy over, similar to the chapter sites.
    • DNS should be included in the rest of the settings from Jon Hilt
    • Contact for AGHS control will be given to KGS:odnihs when we get close to the final switch over
    • Content copied over as of 10/24/2011. They will add any changes when we move servers. DNS

  • Assigned: Roger did this
  • Progress:
    • DNS for moved to Linode 11/26/2011.
    • As of 11/26/2011 some small DNS cleanup remains - removing obsolete entries. Josh to look at this eventually.


  • I am increasingly worried that SSMTP which I installed won't be enough. I don't think it does any kind of queuing if it can't send the emails, which is bad.
  • Assigned: Steve
  • Might be able to wait until after migration, not sure
    • Did, undid, will do again

Initial Server Configuration

Please add full paths to the various pieces of software installed so that others can invoke the software on the command line.

That shouldn't be necessary in order to run them, and can be determined using the "which" command. Listing them all here would seem rather superfluous. Anything in particular that you couldn't find?

Please add the full pathname to the configuration files for the installed packages

Added the apache config files, vhosts, and php. Will add others as I think of them.

  1. setup linode in atlanta data center
  2. installed ubuntu 10.04 LTS x86
  3. renamed to usgo
  4. installed basic LAMP stack
  5. setup first user
  6. check php setup
  7. setup mysql/phpmyadmin
  8. restarted apache a bunch of times
  9. creating ssh and mysql user accounts - emailing users ident's, setting permissions
  10. added users to sudoers file.
  11. installed (any and all dependencies needed/suggested by ubuntu also installed, named here are those when going to aptitude)
    • php5,
    • libphp-jpgraph-examples
    • libphp-jpgraph, smarty,
    • libxml-dumper-perl,
    • libaprutil1-dbd-mysql,
    • libclass-dbi-mysql-perl,
    • libgetopt-long-descriptive-perl
  12. need to do:
    • install Perl packages: File::Basename,
    • Phil Waldron’s C++ rating software, (Jonathan Bresler to perform this task)
  13. hardened following [instructions] from step #3 on, will do first two when setting up vhosts
  14. installed git
  15. installed bash-completion package
  16. setup vhost for and tested.
  17. setup other vhosts for
    •, and
  18. edited conf files to make one file for each subdomains.
  19. enabled apache mods:
    • rewrite,
    • proxy_ajp,
    • proxy_balancer,
    • proxy_connect,
    • proxy_ftp,
    • proxy_http,
    • proxy,
    • proxy_scgi,
    • speling,
    • vhost_alias

cleaned up the apache config files a bit, got rid of or commented out old crud, moved the config files to sites-available and linked them in sites-enabled, moved out of sites-enabled to the parent directory (it shouldn't be in sites-enabled, that meant it was getting included on its own, and then explicitly in - josh

i also added index.html files into all of the vhost root directories that were empty, so they announce what vhost you're connecting to for testing. get rid of those if you set up the real files in a vhost. - josh

oh, i also made the apache config directory a git repo, and commited everything as an initial commit. seemed like a good idea. - josh

installed ghc (haskell compiler), will be using for a script and possibly other stuff later. also installed cabal, which is like a package manager type thing for haskell libraries.

Permissions related changes by Josh

Changed global umask to 002 (was 022) in /etc/profile (default group writable, easier sharing of www data)

Changed ownership of /var/www/ tree to www-data:www-data

Set group as sticky for the same, this will make everything you create in those directories owned by group www-data by default (so we can all edit it collectively)

Added everyone who currently has accounts to the www-data group

This all sound fine to everyone? Seemed logical enough.

Ideally almost nobody should need sudoers permissions, at least none of the people who just edit web pages. I can set up separate groups too, like if agfgo people should be able to edit their stuff but no other. I'd need more organizational info though, like who should be allowed to do what. Talk to josh if anyone cares to get this set up like that.

Start of copying over stuff (Josh)

Created the theme directory as a shared git repository (it's not a bare repository, it's a working tree)

Created the drupal and wordpress directories as the same

The command to do that, if anyone wants to make similar directories in there is "git init --shared=group <somedir>". Together with the umask I talked about before, means we'll all be able to muck about (with git) in that directory freely.

Pushed into the theme directory, so the theme crap is copied over for the moment.

Created mysql accounts for wordpress and drupal (separate), and a database for each of them to play in.

Installed base versions of drupal and wordpress.

Copied over and installed almost all of the old stuff. The theme and such are 90% working.

Worked on the /ratings code with Josh and made the search and basic info pages work with the new theme. (Steve)

Temporary Domain Names and Virtual Hosts

Until we get around to changing the DNS records to point to the new server(s), I've created some subdomains on my vanity domain ( for our use during the migration. They should now be set up as vhosts, thanks steve.

Let me know if we need others.


Apache Configuration

  • The base Apache config is in /etc/apache2/apache2.conf Usually you should edit one of the vhost configuration files listed above though.
  • PHP configs are:
    • apache mod_php: /etc/php5/apache2/php.ini
    • command line: /etc/php5/cli/php.ini

Apache Rewrite Rules

Set up the rewrite rules (only in the vhost) as follows: is the canonical domain, connecting to any of the aliases (or the ip address) will redirect you there. Obviously we'll change that to later. Works fine for https too.

Any requests that don't start with a valid top level directory, are redirected to drupal. So /theme/blah will be left alone, but /admin will be handled as it should by drupal. /news/ is the wordpress directory, so all of that works fine. Drupal and Wordpress each have their own internal, per-directory, rewrite rules that seem to be working. Thinking it might be more stable if I explicitly set all directory exceptions instead of allowing any directory that exists, I'll see. For now it's working enough to go forward.

There's a few exceptions to that, namely (so far) /phpmyadmin, /robots.txt /favicon.ico.

The only thing not really right so far is that you can manually request /drupal/ and it'll work fine, which isn't good. Need to find the right way to block that (the obvious way doesn't work, because it interferes with the internal redirects to drupal content that should go through). Will need to find the right way to block only external, explicit requests. Shouldn't be too hard, but not done yet.


I installed the backup-manager package, and set it up. It's set up to run nightly, at 4am and change. It will back up /etc, /home and /var/www directories and the drupal, wordpress, mysql, and phpmyadmin MySQL databases. Incremental backups were chosen, with weekly masters. Restoring needs to be tested. It backs up to /var/archives

It's set up in root's crontab, and the config is in /etc/backup-manager.conf (/etc/backup-manager.mysql.conf just sets it to do mysql, and keeps everything else the same).

We can also set it up to do uploading to another machine (via ssh), or even Amazon's S3 storage (we need to set it up to do encryption first, if we do either).


Drupal 7

Installed Drupal 7

And, turned off PHP safe_mode, it's too annoying and doesn't even seem useful. Look if there are any alternatives. suPHP?

Got clean URLs working again, had to edit drupal's .htaccess rewrite rules. Let josh know of any weird drupal behavior please, I think the change shouldn't be a problem though.

Drupal Theme

The drupal theme is now back up to par for where it was in drupal 6. I still need to do a little on the menu code, and also need to fix a CSS bug that drupal is causing in the flyout fallback.

The eidogo viewers in Wordprehss content are buggy, but I've verified that it is only because they are set to use They'll work fine when we switch over.

The theme is basically done now, at least enough to go forward. I'm working on input filters and UI things to get everything ready for people to start adding content.

I'm pretty sure that I'll be able to make Markdown a nice and usable input system for us, so I'm attempting that. It may take a little work to get perfect, but I think it'll be cool.


Made repositories on github for the drupal theme, drupal setup, wordpress setup, and the apache config files. They're all private. It's possible one or more of them could be public at some point, but don't change it without discussion please. Also don't create your own public forks; nobody has done an audit of any kind to figure out if anything private or security related is in any of those.

The settings files (which include database login info), and stuff like that are not included, but there could be many other issues.

If you guys edit any of that stuff, use git! Commit early and often. Real commit messages too please.

I also set up ssh-agent for us on the server (eval `ssh-agent -s` in /etc/profile), so ssh-add will work now (very useful with github, set up your ssh keypairs. ask josh for help, it's pretty easy).


MySQL Installation

MySQL Configuration

PHP mail setup

I set up php to be able to send email through a special address, It's working, probably needs more testing though. It just sends using gmail's SMTP servers, so no need to set our server box as an email sender in SPF, when we set that up.

We will need to see if that satisfies all scripts and users later though.

I used ssmtp, which is a really simplistic program that emulates (some of) the command line functionality of sendmail.

Go Problem of the Week

The problem of the week stuff is looking pretty good.

It automatically creates preview images, uses eidogo to show the full view, and you can queue up a crapload of problems and they'll automatically get published every week.

All done now. Needs some testing, but it seem basically solid.

All of the old problems are transferred over as well, and an archive view turned out to be pretty easy so I threw one together.

American Go Association Go Database | Congress | Membership | Membership DB | Ratings | Software And IT Operations | Staff and Volunteers | Tournaments And Rules | Usgo_migration

Personal tools