Jason.Bryer.org Blog

Version 0.9 of timeline on CRAN

2013-05-09T00:00:00-07:00

The initial version of the timeline package has been released to CRAN. This package provides creates timeline plots using ggplot2 in a style similar to Preceden. I would considered this beta quality as there are more features I would like to add but has enough functionality to possibly be useful to others.

install.packages('timeline',repos='http://cran.r-project.org')
require(timeline)
data(ww2)
timeline(ww2, ww2.events, event.spots=2, event.label='', event.above=FALSE)

The ww2 demo (type demo(ww2) at the R console to start) provides many variations of the timeline figure. There is also a Shiny app to explore some of the parameters to the timeline function.

timelineShinyDemo()

Or try the Shiny App from the RStudio Server at http://spark.rstudio.com/jbryer/timeline/.

You can always download the latest development version using devtools.

require(devtools)
install_github('timeline','jbryer')

Read full post...

Gambler's Run With Shiny

2013-05-08T00:00:00-07:00

I finally had an opportunity to play with Shiny, and I am very impressed. I have created a Github Project so head over there for the source code. There are a number of ways to distribute Shiny apps. If you are running R (and mostly likely you are if you are reading this), you can download and run Shiny apps using the runApp (if already downloaded), runGitHub, runGist, or runUrl functions. RStudio also make the Shiny Server available and you can also request an account on their servers. Also be sure to check out the excellent tutorial on Shiny.

First, install shiny and shinyIncubator (for the ActionButton) packages, preferably the development versions.

require(devtools)
install_github('shiny', 'rstudio')
install_github('shiny-incubator', 'rstudio')
require(shiny)

Gambler’s Run

This simple app that lets you simulate a sequence of random events, for example coin flips, and plot the cummulative sum. This app allows you choose the odds of winning, the number of games to simulate, and the number of simulations to display simultaneously.

To run the app locally:

shiny::runGitHub('ShinyApps', 'jbryer', subdir='gambler')

Or from the RStudio server (note that RStudio does not guarantee the server will always be up so this link may or may not work).

Lottery Tickets

Similar to the gambler app, this simulates buying a series of lottery tickets with varying odds of winning different amounts. Each previous run is saved and plotted in light grey to show how the current run compares to past runs.

To run the app locally:

shiny::runGitHub('ShinyApps', 'jbryer', subdir='lottery')

Or from the RStudio server (note that RStudio does not guarantee the server will always be up so this link may or may not work).

Just to try out all the ways to distribute Shiny apps, I also created a Gist for this app.

shiny::runGist("5525690")

Read full post...

Cut Dates Into Quarters

2013-04-18T00:00:00-07:00

Frequently I need to recode a date column to quarters. For example, at Excelsior College we have continuous enrollment so we report new enrollments per quarter. To complicate things a bit, our fiscal year starts in July so that July, August, and September represent the first quarter, January, February, and March are actually the third quarter. But sometimes we do need need to report out based upon calendar years (i.e. where January is in the first quarter). I am sure this is pretty common practice in many disciplines. There are probably other ways to do this in R (please comment below about other methods), but could not find one that satisfies my needs.

We can begin by sourceing the function from Gist using the devtools package.

require(devtools)
source_gist(5412193)

Create a vector of Dates.

> dates <- as.Date(c('2013-04-03','2012-03-30','2011-10-31',
                   '2011-04-14','2010-04-22','2004-10-04',
                   '2000-02-29','1997-12-05','1997-04-23',
                   '1997-04-01'))

The default is to use the typical academic fiscal year with the year staring July 1.

> getYearQuarter(dates)
 [1] FY2013-Q4 FY2012-Q3 FY2012-Q2 FY2011-Q4 FY2010-Q4 FY2005-Q2 FY2000-Q3 FY1998-Q2 FY1997-Q4
[10] FY1997-Q4
65 Levels: FY1997-Q4 < FY1998-Q1 < FY1998-Q2 < FY1998-Q3 < FY1998-Q4 < FY1999-Q1 < ... < FY2013-Q4

However, it easy to use get a quarters within a calendar year.

> getYearQuarter(dates, firstMonth=1)
 [1] FY2013-Q2 FY2012-Q1 FY2011-Q4 FY2011-Q2 FY2010-Q2 FY2004-Q4 FY2000-Q1 FY1997-Q4 FY1997-Q2
[10] FY1997-Q2
65 Levels: FY1997-Q2 < FY1997-Q3 < FY1997-Q4 < FY1998-Q1 < FY1998-Q2 < FY1998-Q3 < ... < FY2013-Q2

You can also alter the format of the levels using the fy.prefix, quarter.prefix, and sep parameters.

> getYearQuarter(dates, 1, '', '', '')
 [1] 20132 20121 20114 20112 20102 20044 20001 19974 19972 19972
65 Levels: 19972 < 19973 < 19974 < 19981 < 19982 < 19983 < 19984 < 19991 < 19992 < ... < 20132

Lastly, the function by default will create a level for each quarter between the minimum and maximum dates in the date vector passed in. You can override the range for defining the levels with the level.range parameter. If the specified range is smaller than the range of the passed in vector, the function will print a warning because values outside that range will be returned as NA.

> getYearQuarter(dates, level.range=as.Date(c('2010-01-01','2013-01-01')))
 [1] <NA>      FY2012-Q3 FY2012-Q2 FY2011-Q4 FY2010-Q4 <NA>      <NA>      <NA>      <NA>     
[10] <NA>     
13 Levels: FY2010-Q3 < FY2010-Q4 < FY2011-Q1 < FY2011-Q2 < FY2011-Q3 < FY2011-Q4 < ... < FY2013-Q3
Warning message:
In getYearQuarter(dates, level.range = as.Date(c("2010-01-01", "2013-01-01"))) :
  The range of x is greater than level.range. Values outside level.range will be returned as NA.

Here is a link to the Gist or copy-and-paste from below.

#' Returns the year (fiscal or calendar) and quarter in which the date appears.
#' 
#' This function will cut the given date vector into quarters (i.e. three month
#' increments) and return an ordered factor with levels defined to be the quarters
#' between the minimum and maximum dates in the given vector. The levels, by
#' default, will be formated as \code{FY2013-Q1}, however the \code{FY} and \code{Q}
#' can be changed using the \code{fy.prefix} and \code{quarter.prefix} parameters,
#' respectively.
#' 
#' @param x vector of type \code{\link{Date}}.
#' @param firstMonth the month corresponding to the first month of the fiscal year.
#'        Setting \code{firstMonth=1} is equivalent calenadar years.
#' @param fy.prefix the character string to paste before the year.
#' @param quarter.prefix the character string to paste before the quarter.
#' @param sep the separater between the year and quarter.
#' @param level.range the range to use for defining the levels in the returned
#'        factor.
#' @export
#' @examples
#' 	dates <- as.Date(c('2013-04-03','2012-03-30','2011-10-31',
#' 	                   '2011-04-14','2010-04-22','2004-10-04',
#' 	                   '2000-02-29','1997-12-05','1997-04-23',
#' 	                   '1997-04-01'))
#' 	getYearQuarter(dates)
#' 	getYearQuarter(dates, firstMonth=1)
#' 	getYearQuarter(dates, 1, '', '', '')
#' 	\dontrun{
#' 	getYearQuarter(dates, level.range=as.Date(c('2010-01-01','2013-01-01')))
#' 	}
getYearQuarter <- function(x, 
					   firstMonth=7, 
					   fy.prefix='FY', 
					   quarter.prefix='Q',
					   sep='-',
					   level.range=c(min(x), max(x)) ) {
	if(level.range[1] > min(x) | level.range[2] < max(x)) {
		warning(paste0('The range of x is greater than level.range. Values ',
					   'outside level.range will be returned as NA.'))
	}
	quarterString <- function(d) {
		year <- as.integer(format(d, format='%Y'))
		month <- as.integer(format(d, format='%m'))
		y <- ifelse(firstMonth > 1 & month >= firstMonth, year+1, year)  
		q <- cut( (month - firstMonth) %% 12, breaks=c(-Inf,2,5,8,Inf), 
		      labels=paste0(quarter.prefix, 1:4))
		return(paste0(fy.prefix, y, sep, q))
	}
	vals <- quarterString(x)
	levels <- unique(quarterString(seq(
		as.Date(format(level.range[1], '%Y-%m-01')), 
		as.Date(format(level.range[2], '%Y-%m-28')), by='month')))
	return(factor(vals, levels=levels, ordered=TRUE))
}

Read full post...

i Before e Except After c

2013-03-26T00:00:00-07:00

When I went to school we were always taught the “i before e, except after c” rule for spelling. But how accurate is this rule? Kevin Marks tweeted today the following:

»@uberfacts: There are 923 words in the English language that break the “I before E” rule. Only 44 words actually follow that rule.« Science
— Kevin Marks (@kevinmarks) March 25, 2013

Not sure where he came up with that result, but seems simple enough to verify. First, download a English language word list compiled by Kevin Atkinson and available at SourceForge (I will use the Parts of Speech Database, or download my version from Github). I also create a data frame (from the README file) partsOfSpeech that maps the codes to descriptions that we will use later.

require(ggplot2)
require(reshape)

partsOfSpeech <- as.data.frame(matrix(c("N", "Noun", "P", "Plural", "h", "Noun Phrase", 
    "V", "Verb (usu participle)", "t", "Verb (transitive)", "i", "Verb (intransitive)", 
    "A", "Adjective", "v", "Adverb", "C", "Conjunction", "P", "Preposition", 
    "!", "Interjection", "r", "Pronoun", "D", "Definite Article", "I", "Indefinite Article", 
    "o", "Nominative"), ncol = 2, byrow = TRUE), stringsAsFactors = FALSE)
names(partsOfSpeech) <- c("Code", "Description")

words <- read.table("part-of-speech.txt", sep = "\t", header = FALSE, quote = "", 
    col.names = c("Word", "POS"), stringsAsFactors = FALSE)
nrow(words)

## [1] 295172

The parts-of-speech is coded such that the letters before | character come from the original Moby database and letters after the | character come from WordNet. The first character corresponds to the primary classification. The following R code will split this field into two new variables, Moby and WordNet, and then strip the first character from WordNet to create a WordNetPrimary variable. We will use this classification later for plotting purposes.

tmp <- lapply(words$POS, FUN = function(x) {
    x <- unlist(strsplit(x, "|", fixed = TRUE))
    if (length(x) == 1) 
        return(c(NA, x[[1]])) else if (x[[1]] == "") 
        return(c(NA, x[[2]])) else return(c(x[[1]], x[[2]]))
})
words$Moby <- sapply(tmp, function(x) x[1])
words$WordNet <- sapply(tmp, function(x) x[2])
words$WordNetPrimary <- substr(words$WordNet, 1, 1)
table(words$WordNetPrimary, useNA = "ifany")

## 
##      !      A      C      D      h      i      N      p      P      r 
##    260  51914     54     60  71566   2239 119441   8506     99     85 
##      t      v      V   <NA> 
##  12399  13730  12124   2695

We use the grep function to get three vectors representing all the “ie”, “ei”, and “cei” words. We also print the number of each type word and the percentage of all words this represents.

ie <- grep("ie", words$Word)
ei <- grep("ei", words$Word)
cei <- grep("cei", words$Word)
cie <- grep("cie", words$Word)

length(ie)

## [1] 10647

length(ie)/nrow(words) * 100

## [1] 3.607

length(ei)

## [1] 3542

length(ei)/nrow(words) * 100

## [1] 1.2

length(cei)

## [1] 202

length(cei)/nrow(words) * 100

## [1] 0.06843

length(cie)

## [1] 654

length(cie)/nrow(words) * 100

## [1] 0.2216

Number of words that follow the rule, “i before e except after c”

length(ie) + length(cei) - length(cie)

## [1] 10195

Number of i after e words that are not after c (first way to break the rule).

length(ei[!(ei %in% cei)])

## [1] 3340

Number of i before e words that are after c (the other way to break the rule).

length(cie)

## [1] 654

Percentage of words that break the rule.

(length(ei[!(ei %in% cei)]) + length(cie))/sum(length(ie), length(ei)) * 100

## [1] 28.15

So of the 14,189 “ie” and “ei” words, 3,994 break the “i before e, except after c” rule, or about 28.1%.

Let’s see how this breaks out by part-of-speech.

thewords <- words[c(ie, ei), ]
thewords$BreakRule <- TRUE
thewords[which(row.names(thewords) %in% c(cei, ie[!(ie %in% cie)])), ]$BreakRule <- FALSE

# Counts
tab <- as.data.frame(table(thewords$WordNetPrimary, thewords$BreakRule, useNA = "ifany"))
tab <- merge(tab, partsOfSpeech, by.x = "Var1", by.y = "Code", all.x = TRUE)

ggplot(tab, aes(x = Description, y = Freq, fill = Var2)) + geom_bar(stat = "identity", 
    position = "dodge") + ylab("Number of Words") + xlab("Part of Speech") + 
    scale_fill_hue("Break the Rule") + ggtitle("i Before e, Except After c") + 
    coord_flip()

# Percentages
tab2 <- as.data.frame(prop.table(table(thewords$WordNetPrimary, thewords$BreakRule, 
    useNA = "ifany"), 1) * 100)
tab2 <- merge(tab2, partsOfSpeech, by.x = "Var1", by.y = "Code", all.x = TRUE)
ggplot(tab2, aes(x = Description, y = Freq, fill = Var2)) + geom_bar(stat = "identity", 
    position = "dodge") + ylab("Percentage of Words by Part of Speech") + xlab("Part of Speech") + 
    scale_fill_hue("Break the Rule") + ggtitle("i Before e, Except After c") + 
    coord_flip()

A few last details. Here is the proportional table of words that break the rule by part-of-speech. Lastly, the definite article and pronoun words (three of each) that all break the rule.

cast(tab2, Description ~ Var2, mean, value = "Freq")

##              Description FALSE   TRUE
## 1              Adjective 83.17  16.83
## 2                 Adverb 66.56  33.44
## 3            Conjunction 25.00  75.00
## 4       Definite Article  0.00 100.00
## 5           Interjection 55.56  44.44
## 6                   Noun 65.84  34.16
## 7            Noun Phrase 63.70  36.30
## 8                Pronoun  0.00 100.00
## 9    Verb (intransitive) 54.55  45.45
## 10     Verb (transitive) 49.42  50.58
## 11 Verb (usu participle) 65.45  34.55
## 12                  <NA> 67.26  32.74

thewords[which(thewords$WordNetPrimary == "D"), ]

##           Word POS Moby WordNet WordNetPrimary BreakRule
## 113927  either DCv <NA>     DCv              D      TRUE
## 182679 neither DCv <NA>     DCv              D      TRUE
## 262111   their   D <NA>       D              D      TRUE

thewords[which(thewords$WordNetPrimary == "r"), ]

##               Word POS Moby WordNet WordNetPrimary BreakRule
## 262112      theirs   r <NA>       r              r      TRUE
## 262113   theirself   r <NA>       r              r      TRUE
## 262114 theirselves  rp <NA>      rp              r      TRUE

Part II - Using only the 5,000 Most Frequently Used Words

Here is an update using the list of 5,000 most commonly used words from http://www.wordfrequency.info/top5000.asp (note there really are only 4,354 unique words since the same word can be used in different parts-of-speech). Of the 4,354 unique words, 96, or about 2.2%, have an “ie” or “ei” in the word. Of those 96 words, 31, or 32.3% break the “i before e except after c” rule.

words <- read.csv("MostUsedWords.csv")
dups <- words[words$Word %in% words[duplicated(words$Word), ]$Word, ]
head(dups[order(dups$Word), ])

##      Rank  Word Part.of.speech Frequency Dispersion
## 47     46 about              i    874406       0.96
## 180   179 about              r    208550       0.97
## 897   896 above              i     44130       0.95
## 1604 1599 above              r     23866       0.92
## 1553 1548 abuse              n     24534       0.93
## 3783 3778 abuse              v      7554       0.94

length(unique(words$Word))

## [1] 4354

words <- words[!duplicated(words$Word), ]

ie <- grep("ie", words$Word)
ei <- grep("ei", words$Word)
cei <- grep("cei", words$Word)
cie <- grep("cie", words$Word)

# Percentage of words that break the rule.
(length(ei[!(ei %in% cei)]) + length(cie))/sum(length(ie), length(ei)) * 100

## [1] 32.29

Part III - Weighted by Frequency of Words

Using the same list as part II above, let’s consider the word frequency. That is, we’ll weight each word by it’s frequency according to WordFrequency.info. Using this approach, 47% of “ie” words break the rule. Put another way, for each “ie” word you encounter reading, there is a 47% chance it does not follow the “i before e, except after c” rule.

words <- read.csv("MostUsedWords.csv")
ie <- grep("ie", words$Word)
ei <- grep("ei", words$Word)
cei <- grep("cei", words$Word)
cie <- grep("cie", words$Word)
(sum(words[ei[!(ei %in% cei)], "Frequency"]) + sum(words[cie, "Frequency"]))/sum(words[ie, 
    "Frequency"], words[ei, "Frequency"]) * 100

## [1] 46.81

Read full post...

Version 1.0 of multilevelPSA Available on CRAN

2013-02-14T00:00:00-08:00

Version 1.0 of multilevelPSA has been released to CRAN. The multilevelPSA package provides functions to estimate and visualize propensity score models with multilevel, or clustered, data. The graphics are an extension of PSAgraphics package by Helmreich and Pruzek. The example below will investigate the differences between private and public school internationally using the Programme of International Student Assessment (PISA). The multilevelPSA package includes a subset of the full 2009 PISA dataset including North American countries (i.e. Canada, Mexico, & United States). However, the full dataset is available in the pisa R package and can be downloaded using the install_github function in the devtools package (note that the package is approximately 80mb and as such, is not available on CRAN).

> install.packages(c('multilevelPSA','devtools'), repos='http://cran.r-project.org')
> require(devtools)
> install_github('pisa','jbryer')
> require(multilevelPSA)
> require(pisa)

Load and setup the data. If the pisa package is installed we will load and subset from the full PISA dataset, otherwise we will use only the North American countries.

> data(pisa.colnames) #Data catalog
> data(pisa.psa.cols) #Character vector listing the covariates we will use in phase I
> student <- NULL
if(require(pisa, quietly=TRUE)) {
	data(pisa.student)
	data(pisa.school)
	student = pisa.student[,c('CNT', 'SCHOOLID',
	                          paste0('PV', 1:5, 'MATH'),
	                          paste0('PV', 1:5, 'READ'),
	                          paste0('PV', 1:5, 'SCIE'),
	                          pisa.psa.cols)]
	school = pisa.school[,c('COUNTRY', "CNT", "SCHOOLID",
	                        'SC02Q01', #Public (1) or private (2)
	                        'STRATIO' #Student-teacher ratio 
	)]
	names(school) = c('COUNTRY', 'CNT', 'SCHOOLID', 'PUBPRIV', 'STRATIO')
	school$SCHOOLID = as.integer(school$SCHOOLID)
	school$CNT = as.character(school$CNT)
	student$SCHOOLID = as.integer(student$SCHOOLID)
	student$CNT = as.character(student$CNT)
	student = merge(student, school, by=c('CNT', 'SCHOOLID'), all.x=TRUE)
	student = student[!is.na(student$PUBPRIV),] #Remove rows with missing PUBPRRIV
	rm(pisa.student)
	rm(pisa.school)
} else {
	data(pisana)
	student = pisana
	rm(pisana)
}

Phase I

Propensity score analysis is generally conducted in two phases. In phase one, the dependent measure of interest is treatment placement. In this example, we will consider attending private school to be the treatment. There are a variety of methods we can use including logistic regression (see the mlpsa.logistic function) and classification trees (see the mlpsa.ctree function). In this example we will use the ctree function in the party package to model private school attendance using the conditional inference tree framework.

> mlctree = mlpsa.ctree(student[,c('CNT','PUBPRIV',pisa.psa.cols)], 
                        formula=PUBPRIV ~ ., level2='CNT')
> student.party = getStrata(mlctree, student, level2='CNT')

The mlpsa.ctree estimates separate models for each level. As a result, different sets of covariates are likely to be used within each level. The tree.plot function creates a heat map of covariate use by level. The shading indicates the shallowest depth each covariate appears. That is, if a covariate is utilized more than once within a tree, then the smallest depth will be used to determine the shading.

> tree.plot(mlctree, level2Col=student$CNT)

Phase II

Phase two involves comparing students between the two groups with similar covariate profiles. In the case of classification trees, we consider students in the same leaf node to have sufficient similar covariate balance. However, it is important to verify that sufficient covariate balance has been achieved and the functions in the PSAgraphics can assist with that.

First, we will calculate a mean math score of the five plausible values provided. This is not entirely correct and final tabular results should utilize all the plausible values separately and combined to provide a pooled estimate (see the survey package for example). However, for visualization purposes the mean value will provide a sufficient estimate of the final results.

> student.party$mathscore = apply(student.party[,paste0('PV', 1:5, 'MATH')], 1, sum) / 5

> results.psa.math = mlpsa(response=student.party$mathscore, 
                           treatment=student.party$PUBPRIV, 
                           strata=student.party$strata, 
                           level2=student.party$CNT, minN=5)

 > summary(results.psa.math)
 Multilevel PSA Model of 694 strata for 64 levels.
 Approx t: 3.17
 Confidence Interval: 15.71, 18.41

The mlpsa function returns an object of class type mlpsa. The S3 methods for summary, print, plot, and xtable are implemented. Additionally, the returned object contains the following elements:

approx.t The approximate overall t-value.
level1.summary A data frame containing the results of each individual strata.
level2.summary A data frame with the overall results for the clustering level two variable (country in this example).
overall.ci An integer vector with two values for the overall confidence interval.
overall.mnx The overall adjusted mean for the control group (i.e. public schools).
overall.mny The overall adjusted mean for the treatment group (i.e. private schools).
overall.n The overall n.
overall.nx The overall n for the control group (i.e. public schools).
overall.ny The overall n for the treatment group (i.e. private schools).
overall.se.wtd The overall weighted standard error of the mean difference.
overall.wtd The overall weighted difference.
removed An integer vector with the positions of rows from the original data frame that were removed do to insufficiently small strata (default minimum strata size is five, see minN parameter of mlpsa function).
unweighted.summary Data frame containing the overall unadjusted means for the treatment and control groups.

Other fields not listed above are used for plotting purposes.

 > results.psa.math$level2.summary[,c('level2','n','Private','Private.n','Public','Public.n',
                                      'diffwtd','ci.min','ci.max','df')]

    level2     n  Private Private.n   Public Public.n      diffwtd     ci.min     ci.max    df
 1     ALB  4596 420.8655       395 373.4865     4201  47.37901925  37.112435  57.645604  4578
 2     ARG  4707 409.9844      1535 381.7359     3172  28.24847947  19.860348  36.636611  4681
 3     AUS 14251 525.7220      5536 497.6153     8715  28.10669008  23.477031  32.736349 14193
 4     AUT  6405 494.3873       842 500.5522     5563  -6.16487922 -16.364671   4.034913  6377
 5     AZE  2520 489.5733        80 437.1517     2440  52.42154979  41.202358  63.640742  2514
 6     BEL  8488 530.4688      5830 490.3529     2658  40.11586405  32.093483  48.138245  8460
 7     BGR  4488 510.5927        59 426.6689     4429  83.92386223  64.984555 102.863169  4480
 8     BRA 19112 420.6211      2265 370.9140    16847  49.70712118  44.353349  55.060893 19032
 9     CAN 23035 574.6481      1609 512.7957    21426  61.85247013  54.140194  69.564746 22981
 10    CHE 11645 538.9332       383 529.5514    11262   9.38179653  -4.692227  23.455820 11631
 11    CHL  5161 432.2037      3022 414.2283     2139  17.97548415   8.280336  27.670632  5135
 12    COL  7695 410.2222      1448 381.9332     6247  28.28900773  22.479379  34.098637  7651
 13    CZE  5751 513.4700       270 510.6293     5481   2.84071242  -8.075300  13.756725  5745
 14    DEU  4555 529.8054       241 510.5953     4314  19.21013318   3.206619  35.213647  4545
 15    DNK  5839 500.7247      1041 486.4226     4798  14.30205768   5.809685  22.794430  5825
 16    ESP 25363 501.6931     10034 484.6639    15329  17.02913971  13.582441  20.475839 25295
 17    EST  4727 514.0800       127 513.8796     4600   0.20041004 -14.952982  15.353802  4723
 18    FIN  5755 533.7971       279 539.0404     5476  -5.24326787 -18.901244   8.414708  5743
 19    GBR  8202 538.5366       439 501.2581     7763  37.27846710  27.933956  46.622978  8176
 20    GRC  4665 489.9827       321 468.1922     4344  21.79050984  10.808237  32.772783  4643
 21    HKG  4804 552.7403      4486 591.3133      318 -38.57307922 -50.860842 -26.285316  4800
 22    HRV  3059 468.1499        66 468.1106     2993   0.03929578 -19.559319  19.637911  3051
 23    HUN  4583 501.5931       539 494.2722     4044   7.32087259  -4.284299  18.926044  4571
 24    IDN  5136 362.4735      2375 381.7147     2761 -19.24121312 -23.520458 -14.961968  5118
 25    IRL  3928 493.7999      2462 478.9441     1466  14.85584371   3.175379  26.536309  3916
 26    ISL  3207 538.7074        17 505.9170     3190  32.79040612  -7.925951  73.506763  3205
 27    ISR  5607 471.0903       977 445.0270     4630  26.06329628  15.921608  36.204985  5585
 28    ITA 30234 464.5795      1641 491.9211    28593 -27.34161124 -37.826591 -16.856631 30208
 29    JOR  6439 419.7438       890 389.4549     5549  30.28893975  20.712494  39.865385  6411
 30    JPN  6088 521.8295      1672 532.6987     4416 -10.86927112 -20.465999  -1.272544  6070
 31    KAZ  3688 439.8842       140 415.6892     3548  24.19497667   5.874004  42.515949  3670
 32    KGZ  4128 418.3577       111 334.0767     4017  84.28102627  70.077098  98.484955  4118
 33    KOR  4989 549.2523      1898 548.4127     3091   0.83959029  -7.121606   8.800787  4961
 34    LIE   329 523.4837        18 536.4461      311 -12.96248124 -58.831205  32.906242   327
 35    LTU  4500 471.7728        44 476.7377     4456  -4.96495842 -26.761449  16.831532  4498
 36    LUX  4613 498.1891       666 493.4815     3947   4.70766110  -4.990032  14.405354  4599
 37    LVA  4343 469.8542        30 488.0522     4313 -18.19794867 -49.563973  13.168076  4339
 38    MAC  5628 527.8113      5392 480.8967      236  46.91466591  34.677379  59.151953  5618
 39    MEX 38124 430.2105      4044 423.0054    34080   7.20509210   4.038334  10.371850 38038
 40    MNE    41 386.5525        13 369.8808       28  16.67167582 -19.164385  52.507737    39
 41    NLD  4667 531.9771      2872 535.8870     1795  -3.90982011 -18.107778  10.288137  4657
 42    NOR  4353 462.8531        49 499.1450     4304 -36.29191402 -59.376655 -13.207173  4345
 43    NZL  4643 573.8157       242 519.8993     4401  53.91640732  44.564926  63.267889  4631
 44    PAN  3608 409.4171       919 344.0597     2689  65.35745641  56.961259  73.753654  3568
 45    PER  5985 387.2319      1155 357.5425     4830  29.68934040  19.766241  39.612440  5941
 46    POL  4803 522.3807       328 496.4239     4475  25.95684445  14.719257  37.194432  4787
 47    PRT  6298 503.2090       682 483.6007     5616  19.60823945  11.927275  27.289204  6286
 48    QAR  4435 444.2281      3281 384.2296     1154  59.99852251  51.270428  68.726617  4371
 49    QAT  7856 410.2191      2244 344.6134     5612  65.60569937  58.660056  72.551343  7742
 50    QCN  4966 627.8917       512 597.5703     4454  30.32137741  15.221133  45.421622  4954
 51    ROU  1017 370.9295        15 387.7845     1002 -16.85494354 -46.227241  12.517354  1013
 52    RUS  5308 469.4605        12 469.5508     5296  -0.09026699 -42.681943  42.501409  5306
 53    SGP  4981 524.9804       126 557.3697     4855 -32.38926291 -53.436208 -11.342317  4973
 54    SRB  5353 410.7185        53 442.6754     5300 -31.95684080 -57.341095  -6.572587  5347
 55    SVK  4555 516.3498       343 494.3370     4212  22.01279817   6.597103  37.428493  4547
 56    SVN  6155 566.1863       129 477.3868     6026  88.79944245  73.648893 103.949992  6145
 57    SWE  4567 516.5581       543 492.3527     4024  24.20539406  14.103703  34.307085  4555
 58    TAP  5785 518.3554      2225 566.2069     3560 -47.85153187 -54.453618 -41.249446  5759
 59    THA  6209 406.9979       800 429.9237     5409 -22.92581091 -35.658631 -10.192991  6197
 60    TTO  4604 402.7190       815 420.4209     3789 -17.70190852 -25.938477  -9.465340  4594
 61    TUN  2414 301.4003        95 377.2423     2319 -75.84192626 -88.562235 -63.121618  2404
 62    TUR   125 578.1132        17 515.9970      108  62.11617396  32.725739  91.506609   121
 63    URY  5462 469.0636      1018 416.3180     4444  52.74565867  44.245235  61.246083  5414
 64    USA  5233 504.5025       345 484.7964     4888  19.70610175   7.267340  32.144863  5215

The Multilevel PSA Assessment Plot provides a detailed visualization of the results. The default plot method is comprised of three panels (each panel can be plotted separately, see the commands below). The panels to the left and bottom represent the means for private and public schools, respectively. Grey dots correspond to individual strata and the colored dots to the overall mean for that country. The main plot is a scatter plot with the overall of public school mean on the x-axis and the private school mean on the y-axis. The size of the dots are proportional to the number of students sampled within each country. Rug plots on the right and top represent the distribution of scores. For each point, a line is projected parallel to the unit line (i.e. y=x) to another line perpendicular to the unit line. The tick marks along that line correspond to the distribution of differences. That is, the distance of each tick mark to the unit line is equal to the difference between private and public school scores for that country (a separate plot for differences is provided below). Therefore, the unit line indicates no difference such that points that lie above the line indicate a difference favoring private schools and points that lie below the line indicate a difference favoring public schools. The dashed blue line parallel to the unit line represents the overall mean difference across all countries and the dashed green line represents the confidence interval of that difference. In this example, given the relatively large n, the confidence interval is so narrow that it is practically overlaps the mean.

> plot(results.psa.math)

Click here for a larger version of the PSA Assessment Plot.

From this figure, we can conclude that there is a small, statistically significant, effect in favor of private schools. However, for many countries, that difference is small as exemplified by the fact that many of the points cluster around the unit line. And there are some countries where public schools outperform private schools. The largest of these is Tunisia, although the overall performance of that country is also the lowest when adjusted for private or public school attendance.

The difference plot below provides more detail with regard to the distribution of differences. In this plot, the grey points correspond to the difference of each strata (i.e. the leaf nodes from phase I above in this example). The blue dots correspond to the overall difference for each country, and like above, the size is proportional to the number of students sampled within each country. However, this figure also included confidence intervals for each country as well as overall. Since the standard deviation was specified vis-à-vis the sd parameter, the scale of the x-axis is in standardized units.

> mlpsa.difference.plot(results.psa.math, sd=mean(student.party$mathscore, na.rm=TRUE))

You can plot the individual parts of the Multilevel PSA Assessment plot with the following functions.

> mlpsa.circ.plot(results.psa.math, legendlab=FALSE)
> mlpsa.distribution.plot(results.psa.math, 'Public')
> mlpsa.distribution.plot(results.psa.math, 'Private')

Read full post...

Converting a list to a data frame

2013-01-30T00:00:00-08:00

There are many situations in R where you have a list of vectors that you need to convert to a data.frame. This question has been addressed over at StackOverflow and it turns out there are many different approaches to completing this task. Since I encounter this situation relatively frequently, I wanted my own S3 method for as.data.frame that takes a list as its parameter. I should note that it only works with atomic vectors (i.e. logical, integer, numeric, complex, character and raw). If any one of the elements in the list are of some other class type, the function will call NextMethod. However, on my R instance at least, this will end up calling as.data.frame.default which will in turn throw an error.

To use the function you can source the function directly from Gist using the source_gist function in the devtools package.

require(devtools)
source_gist(4676064)

Or you can download the code at https://gist.github.com/4676064

Example One

In this first example we have a list with two vectors, each with the same length and the same names.

> test1 <- list( c(a='a',b='b',c='c'), c(a='d',b='e',c='f'))
> as.data.frame(test1)
  a b c
1 a b c
2 d e f

Example Two

In this example we have a list of two vectors, same length, but only one has names. The function in this case will use the names from the first vector with names for the column names of the data frame.

> test2 <- list( c('a','b','c'), c(a='d',b='e',c='f'))
> as.data.frame(test2)
  a b c
1 a b c
2 d e f

Example Three

This example has two named vectors, but only have one overlapping named element.

> test3 <- list('Row1'=c(a='a',b='b',c='c'), 'Row2'=c(a='d',var2='e',var3='f'))
> as.data.frame(test3)
     a    b    c var2 var3
Row1 a    b    c <NA> <NA>
Row2 d <NA> <NA>    e    f

Example Four

This is an example of what to avoid, three vectors of differing lengths and not named. The number of columns in the resulting data frame will be equal to the longest vector. For vectors less than that, NAs will be filled in on the right most columns. This method will also print a warning.

> test4 <- list('Row1'=letters[1:5], 'Row2'=letters[1:7], 'Row3'=letters[8:14])
> as.data.frame(test4)
     Col1 Col2 Col3 Col4 Col5 Col6 Col7
Row1    a    b    c    d    e <NA> <NA>
Row2    a    b    c    d    e    f    g
Row3    h    i    j    k    l    m    n
Warning message:
In as.data.frame.list(test4) :
  The length of vectors are not the same and do not are not named, the results may not be correct.

Example Five

Another example of equal length vectors.

> test5 <- list(letters[1:10], letters[11:20])
> as.data.frame(test5)
  X1 X2 X3 X4 X5 X6 X7 X8 X9 X10
1  a  b  c  d  e  f  g  h  i   j
2  k  l  m  n  o  p  q  r  s   t

Example Six

This example shows the warning (and likely error too) that occurs when all of the elements of the list are not atomic vectors.

> test6 <- list(list(letters), letters)
> as.data.frame(test6)
Error in as.data.frame.default(test6, row.names = NULL, optional = FALSE) : 
  cannot coerce class '"list"' into a data.frame
In addition: Warning message:
In as.data.frame.list(test6) : All elements of the list must be a vector.

Read full post...

Comparing two data frames with different number of rows

2013-01-24T00:00:00-08:00

I posted a question over on StackOverflow on an efficient way of comparing two data frames with the same column structure, but with different rows. What I would like to end up with is an n x m logical matrix where n and m are the number of rows in the first and second data frames, respectively; and the value at the ith row and jth column indicates whether all the values from row i from data frame one is equal to row j from data frame two. To provide some context, this will be used in a propensity score matching algorithm to identify candidate matches that match exactly on any number of covariates. In addition to the approaches I had, joran provided an approach using the Vectorize function (thanks again as I learned another nice function). I decided to put three approaches to a race…

To understand what I need, I’ll start with a small example with two data frames, one with 4 rows, the other with 3, and each has two variables, one logical and the other numeric. As an aside, I only need this to work for integers, factors, characters, and logical types therefore avoiding issues of comparing numerics.

> df1 <- data.frame(row.names=1:4, var1=c(TRUE, TRUE, FALSE, FALSE), var2=c(1,2,3,4))
> df2 <- data.frame(row.names=5:7, var1=c(FALSE, TRUE, FALSE), var2=c(5,2,3))
> df1
   var1 var2
1  TRUE    1
2  TRUE    2
3 FALSE    3
4 FALSE    4
> df2
   var1 var2
5 FALSE    5
6  TRUE    2
7 FALSE    3

First, let’s consider the case when there is only one variable:

> system.time({
+ 	df3 <- sapply(df2$var1, FUN=function(x) { x == df1$var1 })
+ 	dimnames(df3) <- list(row.names(df1), row.names(df2))
+ })
   user  system elapsed 
      0       0       0 
> df3
      5     6     7
1 FALSE  TRUE FALSE
2 FALSE  TRUE FALSE
3  TRUE FALSE  TRUE
4  TRUE FALSE  TRUE

This is pretty straight forward. Now I want the same type of result, but to compare more than one column (in the final implementation I need to handle any number of columns so not necessarily limited to one or two).

The first approach uses nested apply functions.

> system.time({
+ 	m1 <- t(as.matrix(df1))
+ 	m2 <- as.matrix(df2)
+ 	df4 <- apply(m2, 1, FUN=function(x) { apply(m1, 2, FUN=function(y) { all(x == y) } ) })
+ })
   user  system elapsed 
  0.001   0.000   0.001 
> df4
      5     6     7
1 FALSE FALSE FALSE
2 FALSE  TRUE FALSE
3 FALSE FALSE  TRUE
4 FALSE FALSE FALSE

Secondly, using the Vectorize and outer functions.

> system.time({
+ 	foo <- Vectorize(function(x,y) { all(df1[x,] == df2[y,]) })
+ 	df5 <- outer(1:nrow(df1), 1:nrow(df2), FUN=foo)
+ })
   user  system elapsed 
  0.005   0.000   0.006 
> df5
      [,1]  [,2]  [,3]
[1,] FALSE FALSE FALSE
[2,] FALSE  TRUE FALSE
[3,] FALSE FALSE  TRUE
[4,] FALSE FALSE FALSE

Lastly, we’ll create a new character vector by pasting the other variables together.

> system.time({
+ 	df1$var3 <- apply(df1, 1, paste, collapse='.')
+ 	df2$var3 <- apply(df2, 1, paste, collapse='.')
+ 	df6 <- sapply(df2$var3, FUN=function(x) { x == df1$var3 })
+ 	dimnames(df6) <- list(row.names(df1), row.names(df2))
+ })
   user  system elapsed 
  0.000   0.000   0.001 
> df6
      5     6     7
1 FALSE FALSE FALSE
2 FALSE  TRUE FALSE
3 FALSE FALSE  TRUE
4 FALSE FALSE FALSE

We can already see with this small example that the Vectorize approach is the slowest. However, let’s try a larger example. First we’ll create two data frames, one with 1,000 rows and the second with 1,500. The resulting matrix will be 1,000 x 1,500.

set.seed(2112)
df1 <- data.frame(row.names=1:1000, 
				  var1=sample(c(TRUE,FALSE), 1000, replace=TRUE), 
				  var2=sample(1:10, 1000, replace=TRUE) )
df2 <- data.frame(row.names=1001:2500, 
				  var1=sample(c(TRUE,FALSE), 1500, replace=TRUE),
				  var2=sample(1:10, 1500, replace=TRUE))

Nested apply functions approach:

> system.time({
+ 	m1 <- t(as.matrix(df1))
+ 	m2 <- as.matrix(df2)
+ 	df4 <- apply(m2, 1, FUN=function(x) { apply(m1, 2, FUN=function(y) { all(x == y) } ) })
+ })
   user  system elapsed 
 10.807   0.043  11.096

Vectorize approach:

> system.time({
+ 	foo <- Vectorize(function(x,y) { all(df1[x,] == df2[y,]) })
+ 	df5 <- outer(1:nrow(df1), 1:nrow(df2), FUN=foo)
+ })
   user  system elapsed 
390.904   0.808 392.134

Combined columns approach:

> system.time({
+ 	df1$var3 <- apply(df1, 1, paste, collapse='.')
+ 	df2$var3 <- apply(df2, 1, paste, collapse='.')
+ 	df6 <- sapply(df2$var3, FUN=function(x) { x == df1$var3 })
+ 	dimnames(df6) <- list(row.names(df1), row.names(df2))
+ })
   user  system elapsed 
  0.421   0.000   0.422

The combined column approach is by far the fasted way, and it makes good since. It is a bit surprising (at least to me), how much worse the Vectorize and outer functions are. Moreover, I am a bit concerned about potential issues with the paste method and doing comparisons on those results. Please feel free to leave comments below if there are other approaches.

Read full post...

Version 1.0 of sqlutils available on CRAN

2013-01-15T00:00:00-08:00

Version 1.0 of sqlutils has been released to CRAN. The sqlutils package is designed to manage a library of SQL files. This package grew out of the needs of an Office of Institutional Research where the vast majority of analysis is conducted on data from our Student Information System (SIS) which is stored in an Oracle database. A lot of our analyses and reports are derived from the same types of datasets but from easily extracted parameters (e.g. date range, program name, status, etc.). We used to store SQL commands in our R scripts but that can become quite cumbersome and in many ways, reduced the ease of reusability which is a major reason for using R in the first place, hence the birth of sqlutils. For our purposes we currently have over 40 SQL files that have been well vetted and documented. To share the library we simply add the following to our .Rprofile script:

require(sqlutils)
sqlPaths('/Path/to/shared/directory')

A full introduction to the squtils package is available here as well as on the Github project page. A key advantage to using sqlutils is that you can store your queries in plain text files (with a .sql file extension) and document them using roxygen2 style comments. Moreover, R function parameters are used to set parameters within the SQL command. Parameters are defined in SQL files using colon, parameter name, colon (i.e. :paramName:) format. Using this framework, it is easy to create a data dictionary of the library of SQL files.

Lastly, I wrote about an interactive SQL mode in R a few days ago. The isql function is included in the sqlutils package.

Read full post...

Interactive SQL in R

2013-01-12T00:00:00-08:00

I recently taught a very basic introduction to SQL workshop and needed a way to have participants interact with SQL statements. Obviously there are lots of tools to interface with a database, but since we are all R users I thought it would be nice to be able interact without leaving R. Although this interface is fairly basic, the fact that we can type in a SQL statement and get the results as an R data frame provides all the advantages of having data in R. Moreover, I found this to be an interesting exercise in see the power of R as programming language, not just as statistical software. The function described here is part of the sqlutils package which was created to manage a library of SQL files. More information about that is provided on the project page and I will likely have a forthcoming blog post too.

First we need to create a database to interact with. In this example we will use the students data frame from the retention package. We will save this data frame into a SQLite database using the RSQLite package. The R code to setup the database is provided as a demo in the package. Type demo('isql') to start.

require(sqlutils)
require(RSQLite)
require(retention)
data(students)
students$CreatedDate = as.character(students$CreatedDate)
m <- dbDriver("SQLite")
tmpfile <- tempfile('students.db', fileext='.db')
conn <- dbConnect(m, dbname=tmpfile)
dbWriteTable(conn, "students", students[!is.na(students$CreatedDate),])

We begin an interactive SQL environment with the isql function. The only required parameter is conn which is the connection to the database that SQL statements will be executed. The sql parameter is optional and sets the initial SQL statement for the session that can be edited or executed.

> hist <- isql(conn=conn, sql=getSQL('StudentSummary'))
Interactive SQL mode (type quit to exit, help for available commands)...
SQL>
help
   Command      Description
   ___________  ______________________________________________________
   quit         quit interactive mode
   help         display this message
   sql          enter SQL statement
   edit         edit SQL in a separate text window
   print        print the last entered SQL statement
   exec         execute that last entered SQL statement
   result       prints the last results
   save [name]  save the last executed query to the global environment
SLQ>
print
SELECT CreatedDate, count(StudentId) AS count FROM students GROUP BY CreatedDate ORDER BY CreatedDate
SLQ>
edit

SLQ>
print
SELECT CreatedDate, count(StudentId) AS count FROM students GROUP BY CreatedDate ORDER BY CreatedDate
SLQ>
exec
Executing SQL...
118 rows of 2 variables returned
SLQ>
save
Data frame sql.results saved to global environment
SLQ>
quit

The isql function returns the history of the session invisibly (that is the results will not be printed but can be assigned to a variable). There are two elements in the returned list, commands is a character vector listing all the commands entered and sql is a character vector containing all the SQL statements entered.

> names(hist)
[1] "sql"      "commands"

Read full post...

Reading Codebook Files in R

2013-01-10T00:00:00-08:00

One issue I continuously encounter when starting to work with a new dataset is that of the codebook. In general, I prefer to load a codebook into R like any other data source, specifically as a data frame. And ideally, one data frame to provides the variable names with descriptions and any other meta data available, and a separate list of named vectors that can be used to recode factors. Although there is no standard format for codebooks, most follow a similar format. This post outlines the parse.codebook function that will read codebooks that have the following features:

Each line in the file provides information about a variable (which I refer to as a variable row), or the mapping of factor (which I refer to as a level row).
Variable rows start on the left edge (that is, there is a non-whitespace character at position 1 of the row).
Level rows do not start on the left edge (that is, there is a whitespace character at position 1 of the row, for example a tab or space).
Rows are either fixed (see ?read.fwf for more information as to specifics) or character delimited (e.g. comma, colon, etc.).

Although all codebooks may not strictly adhere to these rules, it is often trivial, even if not a bit tedious, to reformat the file to adhere to these rules. Also, blank lines are permissible and will simply be ignored.

If the codebook file adheres to these rules, the parse.codebook function will parse the file and return an object of type codebook that inherits from data.frame, therefore all the data frame functions are valid (e.g. head, nrow, names, etc.). This data frame contains all the information about the variables vis-a-vis the variable rows. Information about factor levels are stored in a list as an attribute of the returned object which can be retrieved using attr(mycodebook, 'levels'). Example from the Common Core of Data and the American Community Survey are provided below.

Installation

The source.codebook function is currently provided on Gist. You can either download the R script file or source it directly from Gist using the devtools package.

require(devtools)
source_gist(4497585)

Parameters

The parse.codebook has a number of parameters to indicate the format of variable and level rows. The function will handle both character delimited rows and fixed with rows. Therefore, either var.sep or var.widths must be specified as well as level.sep or level.widths. The available parameters are:

file codebook file name.
var.names the name of the columns for variable rows.
level.names the name of the columns for level rows.
var.sep the separator for variable rows.
level.sep the separator for level rows.
level.indent character vector providing character(s) at the beginning of the line that indicate the line represents a factor level. Each element should have 1 character as only the first character of the line is compared.
var.name the name in var.names that represents the variable name. This should be a valid R variable name as this will be the column name in the corresponding data file, as well as the name used in the list of levels stored as an attribute to the returned object.

Example One: Common Core of Data

The Common Core of Data (CCD) is a dataset provided by the National Center for Education Statistics that provides information about K-12 schools in the United States. The codebook provided is in plain text and required two modifications: One, general file information at the top of the file was deleted, and two, any descriptions that spanned lines need to be modified so the are on only one line. Here are the first 15 lines of the modified file, the full file can be downloaded at here

SURVYEAR      1      AN     Year corresponding to survey record.

NCESSCH       2      AN     Unique NCES public school ID (7-digit NCES agency ID (LEAID) + 5-digit NCES school ID (SCHNO).    

FIPST         3      AN     American National Standards Institute (ANSI) state code..

                             01  =  Alabama        
                             02  =  Alaska          
                             04  =  Arizona
                             05  =  Arkansas       
                             06  =  California      
                             08  =  Colorado
                             09  =  Connecticut    
                             10  =  Delaware        
                             11  =  District of Columbia

This codebook uses fixed withs for variable rows, and separators (using the equal sign) for level rows (although it also possible to use fixed with for level rows as well). First, we will parse the file:

ccd.codebook <- parse.codebook('ccdCodebook.txt', 
				var.names=c('variable','order','type','description'),
				level.names=c('level','label'),
				level.sep='=', 
				var.widths=c(13, 7, 7, Inf) )

Here are the first six rows of the returned data frame.

> head(ccd.codebook)
  linenum variable order type                                                                                    description isfactor
1       1 SURVYEAR     1   AN                                                           Year corresponding to survey record.    FALSE
2       3  NCESSCH     2   AN Unique NCES public school ID (7-digit NCES agency ID (LEAID) + 5-digit NCES school ID (SCHNO).    FALSE
3       5    FIPST     3   AN                                      American National Standards Institute (ANSI) state code..     TRUE
4      67    LEAID     4   AN                                                          NCES local education agency (LEA) ID.    FALSE
5      69    SCHNO     5   AN                                                                                NCES school ID.    FALSE
6      71     STID     6   AN                                                       State?s own ID for the education agency.    FALSE

In addition to the columns corresponding to var.names, the function also returns a linenum and isfactor column. The former is an integer corresponding to the line number in the original file from which this row was parsed. This is useful for tracking down issues in the parsing or text formatting. The isfactor is a logical column indicating whether there are factor levels specified for that variable. Factor levels can be retrieved as follows:

> ccd.var.levels <- attr(ccd.codebook, 'levels')
> names(ccd.var.levels)
[1] "FIPST"  "TYPE"   "STATUS" "TITLEI" "STITLI" "MAGNET" "CHARTR" "SHARED"
> ccd.var.levels[['TYPE']]
  linenum level                    label
1     103     1           Regular school
2     105     2 Special education school
3     107     3        Vocational school
4     109     4 Other/alternative school
5     111     5       Reportable program

Example Two: American Community Survey

The American Community Survey is the current version of the Census Long Form. The codebook provided by the United Census Bureau is in PDF format, but is easily converted to a plain text file. This file required more modification that the CCD file described above, mostly removing line numbers that pasted over from the PDF as well as ensuring that descriptions did not span lines. The final modified version can be downloaded (here)http://jason.bryer.org/codebook/acsPersonCodebook.txt. Here are the first 10 lines of the file:

SPORDER .Person number
ST .State Code
	01 .Alabama/AL
	02 .Alaska/AK
	04 .Arizona/AZ
	05 .Arkansas/AR
	06 .California/CA
	08 .Colorado/CO
	09 .Connecticut/CT
	10 .Delaware/DE

For this codebook file, all rows are character delimited on . (space period). We parse the file as follows:

acs.codebook <- parse.codebook('acsPersonCodebook.txt', 
				   var.names=c('var','desc'), 
				   level.names=c('level','label'),
				   var.sep=' .', level.sep=' .')

The first six lines of the returned data frame are:

> head(acs.codebook)
      var                                                                                desc linenum isfactor
1 SPORDER                                                                       Person number       1    FALSE
2      ST                                                                          State Code       2     TRUE
3  ADJINC Adjustment factor for income and earnings dollar amounts (6 implied decimal places)      55    FALSE
4   PWGTP                                                                     Person's weight      56    FALSE
5    AGEP                                                                                 Age      57    FALSE
6     CIT                                                                  Citizenship status      58     TRUE

And factor levels:

> var.levels <- attr(acs.codebook, 'levels')
> names(var.levels)
 [1] "ST"      "CIT"     "COW"     "DRAT"    "ENG"     "GCM"     "JWRIP"   "JWTR"    "MAR"     "MARHM"  
[11] "MARHT"   "MARHW"   "MIG"     "MIL"     "NWAV"    "RELP"    "SCH"     "SCHG"    "SCHL"    "SEX"    
[21] "WKL"     "WKW"     "WRK"     "ANC"     "ANC1P"   "ANC2P"   "DECADE"  "DIS"     "DRIVESP" "ESP"    
[31] "ESR"     "FOD1P"   "6402"    "FOD2P"   "HICOV"   "HISP"    "INDP"    "JWAP"    "JWDP"    "LANP"   
[41] "MIGSP"   "MSP"     "NAICSP"  "NOP"     "OCCP02"  "OCCP10"  "PAOC"    "POBP"    "POWSP"   "PRIVCOV"
[51] "PUBCOV"  "QTRBIR"  "RAC1P"   "RAC2P"   "RAC3P"   "SFN"     "SFR"     "SOCP00"  "SOCP10"  "VPS"    
[61] "WAOB"    "FHINS3C" "FHINS4C" "FHINS5C"
> var.levels[['CIT']]
  linenum level                                                                        label
1      59     1                                                             Born in the U.S.
2      60     2 Born in Puerto Rico, Guam, the U.S. Virgin Islands, or the Northern Marianas
3      61     3                                            Born abroad of American parent(s)
4      62     4                                               U.S. citizen by naturalization
5      63     5                                                    Not a citizen of the U.S.

Conclusion

Although a standard codebook format doesn’t exist, most adopt a similar format. I have outlined the parse.codebook function that, with minimal reformatting of the original codebook file, be used to read a codebook into R. This is tremendously useful as we can now merge in variable descriptions when creating tables and figures, as well as recode factors with their longer descriptions in an automated fashion.

Read full post...