Program Evaluation Proposal

A Proposal to Evaluate the Graduate Student Writing Studio

DISCLAIMER: This proposal is hypothetical. No such evaluation has been requested. There are no plans to conduct such an evaluation, nor is the funding for the Graduate Student Writing Studio under review. Dr. Palm was interviewed for this proposal with the understanding that it was for a course requirement and only hypothetical. Fall 2010

Background and Rationale

The Graduate Student Writing Studio (GSWS, or the Writing Studio) was established five years ago by the College of Education (COE) at the University of New Mexico (UNM) to address perceived deficiencies in the quality of graduate student academic writing.  Rebel Palm, Ph.D., has led the Writing Studio as Director since its inception. Beginning with two graduate assistants, she now employs four graduate assistants who each work from 10-20 hours per week.   With Dr. Palm herself serving as a tutor, GSWS employs approximately two full-time equivalents.  The Studio is funded by graduate student fees and typically assists 130-140 COE graduate students each semester.  As a service available exclusively to graduate students in the College of Education, the GSWS program has no counterpart on the UNM Main Campus, although the Center for Academic Program Support (CAPS) has recently begun offering writing assistance to graduate students across the campus.

Dr. Palm reports that most students seek tutoring on more than one writing project, which suggests students find value in the services provided by the GSWS tutors.  Some professors within COE regularly refer their students to the Studio, while others do not. Although she suspects that students who use the services offered by the Writing Studio are demographically representative of the graduate student population within the COE, her records and assessment surveys do not provide that type of sensitive demographic data.

There has been no formal evaluation of GSWS in its five year history.  Dr. Palm regularly sends out an assessment survey at the end of each semester to students who have received tutoring assistance.  She estimates a return rate of about 60%.  These assessments provide feedback regarding the students’ degree of satisfaction with the tutoring as well as the quality of the individual tutors who assisted them.

Given the continued budgetary pressures faced by the University, Richard Howell, Dean of the College of Education, prompted by a challenge from certain departments who believe the budget allocated to the Graduate Student Writing Studio could be better spent elsewhere, has requested a comprehensive evaluation of the Writing Studio (the evaluand).

Purpose

The purpose of the proposed evaluation, to be completed over the course of the Spring 2011 semester, is to: 1) assess the quality of the tutoring services GSWS provides to COE graduate students; 2) evaluate the value proposition that the Writing Studio, as presently implemented, provides to COE; and 3) provide a range of alternatives to Dean Howell regarding the future of GSWS in terms of how COE and its graduate students can realize the most cost-effective results from the Writing Studio.

Stakeholders

Primary stakeholders consist of the leadership of the College of Education and the Writing Studio.  Dean Howell is responsible for all academic programs and operations within the College of Education. As the decision maker for  all budgetary matters, Dean Howell will use the results of the evaluation in determining future budget levels for the GSWS program.  He will also consider how the three alternatives to be provided by the evaluation (status quo, contract the program, or expand the program) can best position GSWS fits into the overall COE five-year strategy.  He will be asked to participate in an interview at the beginning of the evaluation period, and his office will be asked to provide historical financial data for GSWS.  Otherwise, he will not contribute to the evaluation.   He will receive the four Monthly Progress Reports, the Final Report, and be invited to the presentation of the Final Report.  As the decision maker for COE, Dean Howell will have the authority to implement any strategic decisions resulting from this proposed evaluation.

The heads of the six academic departments reporting directly to Dean Howell are also primary stakeholders.  As responsible for the academic performance of their departments, and in their leadership roles in directing the COE faculty, they will be interested in the execution of this evaluation as well as its results.  They will each be individually interviewed for this evaluation, receive the Monthly Progress Reports, the Final Report, and be invited to the presentation of the Final Report.

Other primary stakeholders include Dr. Palm and her staff of tutors, as they contribute directly to the evaluation and will be affected directly by the evaluation.  Over the course of the evaluation, they will be asked to participate in individual interviews, one group interview, and one survey.  It is recognized that as vested stakeholders with personal interests in the outcome of the evaluation, the tutors and Dr. Palm herself are not impartial participants. However, their participation is required in order to fully assess the overall value proposition of the GSWS program.  Their contributions to the evaluation will therefore be accepted and considered with this in mind.

Secondary stakeholders include the current faculty and graduate students in the College of Education who would be affected by any changes to the tutoring program. These include those faculty and students who use the services of the Writing Studio, as well as those who do not.  Tertiary stakeholders with less immediate interests in the outcome of the evaluation are future COE graduate students. Additional tertiary stakeholders who could be affected by the results of the evaluation include graduate programs in other colleges on the UNM campus, depending on the recommendations produced by the evaluation.  For example, the services offered by CAPS could be affected by the results of the evaluation.  The secondary and tertiary stakeholders will not be asked to contribute to the evaluation.  Distribution of the monthly and final reports to these shareholders will be at the discretion of the COE department heads.

It should be noted that GSWS will continue to provide tutoring services to students during the course of the evaluation.  The evaluation has been designed to minimize the impact on the tutors and the students and not interfere with their important work.  Ultimately, the results of the interactions between these two parties drives this evaluation.  The evaluation will therefore focus on these individuals and value their inputs and comments as most critical data.  While there might be merit in pursuing a broader survey of COE faculty and students regarding their opinions about the GSWS program, that is not the purpose of this evaluation.

Key Questions

The key questions to be addressed during the proposed evaluation include:

1. How successfully has the Graduate Student Writing Studio achieved its stated purpose to improve the quality of graduate student writing from the perspectives of:
   • COE graduate students who use the tutoring services?
   • COE Department Heads, as communicated through their faculty members?
   • the GSWS tutors?
2. What value does the Writing Studio provide to COE in its delivery of tutoring services in terms of cost-effectiveness and quality of service?
3. What factors should be considered by COE administration in deciding the future direction of the Graduate Student Writing Studio?

Evaluation Design

The proposed evaluation is formative in nature. It will assess an ongoing program with the intent of improving the program for future implementation, and it will provide well-justified alternative strategies for Dean Howell to consider regarding GSWS.  Because the purpose of the proposed evaluation is to determine the effectiveness of an ongoing program, the behavioral objectives evaluation approach will be employed.  This approach is best adapted to determinations of whether or not a program is meeting its objectives.

A mixed method approach has been selected for this evaluation.  While judgments about writing and improvements in writing are inherently subjective, quantitative data do exist and can be collected throughout the evaluation to provide a productive balance of qualitative and quantitative techniques.  Qualitative observations and assessments will be derived from analysis of archived assessments from prior years, personal interviews, and group interviews.  In keeping with a qualitative approach, data collected from verbal sources (previous assessments and the interviews will be analyzed and summarized in narrative form in order to provide a “thick description” of the how GSWS tutoring is delivered and how it results in benefits to the students.

A quantitative approach will be employed in analyzing the survey results, and in coding and characterizing data from the prior assessments so that both the previous and new data can be aggregated for analysis.  Financial data will factor into determinations about the value proposition delivered by GSWS.  Financial metrics will be developed for analysis and comparison, such as average cost per student, average cost per hour, and the average value of benefit as perceived and reported by students.

Data Collection and Sampling

The intent of the proposed evaluation is to collect data from all primary stakeholders and student participants during the evaluation period, as well as all available archived data.  Therefore sampling is not applicable as the objective of this evaluation is to provide a census of direct participants (tutors and students) of the GSWS program.  It is acknowledged, however, that some students will elect to not participate in the evaluation, therefore it is assumed that 60% of students will complete the survey (estimated 84 of 140) and 25% will agree to provide writing samples (35 of 140).

The available archived data consist of post-semester assessment surveys completed by GSWS students in prior years.  According to Dr. Palm, approximately 60% of past students completed and returned the surveys, so it is estimated that approximately 336 should be available for analysis.  Other archived data assumed to be available to the evaluator are financial records pertaining to GSWS for the past five years, including salaries paid, cost of computer resources, supplies, and facilities charges. [TASK #3 on the Management Plan.]

Upon contract award, the first task will be to schedule individual interviews with Dean Howell and Dr. Palm. Prior to the scheduling the interviews, both Dean Howell and Dr. Palm will be asked to provide a short declaration of 500-750 words that generally expresses what they want the evaluation to accomplish.  These statements will be provided to the evaluator and available at least two days prior to the interviews.  The purpose of these interviews will be to gather information regarding their personal expectations for and inputs to the evaluation process.   Inputs will include their desires for specific types of data or analysis output, as well as their initial feedback on the preliminary content of the evaluation instruments (interview and survey scripts). [TASK #1]  The evaluator will incorporate their individual feedback comments into a second iteration of the instruments, obtain their feedback, and continue to revise until both Dean Howell and Dr. Palm concur with the formats and contents of the interview and survey scripts.  As part of the instrument finalization process, the student surveys will be piloted tested with students from prior semesters. [TASK #2]

The heads of the six departments within the College of Education will be individually interviewed as soon as can be scheduled after the evaluation instruments have been approved.  [TASK #4]  The purpose of these interviews is to gather impressions, opinions, and comments reflecting the perspectives of faculty regarding the Writing Studio.  These, and all subsequent individual and group interviews, will follow a similar protocol.  The interviews will be conducted by the evaluator and assistant.  The evaluator will ask permission to audio record each interview.  The evaluator and assistant will each take notes on pre-printed forms that contain the script questions and room for notes in order to compare notes and observations afterwards.

The five graduate student tutors, including Dr. Palm, will be scheduled for individual interviews as can be mutually scheduled. [TASK #5]  There is no time or sequence requirement for the staff interviews.  The purpose of these interviews is to give the tutors an opportunity to express their personal feelings and opinions about the general performance of GSWS, tutoring philosophies and processes, notable successes and failures/frustrations, critiques of the current program, and suggestions for how it might be improved.

The tutors will also be asked to individually complete a survey designed specifically to capture their responses to the items of concerns expressed by Dean Howell and Dr. Palm. [TASK #6]  This survey will be constructed using multiple choice and Likert-type formats in which the respondents will indicate their degree of agreement or disagreement with certain statements regarding the GSWS program. This survey will be provided to the tutors, including Dr. Palm, during the second half of the semester. They will have approximately two weeks to complete and return the survey to the evaluator.

Approximately one month prior to the end of the semester, the evaluator will schedule a group interview with all tutors. [TASK #9]  The purpose of the group interview is to allow a free exchange of ideas and discussion among the tutors in order to capture any relevant data (opinions, suggestions, frustrations, anecdotes, etc.) not already collected, as well as to confirm or discuss in more detail findings from the staff surveys. The evaluator recognizes that during these final weeks of the semester, the demands for time from the staff may make scheduling a meeting with all tutors difficult. Should this prove the case, the evaluator will use his discretion in deciding to either schedule the group interview with one tutor absent, or delay the group interview until after finals week in order to obtain the best possible attendance.

The evaluator is cognizant of the fact that students may not wish to participate in the evaluation for a variety of reasons, including time, personal embarrassment, or unwillingness to obligate themselves. Nevertheless, participation from current graduate students seeking assistance from the writing tutors is vital to assess how effectively the GSWS program is performing its objectives, therefore student participation is vital to the success of the evaluation.

The evaluator will ask each tutor to review the following information, provided in the form of a one-page consent form, with each participant prior to the initiation of each tutoring engagement:

1. The Dean of the College of Education and the Director of the GSWS have requested an evaluation of the Writing Studio and its services to the COE graduate student community.
2. Each of the writing tutors is participating in the evaluation.
3. Each COE graduate student who seeks assistance from GSWS tutors will be asked to participate. Participation is strictly voluntary and will not affect the quality of the tutoring the student will receive.
4. Names of participants will not be released to the evaluator. Assigned control numbers will be used to track responses.
5. Students who elect to participate can withdraw at any time.
6. Students who elect to participate will be asked to complete an online survey following their tutoring engagement. Individual survey results will not be released to GSWS staff.
7. Selected participants may be asked to contribute to a one-hour group interview. Participation in the online survey does not commit the student to participate in a group interview if asked.
8. Participants will be asked to provide writing samples that illustrate the effects of their individual tutoring engagement. These samples will be provided through the tutors with student names replaced by assigned control numbers.
9. Participant signature.

It is anticipated that it should require no more than five minutes of the appointment time for the tutor to review the above consent information with the student. Should this not be the case, the evaluator and staff may jointly decide to make the consent form available to potential participants via email prior to the scheduled appointment time in order to more quickly facilitate its completion.

Participants will be contacted after each tutoring session via email by their tutor and given instructions to complete an online survey. [TASK #7]  The tutor will provide the participant with an assigned control number to use to authenticate their online survey completion. The participant survey questions will concentrate on the participant’s perceptions of value received from the tutoring session, satisfaction with the tutor, whether or not expectations were met, and benefits received (increased confidence, knowledge, better grade, etc.) The survey will consist of twenty questions using multiple choice and Likert-type formats for discrete responses, and one text area for general comments.

From those online surveys completed, 24 participants will be selected to participate in one of four participant group interviews. Two group interviews will be comprised of six participants per group who have sought assistance from GSWS more than once (repeat students). Two group interviews will be comprised of six participants per group who have only sought assistance once. Each group interview will be loosely structured to allow the repeat students to discuss why they have returned to the Writing Studio, and first-timers to discuss their experiences with GSWS and the likelihood they will return. These group interviews will be scheduled as soon as practicable so they are completed well before the end of semester.

The final data to be collected will be writing samples offered by the participants. At the completion of the online survey, each student will be provided with instructions on how to submit before/after writing samples to their tutor. They may submit either selections from a paper or an entire paper. They will be requested to remove all identifying information from the papers, save each using a prescribed file naming convention that uses their individual control number, and attach as an email to their tutor. The tutors will then forward the attached files to the evaluator for analysis.

In summary, the data collection methods and number of collected items will include:

[TASK #1]  Individual interviews with Dean Howell and Dr. Palm. (2)

[TASK #4]  Individual interviews with the six COE department heads. (6)

[TASK #5]  Individual interviews with the staff of tutors.  (5)

[TASK #9]  Group interview with staff of tutors (5 individuals). (1)

[TASK #6]  Staff surveys of 20 questions plus comments. (5)

[TASK #7]  Student surveys of 20 questions plus comments. (estimate 60% of 140, or 84)

[TASK #10]  Student participant group interviews (6 individuals each). (4)

[TASK #8]  Writing samples from participants. (estimate 25% of 140, or 35)

[TASK #3]  Archived assessment surveys from prior years (estimate 60% of 140 x 4 years, or 336).   Financial data from prior and current years.

Internal validity has been considered in these data collection designs.  The required approvals by Dean Howell and Dr. Palm should assure that the content of the interviews and surveys are directly applicable to their interest.  The interviews will be conducted by the interviewer and an assistant to provide corroboration of observations and inferences.  The student surveys will be pilot tested prior to use in order to ensure readability, interpretation, and relevance.  And the interim reports for Dean Howell, the department heads, and Dr. Palm will preclude deviations from the plan and allow immediate corrective actions should events dictate.

Sample Topics and Questions for Data Collection Instruments

As previously noted, the data collection instruments will be constructed with input from Dean Howell and Dr. Palm and will be reviewed and approved by them before they are finalized for use. The following examples are representative of topics and questions that will be initially offered to the Dean and Dr. Palm for consideration.

Interviews with Dean Howell and Dr. Palm:

a)     Without respect to cost, what do you believe is the ideal solution for eliminating deficiencies in graduate student writing?

b)    What are your general impressions from students and faculty regarding the success of the GSWS program?

c)     What do you believe distinguishes good student writing from bad?

d)    At a department or college level, how do you assess whether the quality of writing is acceptable or not?

e)     How might this evaluation fail your expectations?

f)     What three pieces of data are you most interested in obtaining through this evaluation?

Interviews with department heads:

a)     To what degree does the faculty in your department refer students to GSWS?

b)    What do your faculty members report in terms of improvement or benefits that students gain from GSWS tutoring?

c)     What reasons do your faculty member give for not referring students to GSWS?

d)    How could GSWS provide better tutoring to students in your department?

Interviews with tutoring staff:

a)     What are the most common reasons students give for seeking tutoring assistance?

b)    How many students have you assisted? How many were repeats?

c)     What training or preparation did you receive prior to working as a writing tutor?

d)    How would you assess the general quality of work that students bring to the Writing Studio, in terms of a) organization and content; b) grammar, vocabulary, and spelling; and c) appropriate format and style?

e)     What are your frustrations about working as a tutor?

f)     What would allow you to do a better job as a tutor?

Student and Tutor surveys (modified accordingly)

a)     How many times have you come to the Writing Studio for writing assistance?

b)    How satisfied have you been with the assistance you’ve received?

c)     Why did you initially decide to seek assistance from the Writing Studio?

d)    Prior to coming to GSWS, how confident were you about the content and organization of your writing? Your grammar, vocabulary, and spelling? Your compliance with the appropriate formatting and style?

e)     What did you find most valuable about your experience with the writing tutor?

f)     If the GSWS services were not available to you, would you have sought assistance elsewhere? Where?

g)     How much time did you spend preparing for your tutoring appointment?

h)    What’s the most you would have paid in out-of-pocket for the services you received at GSWS? In other words, how much monetary value would you place on the benefits you received from the Writing Studio?

Evaluation Project Management Plan

The proposed evaluation will be executed over a 22-week period during the Spring 2011 semester and include the 13 Tasks identified previously.  Key assumptions include:

1. Contract Award for the evaluation will occur no later than January 3, 2011 to allow for adequate preparation prior to evaluation kickoff with Dean Howell and Dr. Palm.
2. Archived assessment data will be made available to the evaluator by mid-February.
3. Any students that do come to the Studio prior to final approval of the evaluation instruments will not be asked to participate in the evaluation.
4. The Final Report to Dean Howell and Dr. Palm will be due on June 15, 2011.
5. The evaluation will not be subject to Institutional Review Board (IRB) approval.

Data Analysis

Qualitative techniques will be used in analyzing data collected via interviews, archived survey assessments, and the participant writing samples. These sources will produce primarily verbal data that must be subjectively interpreted and analyzed, therefore the evaluator will develop, as appropriate, coding, classifying, and summarizing schemes that fit the data in order to  provide the types of information and level of detail necessary to meet the objectives of the evaluation. These primarily categorical data will provide measures of frequency that can be graphically depicted using bar and pie charts, tables, or other techniques to best visually represent the results. Narrative samples of text and statements from the group interviews may be used to provide rich description to further illustrate specific conclusions or assessments.  The evaluator will contract directly with some number of faculty members outside of CO to offer independent assessments of the before/after writing samples.  The criteria for these assessments will be approved with other evaluation instrument requirements by Dean Howell and Dr. Palm.

Quantitative techniques including descriptive statistics will be used to analyze the survey results. These can also be depicted in appropriate graphical formats (bar and line charts, histograms, etc.) in order to best present data associated with prevalence of attitudes, ratings, etc.  Financial data will be analyzed and computed in order to provide the specified metrics, including cost per student, cost per hour, and value of perceived benefits.  The key factor in determining the specific course of the survey data analyses will be the requirements established by Dean Howell and Dr. Palm as the evaluation instruments are created and finalized.

Evaluation Constraints

The fundamental constraints that may threaten the success of this evaluation relate primarily to the willing and committed participation by the primary stakeholders and the student participants throughout the semester.  Each step of the data collection process involves participation in the form of candid, truthful, and accurate responses to the pertinent questions.   The guiding consideration that the evaluators must continually communicate is that this evaluation serves the purposes of everyone involved — everyone wants to provide the best possible services to COE graduate students.

However, any effort to “improve” an ongoing service inevitably runs the risk of being perceived in a negative sense in that some deficiency must usually be identified in order for “improvement” to occur. Therefore the evaluators must be especially sensitive to the possibility of negative perceptions of questions and comments during all interviews.

It is vital to the success of the evaluation that the students who seek assistance from GSWS during the period of the evaluation participate in the interviews, survey, and offering their writing samples for analysis.  If, for whatever reasons, a sufficient number of student participants cannot be obtained, the evaluation will lack a critical component.

Communicating and Reporting

The management plan for the evaluation provides for four monthly progress reports to be delivered to Dean Howell, the department heads, and Dr. Palm.  These interim reports will keep them informed as to accomplishments, status, and issues that may need to be addressed or considered as the evaluation proceeds.  These reports will be submitted as PDF file attachments to email and include: a) accomplishments completed during the reporting period; b) any variances to the planned schedule; c) interim results or items or interest as specified during the instrument finalization process; d) accounting for how many students have elected to become participants, volunteered writing samples, etc.  Distribution beyond these primary stakeholders will be at their discretion.

The Final Report will include a written report of approximately 20 pages that provides a narrative summary of the evaluation process, analysis and significance of data collected, and articulates the considerations and forecasted consequences of three alternative scenarios — the status quo, a contraction of GSWS service, and an expansion of services.  This report, with the explanation of alternatives, will inform Dean Howell and his leadership team of department heads as to how best to utilize the GSWS resources to maximize its value to COE in the coming years.

Cost Estimates and Pricing Support

The proposed price to conduct the evaluation is $25,724. A detailed cost breakdown is provided in the table below. This estimate is based on the following assumptions:

1. The evaluation will be conducted by a local independent contractor who has no affiliation with the University of New Mexico.
2. Appropriate support, estimated below, from University employees will be provided at no cost to the contractor.  Estimated requirements for UNM employee support is included in the table below for informational purposes only.
3. The contractor will arrange for expert faculty review of writing samples through direct contract with the faculty members, therefore that cost is included in the proposed bid.
4. Work space will be provided to the evaluators adequate for conducting private individual interviews and group interviews.
5. The contractor will provide all necessary computer resources.
6. All deliverables (Progress Reports and the Final Report) will be delivered electronically in PDF format. Costs for scanning or re-formatting source data into electronic format are included in the contractor’s hours under Assistant.