Home General Documents Institutional Report Conceptual Framework
Exhibit Room Louisiana St. Supp. Rep. SPA Reports  

 

LIST OF EXHIBITS FOR STANDARD 2

Standard 2

Assessment System and Unit Evaluation 

2a. Assessment System

Evaluating and refining assessment system.  Since the 2002 NCATE Accreditation Visit, the Unit assessment system has undergone changes, most notably the adoption of PASS-PORT, the electronic system for assessing candidates’ accomplishments.  The series of review points begins with a request to the assessment committee.  This group comprised of unit faculty, Director of Assessment, and community partners reviews the request and prepares the proposal for submission to the NCATE Steering Committee, which reviews the document before sending it to the Dean’s Administrative Council (DAC).  The DAC disseminates the proposal, with a request for feedback, to three groups: the Council for Teacher Education (CTE), faculty members, and the professional community (e.g., middle school teachers).  Written comments are forwarded to the Assessment Committee, which finalizes the document by revising the proposal or including an addendum, and resubmits it to the DAC for final approval. At each point, reviewers decide if the proposal should move forward or if clarification or additional information is needed. 

The Unit assessment system (Exhibit 2a.1 and Exhibit 2a.2) is an ongoing and recursive process for improving the Unit programs (Exhibit 2a.3).  Any proposed change must progress through the Unit assessment system for approval (Exhibit 2a.3).  To further refine the assessment process, revisions can be initiated at any step in the process.  The assessment committee meets regularly to review data reports and suggestions from other committees.  At its meetings, refinements of the assessment system are made by the committee, as noted in the minutes of meetings (Exhibit 2c.2, Exhibit 2c.3, Exhibit 2c.4, Exhibit 2c.5, Exhibit 2c.6, Exhibit 2c.7, Exhibit 2c.8).  The results of the review and any changes resulting are disseminated according to system procedures. 

Individual faculty, school partners, and program committees (e.g., literacy committee) may propose a change, such as identifying a need to improve an assessment instrument or procedures and informing the appropriate committee of their recommendations.  Rubrics (e.g., classroom management assessment) approved through the assessment system provide data on candidate performance.  When the need for a change is identified, a recommendation is submitted to the Director of Assessment and Program Evaluation (hereafter the Director of Assessment) who forwards it to the appropriate committee for review and evaluation.  While some artifacts and rubrics are required for PASS-PORT, faculty can also create their own course assessments in PASS-PORT.  Validity and reliability of all Unit assessment tools are continuously upgraded.  The Unit assessment system provides regular and comprehensive data on candidate performance, Unit operations, and program quality.  The Director of Assessment and the assessment committee communicate with the Director of the Office of Institutional Research and Assessment (IRA) to manage activities and initiate changes (i.e., adding item to a survey instrument) for Student Opinions of Teaching (SOTs) and the Exit and Employer surveys.  All other assessments are administered through the process developed by the committee and managed by the Director of Assessment. 

Collection of information on candidate proficiencies.  The Unit assessment system includes criteria to assess each candidate’s strengths and areas for improvement. Considered collectively, results of the assessment criteria indicate program strengths and areas for improvement.  The assessment system provides the means for the collection of data on initial and advanced candidates as the progress through a series of portals (Table 5).  The Unit ensures that the assessment system collects information on candidate proficiencies by aligning program curricula, course requirements, and assessments with the Conceptual Framework (CF), state (LDE) standards, and professional (SPA) standards.  Data are collected on candidates’ knowledge of the CF, including technology and diversity; national standards (INTASC, NBPTS), state standards (LCET), program standards, and Candidate Dispositions.  Candidate knowledge, skills, and dispositions are assessed in each course and across program levels.  Candidates enrolled initial certification programs and advanced programs in all departments move through a series of portals to track their progress and determine whether or not requirements at each portal are met.  Objectives for all professional courses are aligned with the CF., a description of which is included on all syllabi, and notations for the CF elements accompany each objective are included in all syllabi.  Also, artifacts and assessments included in candidate portfolios at three distinct levels must reflect relevant standards.   

Administration and monitoring of key assessments.  In addition to Unit assessments, each initial and advanced program includes a unique set of program and course assessments.  Candidates enrolled in initial certification programs demonstrate progress through Portals 1-5, and candidates in advanced programs demonstrate progress through Portals 6-10 (Major transition points for candidate performance, Table 5: Exhibit 2a.2).  In initial and advanced programs, candidates progress through portals (Exhibit 2a.3) by meeting a set of criteria, including Unit assessments that are standardized across levels, programs, and areas of concentration.  Variability is introduced at the program level through unique portfolio requirements for advanced programs.  PASS-PORT provides the means for the collection, aggregation and disaggregation, and analysis of both individual and group data.  These data provide information to the Unit head and program coordinators for forming, redesigning, or terminating programs.  

Most initial programs are housed in T&L.  The secondary programs and PK-12 programs in kinesiology, music, and art are housed in partner colleges identified in the overview. Areas of concentration within programs are Early Childhood (PK-3), Elementary (Grades 1-5), Middle School (Grades 4-8), Secondary Education & K-12, Special Education, and Master of Arts in Teaching (MAT).  Table 5: Exhibit 2a.2 provides a detailed visual of transition points for initial and advanced programs. 

Fairness, accuracy, consistency, and unbiased assessment procedures.  The issues of fairness, accuracy, consistency, and bias are addressed by examining validity, reliability, scoring, and bias analysis of the assessment instruments.  

Validity and Scoring Validity were addressed in the development of the assessment system, as well as the development of criteria for assessing artifacts for each program. The framework for the assessment system (Table 5) was revised in 2003-2004 by the assessment committee to reflect an operationalization of the CF; that is, the artifacts identified for each portal represent ways by which candidates can demonstrate proficiency in the CF elements.  Within this framework, the Unit and its programs include artifacts unique to the academic areas.  For example, the capstone artifact for the advanced programs in the T&L and ELT is an action-research project, with content and procedures unique to each program.  Using this framework ensures that all artifacts, regardless of the program for which they were developed, reflect the CF. 

Candidates’ artifacts are scored using a rubric developed by program committees comprised of faculty specifically responsible for the material being assessed (e.g., faculty teaching a specific class from which an artifact is submitted) or faculty with expertise in a specific area in collaboration with partners.  The criteria for each rubric are consistent with requirements of the course(s) leading to submission of the artifact, the professional standards related to the CF and the artifact (e.g., LCET or ELCC).  A common scoring scale is used for all artifacts.  Four points define the scale: unacceptable, approaching expectations, meets expectations, and exceeds expectations.  Specific descriptions for each criterion at each level of the scoring scale are provided (e.g., management plan and lesson plan rubrics) in the PASS-PORT handbook, which is accessible online.  Each faculty member assessing an artifact has attended PASS-PORT workshops and is familiar with the criteria, scoring scale, and descriptions of work for that artifact.  

Performance standards were established to reflect acceptable levels of proficiency at each portal and are based on the experiences of candidates working within that portal/transition point.  Different performance standards exist for the same artifact at different portals; that is, the acceptable performance standard for a candidate’s lesson plan submitted at Portal 3 is “Approaching Expectations,” but the acceptable performance standard at Portal 4 is “Meets Expectations,” because the standard is higher at Portal 4 than Portal 3.  Candidates must meet the performance standards for any portal before proceeding further.  Artifacts and the scoring criteria for them are reviewed by individual faculty each semester.  Suggestions for changes can be submitted to program chairs for discussion at monthly department meetings before being submitted to the assessment committee in the formal Unit assessment process.  

Each required artifact must meet expectations on the respective rubric before a candidate can be allowed to move to the next portal.  Each assessment element (i.e., the assessment of a candidate’s portfolio at a specific portal) is scored on the same four point scale as individual artifacts. The overall passing score is based on five criteria: 1) the presence or absence of the required artifacts demonstrate an acceptable level of performance, 2) an acceptable reflection relating the artifact to the appropriate professional standards and the CF, and 3) an understanding of program standards.  The descriptions of the assessment system and artifacts provide evidence of content validity, which ensures that decisions or inferences based on the results of any assessment are appropriate, meaningful, and useful.

Reliability for scoring artifacts is addressed through a systematic analysis of the inter-rater reliability of the rubrics associated with artifacts.  This analysis of data examining the Lesson Plan and Classroom Management Plan, both of which are artifacts at Portal 4 of the Initial Teacher Education Program, began in Fall 2007, and is currently in progress.  An analysis for bias is being carried out in conjunction with the test of reliability by examining the results for the Lesson Plan and Classroom Management Plan across gender and race. 

Assessments used for operations and programs management and improvement.  All assessments are used to measure, manage, and improve the operations and programs at four levels: 1) the Unit, 2) the program, 3) the course, and 4) the candidate.  Results of a few assessments and evaluations provide indicators of the Unit’s overall effectiveness which are fair, accurate, and consistent.  The IRA conducts an exit survey of graduating seniors that includes general items, as well as a set of items specific to each program.  The Office of Student Teaching also surveys candidates at the end of their student teaching semester or internship year (Exhibit 2a.5: Basic Program Follow-up).  Because candidates are also surveyed at the completion of their program, both surveys provide evidence of program elements that have been effective and those for which improvements are indicated.  These assessments are completed through electronic systems–the IRA survey through PeopleSoft and the student teaching survey through PASS-PORT–that allow the results to be aggregated and otherwise used to provide longitudinal reports that track candidate progress.  Results of IRA's Current Student Survey administered to a random sample can also be used each semester to measure program improvement and institutional effectiveness.  The Professional Attributes and Characteristics Scale, a PASS-PORT assessment, is administered at multiple portfolio portals, thereby providing individual, user group, and program data to track candidate progress throughout the initial or advanced programs.  (Samples are located in the Exhibit Room.) 

The IRA collects and aggregates data including SOTs, recruitment, retention rates, and exit surveys.  A survey of PK-12 consumers’ (e.g., parents, teachers, students, school/district administrators) perceptions of graduates’ skills is administered to population samples.  The Unit collects data on programs, reflections, and follow-up surveys.  The IRA and Unit collaborate to develop and improve instruments to appraise the value of candidates’ activities.

2b. Data Collection, Analysis, and Evaluation

Timeline for collecting key assessment data.  The IRA collects data each semester, but data are aggregated annually for reports.  Through PASS-PORT, data on candidate performance are collected and aggregated at the end of each semester.  Data from any prior semester are available to be aggregated or disaggregated whenever needed to generate reports for review.  PASS-PORT artifacts submitted by candidates are assessed by faculty each semester.  When artifacts for each portal are completed and uploaded, candidates’ portfolios are assessed at the end of each semester by the faculty advisor to which they are assigned.  Both formative (e.g., artifact) and summative (e.g., portal) assessments are included.  Representative samples of both types of assessments for ELT and T&L are provided in (Exhibit 2b.5, Exhibit 2b.6, Exhibit 2b.7, Exhibit 2b.8, Exhibit 2b.9, Exhibit 2b.10, Exhibit 2b.11, Exhibit 2b.12, Exhibit 2b.13, Exhibit 2b.14, Exhibit 2b.15, and Exhibit 2b.16).  Questionnaires are sent at the beginning and/or end of each semester determined by the type, pre- or post. 

Process and timeline used to collect, summarize, and analyze data.  The formal process for collecting, summarizing, and analyzing data for operations and program improvement is detailed in the Use of Data for Program Improvement flowchart (Exhibit 2b.1) and in Figure 1: Timeline of Program Improvements Based on Assessments in (Exhibit 2a.1), the COEHD Assessment System Description.  Minimally, data are collected and summarized once each semester.  However, these procedures and analyses can be undertaken at any time.  Whenever a committee (e.g., differentiated instruction committee) identifies a problem based on a review of data in an assessment report, it develops a proposal describing a change or refinement to be made.  At this point, the proposal is submitted to assessment committee to initiate the process detailed in Exhibit 2a.4, the COEHD Assessment System ChartIn addition to this formal system, informal procedures for assessment are ongoing.  When a faculty member or committee identifies a question or area of concern, discussions ensue on an informal basis until the point when a proposal is introduced into the formal system. 

Procedures for collecting data.  The parties responsible for collecting data include Dr. Michelle Hall, Office of Institutional Research and Assessment (IRA) Director, who is responsible for administering institutional assessments; Flo Winstead, Director of Assessment; and the Major Field Assessment (MFA) coordinators at the program level.  These participants collect and aggregate data through PASS-PORT and Peoplesoft for reports.  

Data are summarized and analyzed in various formats.  Through PASS-PORT, data for one candidate, a user group, a program, or the Unit can be summarized in a report or chart.  The IRA reports data using PeopleSoft.  The Unit completes the MFA that is submitted to the IRA annually.    Exhibit 2b.18 includes the webpage listing the Unit MFA reports and a sample of the M.Ed. Special Education January 2008 Results and Use Report. 

Data are summarized and analyzed annually and biennially by the IRA.  Additionally, data derived from PASS-PORT assessments are available each semester.  Data can also be aggregated for reports whenever a request is submitted to the Director of Assessment.  

Information technologies used to maintain the Unit assessment system are PeopleSoft and PASS-PORT.  IRA collects and aggregates data used to document progress and achievements and detect Unit and program areas for improvement.  IRA directs the collection of SOTs, faculty and student surveys, and demographic data.  All data are collected electronically and available to administrators and faculty for review.  IRA surveys and reports are available at http://www.selu.edu/admin/ir/index.html.  The Unit employs PASS-PORT to collect and aggregate data on candidate performance and faculty qualifications. Beginning in Fall 2004, candidates in initial programs and beginning in Fall 06, candidates in advanced programs have used PASS-PORT as the vehicle for portfolios.  

Records for formal candidate complaints and resolutions.  The University’s student handbook outlines the process by which candidates can lodge a formal complaint.  Initially, a formal complaint by a candidate is communicated in writing to the faculty member.  If there is no resolution, the candidate submits the written complaint to the faculty member’s department head.  When the department head and candidate are not able to achieve closure, the COEHD or partner college Dean is presented with the candidate’s written complaint and support documentation.  Procedures for specific complaints (i.e., sexual, racial, and gender harassment; disability discrimination; and academic issues) are also outlined in the Student Handbook.  Charts showing the format for the types and the number of complaints are found in Exhibit 2b.2 and Exhibit 2b.3.

At the conclusion of each academic year, complaints and resolutions are collected and compiled by the Director of Assessment.  The Director sends a chart noting the number and types of complaints, with a summary statement on the predominant complaints, to the COEHD Dean, who then forwards the information to the DAC for discussion.  Actions are taken as deemed appropriate and necessary.  In addition to the formal complaints and resolutions procedure, candidates can voice concerns or lodge complaints anonymously on the SOTs each semester.  Data are collected by the IRA, and a report on each faculty member is forwarded to the faculty member and the respective department heads.  The faculty member and department head review the documents and use the information for each faculty member’s end-of-year performance assessment.  Records of complaints and outcomes are kept on file in the Dean’s office. 

2c. Use of Data for Program Improvement
In regard to candidate performance on the main campus, at off-campus sites, and in distance learning courses, assessment data indicate no significant differences across programs, courses, and sections.  The IRA and the Director of Assessment can provide disaggregated data for candidate performance in courses offered on the main campus, at off-campus sites, and in distance learning courses.  One or more sections of some courses are offered off-campus or 100 % online (distance learning), but no undergraduate initial certification or advanced programs offer all courses online or off-campus.  The Master of Arts in Teaching (MAT) initial certification program, which has been redesigned, began accepting students in Summer 2007.  No courses will be 100 % online.  In the MAT program that is being phased out, candidates could complete some or all of their courses online if they met certain criteria (e.g., employed as a full-time teacher in the certification area).  The ESL concentration provides all ESL courses online.  

Policies and procedures for data collection have been established (Exhibit 2c.1), and candidates and faculty use data to improve their performance.  Candidates’ work in courses and FXs is assessed throughout their programs.  To progress through their program, candidates must satisfy criteria for one portal of the assessment system before moving to the next.  Additionally, artifact and portfolio assessments provide candidates with feedback designed to help them become more effective professionals.  Assessments created or adapted by program faculty are used to identify needed improvements.  The Prospective Education Candidate surveys, completed by initial (PEC) and advanced (Advanced PEC) candidates, reflect the CF.  Candidates’ performance on PRAXIS tests is also reviewed to identify areas for improvement.  When a deficit is identified in a candidate’s performance, individual candidates can avail themselves of services through the Teacher Development Program to help them improve their performance. 

Data are also used to help faculty improve their performance.  The IRA collects and aggregates SOT data, which are used to facilitate the improvement of faculty performance.  Faculty also develop an annual professional development plan to guide their work and identify achievements for the end-of-year assessment for review with the head of their respective departments.  Data are used initiate program and Unit changes on a regular basis. 

Multiple data sources are used to improve Unit programs.  Based on results of evaluations, program changes are initiated.  By reviewing IRA and Unit reports, committees and individual administrators and faculty identify effective and ineffective aspects of the Unit and its programs from formative and summative evaluation reports. Therefore, from the outset, decisions to maintain or revise programs or procedures are data-driven.  Suggested changes are formalized and linked to the Unit’s CF, SPA, and state standards, and follow procedures outlined in the Unit assessment system.  The system includes assessment of specific tasks and requirements.  Data collection and analysis provide the information needed to make sound decisions about curriculum, instruction, and candidate performance.  There is evidence that candidates entering the program will be prepared in terms of the knowledge, skills, and dispositions necessary to perform effectively in their profession and ultimately, to positively impact student learning. 

Changes based on data derived from assessments have occurred over the past three years and are initiated as indicated by Unit assessment system evaluations.  Changes are then systematically linked to the CF and studied to measure effects on teacher education and other professional programs.  Also, since the last NCATE review, the initial certification, alternative certification, and advanced programs were redesigned to meet state guidelines and to reflect needs identified by the Unit.  Data derived from PASS-PORT and the IRA were reviewed to identify areas for improvement in programs and courses. Specific examples of data-driven change are contained in SPA reports and in Exhibit 2c.4: Data-driven changes over the past three years. 

For annual reviews of faculty performance, the department head and faculty member ascertain areas for improvement.  Faculty then develop ways to improve their performance in the area identified.  One change has been the addition of more courses offered online. Following the redesign of the undergraduate, initial certification program, for example, sections of EDUC 201 and 211 have been offered online to meet candidate needs. Modifications in both initial and advanced programs focus on diversity.  On student teaching exit surveys, candidates stated that they wanted to be better prepared to meet the needs of diverse populations in inclusive settings. Several examples of data-driven changes regarding diversity are contained in Exhibit 2c.9.  

Assessment data are shared with candidates, faculty, and other stakeholders in various ways.  Reports are shared with faculty at regular COEHD meetings (e.g., Dean’s Advisory Council), departmental monthly meetings, and committee meetings.  Each semester, data is disseminated when the Unit convenes to recognize candidate and faculty achievements and to hear updates pertaining to the Unit and programs.  At the beginning of each academic year, administrators, faculty, and staff meet at a convocation, at which time the University president gives the state of the University presentation and faculty and staff are recognized for special achievements (Exhibit 2c.12).  Candidates are apprised of assessment data at meetings of student education organizations, and candidate representatives may serve on Unit committees.  The public is notified of information on education in the local newspaper and in the University radio and television stations. 

The IRA Director works with administrators and faculty to create assessment tools used to conduct surveys; collect, aggregate, and analyze data; and generate reports, thereby providing scientifically measured outcomes on these valid and reliable instruments.  The data are used to create reports that are disseminated to stakeholders and posted on the University website to other interested parties.  If other data are needed, individuals can submit a request to the IRA Director.  Another noteworthy accomplishment of the Unit is a research project on the inter-rater reliability of the rubrics associated with assessment of PASS-PORT artifacts.  This analysis of data examining the Lesson Plan and Classroom Management Plan, both of which are artifacts at Portal 4 of the Initial Teacher Education Program, began in fall 2007. 

The Unit is particularly attentive to continuous improvement resulting from the review and analysis of data.  Faculty and administrators regularly critique assessment instruments and review assessment results to identify program elements that are effective and those for which improvements are needed, primarily to better meet the needs of candidates.  Faculty are required to participate in monthly COEHD faculty or program meetings, but groups of faculty choose to form formal or informal ad hoc committees to discuss a specific topic, work on a project, or plan actions to address a problem.  Data are reported at regular meetings, and most groups in which faculty choose to participate originate because of interest in assessment findings.



 

  

 


Southeastern Louisiana University
© 2007Southeastern Louisiana University
All Rights Reserved
Unofficial and external sites are not endorsed by Southeastern Louisiana University