Science Program Evaluation and a System of Science Assessment

Assuming your district/school has established a vision for science education and large-scale, specific goals aligned to that vision, you will next need to determine a system of assessments for evaluating progress toward those goals. As mentioned in my last post, many districts are working to adopt and implement new science standards. Strategically assessing science-related outcomes at multiple levels will provide ongoing evidence of effective change – after all, why make changes if you don’t know whether they actually make any difference?

While it might be obvious, an evaluation of a science program based on these goals will take more than one assessment! In other words, the annual state standardized test, often the only systematic science test used by a school, will not measure the full range of outcomes related to a meaningful vision for science education. That requires leaders to strategically implement a system of assessments. The Wisconsin DPI has a chart that illustrates some components of such a system, including formative, interim, and summative elements.

The majority of assessment will happen formatively at the classroom level. This level is where teachers see the day-to-day use of scientific practices by their students as they investigate, communicate, and ask questions about science. It will be critical for teachers to have the structures to discuss what they’re observing from their students, collaboratively determining next steps. Processes of informal formative assessment should drive instructional practice. If schools are moving toward the NGSS or NRC Science Framework, formative, as well as all levels of assessment, should be three-dimensional.

 Common, interim assessments and rubrics across classrooms and grade-levels can support collaborative understanding of students’ abilities. These types of assessments can provide a more formal view into student growth in relation to science content knowledge and practice. Quality performance tasks can potentially provide the clearest information for collaborative groups of teachers to reflect on progress toward their goals. They need to be implemented well, however, in order to be useful. Teachers must have the time to score papers together and come to an agreement on how particular examples of student work meet the rubric criteria.

Large-scale district summative tests (or state level tests) often afford the least amount of data for specific instructional guidance. They might, however, suggest areas for professional development or foci for revised student project rubrics. For example, a set of district end-of-course exams might all show that students across the district struggle with using data effectively. Often these types of tests are multiple-choice, which provide limited information in relation to authentic science practice, but they can be effectively paired with open-ended opportunities for students to describe their reasoning.

An often forgotten element in such an assessment system is an evaluation of student attitudes about science and their general scientific literacy. Do they see how science relates to their lives? Can they make sense of scientific evidence within popular media? Is science meaningful for them?

In summary, schools and districts reviewing and attempting to improve their science programs will have unclear success in that process if they haven’t defined what outcomes they want and how to measure them. A meaningful and strategic system of science assessment will be an essential part of this process. 

The next series of blog posts will discuss formative, interim, and summative assessments in more depth, as well as effective surveys of student attitudes. Each will provide examples of these assessment types and suggestions for classroom or school use. 

Science Program Objectives

After a focus on disciplinary literacy in the last couple of posts, I’m now returning to the theme of science program review/revision…

I’ve spoken to several districts in the last few months that have established a vision for science education. That excites me a lot! Talking to them further, I often ask how they’re going to measure whether or not they’re achieving that vision. They share their 3-5 year plan with me for revising their science program, and I ask, “How will you know in 3 to 5 years whether you’ve made progress in accomplishing your vision?” Many leaders have no answer to that. Supporting administrators and educators in establishing that evaluation plan is, therefore, the purpose of this post and others to come.

In order to create that evaluation plan, the school/district science leadership team will first need to translate their vision statement into specific and measurable objectives. These are the big picture goals of science for the students. They’re more concrete than the vision but less specific than the more content-related learning objectives that would be part of a standards-based report card (I’ll describe those objectives in a later post). Ideally, these goals will be written out as SMART goals, meaning they are:

• Specific: Clearly states what will be done. 
• Measurable: Links to a particular outcome using a specific test, noting a particular target. 
• Achievable: Want it to be a stretch, but realistic. 
• Results-focused: Should measure student outcomes, not program implementation. 
• Time-bound: Have a due date. 

To craft a couple examples of SMART goals, let’s take a few phrases from the initial vision I shared in this blog from the NRC Framework for K-12 Science:

“[By] the end of 12th grade, all students have some appreciation of the beauty and wonder of science; possess sufficient knowledge of science and engineering to engage in public discussions on related issues …” 

Starting with the first phrase, I’ll turn it into a SMART goal. Let’s say I’m working with a group of middle school teachers.

By the end of their 8th grade year, all of our students will express an appreciation of the importance of science in their lives and a sense of wonder in relation to science, as measured by answering “somewhat agree” or higher on the relevant questions of the Science Attitudes Survey.

The goal is specific. There is a particular outcome wanted for all students. The goal is measurable. The school will be using specific questions on the Science Attitudes Survey, with a ranking of at least “somewhat agree” on those questions (note: I’m not referencing a specific survey here, though there are several available). The goal might be achievable. Science programs and goals should be for ALL students, but will all students really agree with statements about the sense of wonder inherent in science? That’s less certain. After the first year of data collection, having an established baseline will allow for more realistic goals. The key will be continuing to have high expectations for all and pushing on what might be considered “realistic.” The goal is results-focused. It’s not just that teachers will have more engaging activities. It’s focused on an outcome, student engagement in science, where they’re seeing its meaning related to themselves. The goal is time-bound. It’s by the end of 8th grade. As a middle school team, goals could be annual, semi-annual, or by unit, but if it’s collaborative work as a department, having a goal for the end of their three years with you would also make sense.

Here’s another example of a SMART goal, linked to the second phrase in the vision statement:

By the end of the year, all of our students will increase performance task scores by at least one point in each category of the claims, evidence, and reasoning rubric. Three times each year we will use this rubric with performance tasks to measure their ability to communicate claims supported by evidence, with clear scientific reasoning.

A lot of important goals will not be measurable on a standardized test! Staff could create a series of performance tasks requiring students to make a substantiated claim for a particular action their community should take in relation to a particular phenomenon that they studied (pollution, erosion, habitat destruction, etc.).

The goal is specific. There is a clear outcome noted for all students. The goal is measurable. The school will be using common performance tasks, and students’ growth on those tasks, based on a rubric is spelled out. Again, it’s unclear whether the goal is achievable, but we want all students to learn through the year, and moving up one rubric category might not be rigorous enough (notably, such a goal might not be relevant to some students already scoring at the top). The goal is results-focused. It’s focused on an outcome, student performance on specific tasks requiring communicating and defending scientific ideas. The goal is time-bound. Each teacher would expect to see progress by the end of the year.

Of course, the science department will need to come together regularly to look at data in relation to these big-picture goals. Conversations should likely be happening at least weekly in relation to student work and how particular formative or interim assessment data could inform instruction. Those weekly conversations would focus on more particular goals, likely those in standards-based grading. But, at least a few times per year (beginning/middle/end), teachers should be coming together to talk about progress in relation to these big-picture goals. Thompson, et. al., describe a process for this type of collaborative work. Selecting one of these goals per year can provide a focus for teacher collaboration and professional development. A science program does not have to be reviewed in relation to every one of the goals every year.

The next blog post will provide more specific guidance on the evaluation of science program in relation to these goals through a “system” of science assessment.

Conducting a Science Program Audit

With a mission and vision in place for the science program, teachers will need to personally and collaboratively decide whether or not they have a sufficient understanding of what instructional practice looks like under that vision. In future blog posts, I’ll be providing suggestions for professional development on research-based science instruction. For this post, I’ll be assuming (always a risky plan!) that teachers have the skills and understanding necessary to evaluate their work in relation to their vision.

Therefore, to determine progress towards a vision, a school or district will first need to determine where they’re at now. A science program “audit” is a strategy to do that. It asks, “How well does our instruction align with this mission? How well are we accomplishing our vision? How do we know?”

Forming a Committee

Again having an audit committee will be a valuable guide for this process. While there could be some overlap with the leadership committee that crafted the mission and vision statements, the auditors should consist of outsiders, not district employees. A school will need constructive, impartial outsiders to give an unbiased perspective. Some suggestions for members of this committee:

1. A recent graduate of the district currently studying science in college
2. Community members working in science-related fields (healthcare, high tech, university research, etc.)
3. Science education professors or coordinators (college, university, teacher professional organizations)
4. State education department or regional education service agency science education leaders
5. Parents
6. Educators from neighboring school districts (would be great to have districts across a region support each other in this audit process)

The size of the committee needed depends on the size of the district. The committee would ideally be able to visit every school in the district and interview a representative sample of teachers and students across the district that includes some from every school. One strategy would be to have two auditors work together, visiting one school in the morning and one in the afternoon. Thus, in a district with 40 schools, 20 auditors would then be needed to complete all of the visits and interviews in one day. Of course, fewer auditors would be needed if the process was spread over several days. More in-depth audits could be conducted by one or two auditors over the course of several months, though an audit can be a unique opportunity to engage a broad range of community members.

Data Gathering

Before the committee comes to the school, administrators and teachers should collect and share useful data and information about the school/district science program. This data should be broken down by all appropriate subgroups and could include:

• Assessment data: standardized test scores, district or grade level common assessments, classroom level assessment examples;
• Descriptions of courses taught and typical student pathways: for example, it will be important to note if students from particular backgrounds tend to be in “honors” classes to a lesser extent;
• Teachers’ license and longevity information;
• Postsecondary pursuits: career or college/university pathways;
• Any documentation of science program mission/vision and links to science department or teacher websites.

Information gathered before the auditing day(s) should also include broad-scale surveys to gather quantitative data on the science program. These could easily be created and conducted through Google forms or another free survey tool. These surveys could alternatively be developed, conducted, and analyzed by the auditor(s).

Student Surveys Example Questions

• Elementary: at lower elementary the teacher would need to read and explain the questions. A simple yes/no might be most appropriate K-2. 

1. I enjoy science class. [A lot, kind of, not really]
2. We do interesting experiments and investigations in science [A lot, sometimes, not much] 
3. We learn about scientists in class that look like me [yes, no]
4. We study science outside [A lot, sometimes, not much]
5. I get to study my own questions in science class [yes, no]
6. I learn about things in science class that I wonder about in my life.

• Secondary:

1. I enjoy science class. [A lot, sometimes, not really]
2. We do interesting experiments and investigations in science [A lot, sometimes, not much]
3. We learn about scientists in class that look like me [yes, no]
4. We study science outside [A lot, sometimes, not much]
5. I get to study my own questions in science class [A lot, sometimes, not much]
6. I learn about things in science class that I wonder about in my life or the world around me [A lot, sometimes, not much]

Parent/Community Survey Example Questions

1. I am satisfied with the science education that my child (or our community’s children) is receiving [yes, somewhat yes, somewhat no, no]   Explain: [open-ended]
2. Students are involved in science learning relevant to needs and issues in our community [A lot, sometimes, not enough]
3. Students are learning relevant 21st century skills through science classes, such as analyzing scientific studies and evidence, communicating technical information, conducting investigations, and collaborating with peers [A lot, somewhat, not adequately]
4. Students are adequately prepared for postsecondary careers or educational pathways [A lot, somewhat, not adequately] 
5. Community members and experts are invited into science classrooms and to support science projects [A lot, sometimes, not enough]

Teacher Survey Example Questions

• Select how much you agree with the sentence.

1. I am comfortable teaching science. [Yes; somewhat yes; somewhat no; no]
2. I have sufficient collaboration time with other teachers focused on science. [Yes; somewhat yes; somewhat no; no]
3. I receive sufficient science-related professional development from my district/school. [Yes; somewhat yes; somewhat no; no]
4. Our current curricular materials effectively support science instruction. [Yes; somewhat yes; somewhat no; no]
5. I have adequate lab and related science equipment for effectively teaching science. [Yes; somewhat yes; somewhat no; no]

• The next set of questions asks about frequency of various actions.

1. I take advantage of outside professional development opportunities (not school or district sponsored). [At least monthly, five to ten times per year, two to four times per year, about once per year, once every few years, never]
2. Students have opportunities to investigate their own questions in my science class [At least monthly, five to ten times per year, two to four times per year, about once per year, never].
3. Our class goes outside or to other relevant sites in the community (or beyond) for science learning and investigation [At least monthly, five to ten times per year, two to four times per year, about once per year, never].

To better understand the “why” for the survey answers, auditors should also conduct interviews and focus groups of students and teachers based on these questions. This back-and-forth will allow for follow up questions and clarification. Focus groups will allow for greater synergy of respondents and will typically be more appropriate for work with students, where one-on-one interviews are not advisable for student safety and comfort reasons.

Observations

As part of the day(s) that the audit committee is in schools, they should be observing classrooms. Ideally, this observation will happen naturally in the flow of instruction, not just be scheduled for the best lab of the year. The auditors need an authentic perspective on what is happening in the day-to-day work of the classroom. Notably, auditors without an education background will need further guidance on what they’ll be looking for in classrooms. In addition to general, objective observations of what happens in the class, some specific prompts/questions that auditors could be answering include:

1. Describe the classroom. How are desks arranged? What materials and space are available? 
2. What happens during the class? What is the duration of each segment of the class? [lecture, independent work, collaborative work, and whole-class discussion]
3. What types of questions is the teacher asking the students? What questions are students asking of each other and of the teacher? [Yes/no, clarification, one-right answer, logistical, deeper though required, etc.] 
4. What type of work are students doing? [step-by-step lab, worksheet w/ one right answer, open-ended problems, peer discussion, arguing with evidence, modeling or investigating a phenomena, 
5. How does the teacher establish whether or not the students understand the material? 
6. How does the teacher ensure all students are equitably engaging in the instruction?

Bringing It All Together

At the beginning of the day (week, month, or year) the audit committee would come together with administrators and science teacher leaders to establish their roles and the plan of attack. They would receive guidance on what’s expected of them and a schedule, maps, and other needed support in accomplishing those tasks. Ideally, they would have tasty snacks (I prefer cake donuts and seasonal fruit, in case you’re wondering) and lunch available during the day. At the end of the allotted timeframe, the auditors will come together to discuss the data and observations as a group and collaboratively review the day. They should have feedback templates where they can create and share consensus comments of the group and their individual feedback on the day, though individual feedback could come back a few days/weeks later after auditors have had a chance to fully analyze the data provided from the school, their observations, and the observations of their fellow auditors. Templates would include the specific questions that the school/district wants feedback on from the auditors, as well as space for open reflections on the wide-ranging pieces of data reviewed.

After conferring as a group, the auditors should present their initial findings to teachers and administrators, with the opportunity for auditors and district educators to ask clarifying questions of each other. This discussion will need an able facilitator, as it is a time for teachers to reflect on the observations shared by the auditors, not get defensive about perceived negative feedback.

Once final feedback is received by individual auditors, either a primary auditor or a school/ district leader should combine the data and feedback into a final report. Several science department and leadership committee meetings will need to be devoted to reviewing the feedback and determining the next steps in moving toward the vision and mission for science education. 

*Special thanks to Judy Singletary of Stoughton School District for ideas on this audit process.