Formative Assessment and the NGSS

Within this post, I am going to focus on formative assessment as an ongoing assessment conducted as part of daily instruction to guide further instruction. They are informal or formal checks of knowledge happening in conjunction with instruction. I’m not considering more summative type assessments such as end-of-unit tests or student projects. Though to be sure, all assessments should be formative, in that the results will be used to guide decision-making in relation to instruction.

Therefore, formative assessments probably won’t be the primary tools that teachers and administrators will look at when determining whether their school as a whole is making progress toward their vision for science education. Instead, they are tools that will influence daily decisions in the classroom, as well as student and/or teacher collaborative conversations. Do I need to provide more time for peer discussion around crafting a procedure for their investigation? Should I include more scaffolding for students to create an effective data table? Notably, they might inform elements reported on a standards-based report card, but more often they will not.

In a classroom using the NRC Framework and/or the Next Generation Science Standards, educators aim to use three-dimensional instruction, including within formative assessment. Two examples of what that could look like in practice might help:

A fifth grade teacher shows students a large syringe full of air. He asks them to discuss with a neighbor what will happen if he plugs the end and pushes down. Students then get to try it out at their table. After a couple minutes of students investigating the phenomenon, the teacher asks them to model the phenomena by drawing a diagram of it, prompting them to consider how to show things they can and cannot see. Why does it get harder to push down? Students create the model on their own first, then discuss their model with a peer, making revisions to their models as desired and considering evidence. The teacher walks around asking probing questions such as, “What are those little circles in your syringe?” “Are they really that big?” “How do they look different before and after pushing down on the syringe?” As he walks, he’s jotting down student names and occasional notes along the continuum of the rubric, which he has on a tablet or clipboard. Students then discuss their models in relation to the portion of a modeling rubric focused on clearly representing all important aspects of the phenomenon (not yet relationships among components). The teacher walks around, looking and listening for important components of models, evidence, and comments to share with the class to illustrate this aspect of modeling. He has a couple pairs of students share theirs, highlighting key criteria from the rubric where students appeared to be struggling. He also keeps his notes in a file with similar notes about students’ abilities with the science and engineering practices to look for progress over time and keep track of areas that need more work. 

Modeling - Subskill	1	2	3	4
Identifying important components of a scientific model	Student represents the object or occurrence	Student represents the object (etc.) with details (evidence) related to the phenomenon	Students models the phenomenon (etc.) in such a way that it adequately represents important components of it (and not extraneous elements) and evidence gathered	Student models the phenomenon, explains why those are the important components based on evidence, and can analyze why components noted in one model better represent the phenomenon than those in another model
In This Example	Student draws a syringe	Student draws a pushed down syringe with packed little circles inside of it and label of “gas”	Student draws one syringe pulled back and another one pushed down, each has a magnification “bubble” representing the scale of air particles and w/ them being closer together in the pushed down image	Student draws two syringes, as noted in 3, explains the importance of the scale and the particles being closer together, and notes why a model depicting air particles still far apart even with syringe down is more accurate (e.g. evidence – can’t see the air)

As a second example, a high school biology teacher asks students how many types of organisms there are in the world, leaving the question intentionally vague. Students discuss the answer in groups of three (no devices used at this point). The teacher pushes for justification and evidence for quantitative responses, as well as proper vocabulary. After a few minutes, she asks some student groups to share their answers and evidence. And, then she asks, “Why is this diversity of life important?” After sharing ideas with a partner, the whole class discusses it for a few minutes. The teacher then describes an investigation the class will do of “biodiversity” within their school grounds (as part of a larger unit on ecosystems, invasive species, and adaptations). The class goes on a quiet, mindful walk outside where students observe and come up with a testable question(s) about biodiversity on their school grounds and/or in the local area. At the end of the walk, the teacher collects the cards with students’ names and ideas on them. She quickly sorts them into yellow—needs more work, or green—testable; she also jots down some notes as to where students are at in general with this skill and adds it to her assessment file. The next day she anonymously shares a few of her favorite greens and yellows to illustrate key concepts in relation to testable questions, eliciting student ideas first in that conversation.

These formative assessments must be part of a larger cycle of guiding and reflecting on student learning. I like the APEX^ST model described by Thompson, et al., in NSTA’s Nov 2009, Science Teacher:

1. Educator teams collaboratively define a vision of student learning. 
2. They teach and collect evidence of learning. 
3. They collaboratively analyze student work and other formative evidence of learning (such as conversations) to uncover trends and gaps. 
4. They reflect how opportunities to learn relate to evidence of student learning.
5. They make changes, ask new questions, conduct another investigation, etc.

And, then they reflect again on evidence of student learning in light of their vision, considering changes to their vision and objectives as necessary.

One goal here, even in formative assessment, is to be thinking about how student learning fits into the overall picture of three dimensional instruction. Within the gas particle modeling task above, the science/engineering practice (SEP) is modeling, the disciplinary core idea (DCI) is 5-PS1.A—“gases are made from particles too small to see,” and the crosscutting concepts (CCCs) are cause and effect and scale. Within the biodiversity question task, the SEP is asking questions, the DCI is HS-LS2.C—“ecosystem dynamics,” and the CCC is systems and system models (though others could apply).

While those 3D connections are being made overall, the specific, in-the-moment, formative assessment goals here do not capture all three dimensions. In other words, the full task and work throughout the unit will involve students in all dimensions, but this formative snippet is really about one element of one practice in each case. Can students represent important aspects of a phenomenon within a model? Can students generate a testable question? The formative data gathered and acted on could also focus on their understanding of biodiversity or the particle nature of matter (DCIs). Or, it could focus on their ability to reason about the scale of a phenomenon (CCC). But, in this case the teacher kept things simple and more manageable with a specific, narrow goal in mind, which clearly related back to a larger vision for student learning and objectives for this unit. Importantly, while it’s a narrow goal, it’s still a deeper, conceptual learning goal. It’s not just an exit card asking students to regurgitate a fact or plug numbers into a formula.

Examples of assessments (not necessarily exemplars, could be formative or summative)

• Next Generation Science Assessment – project w/ the Concord Consortium - 
• NGSS-based sample classroom tasks - from Achieve
• Behavior of Air – Krajcik -  (w/ several others in this chapter of the NRC report on assessment for the NGSS)
• Oregon Dept of Ed performance tasks 
• Performance Assessment Resource Bank
• NECAP released items 

Formative assessment resources:

• The Informal Formative Assessment Cycle – from STEM Teaching Tools
• Designing an Assessment System - from STEM Teaching Tools
• How Teachers Can Develop Formative Assessments that Fit a Three-Dimensional View of Science Learning 
• Examining Student Work (NSTA article) – by J. Thompson et al., 
• Research article - Flying Blind: An exploration of beginning science teachers’ enactment of formative assessment practices  - by Erin Marie Furtak
• Literature review - Assessment for Learning (Formative Assessment)

Other resources to add? Put them in the comments below!

Syringe image from http://www.sciencefriday.com/wp-content/uploads/2016/01/15-push-in-syringe.jpg

Using Surveys as Part of the Evaluation of School Science Programs

Surveys of students, teachers, and community members will provide critical information in the process of determining whether changes made to your science education program improve desired outcomes. Many important questions cannot be answered through typical science assessments. While it’s clearly essential that students understand and can do science, do they sincerely believe that someone like themselves could be a scientist? Further, are you changing not just knowledge of, but beliefs about, science? Do students see how science relates to their lives? Is it meaningful for them? Or, do they see a need to question “scientific evidence” within popular media?

A recent article in National Geographic noted that solid, research-based science often faces organized and angry opposition. We don’t want students leaving school doubting the consensus of the scientific community (unless they somehow have sufficient, valid evidence to doubt a claim). They can understand how vaccines work and still decide not to have their children vaccinated. It’s unfortunate that our society believes in science, but not its findings.

Furthermore, do students understand who scientists are and what they do? While it was created as a tool for K-5, the “Draw-a-Scientist” test (DAST) could be done at secondary levels as well. My 8^th graders certainly held onto stereotypes of scientists. We want students to see science as including a wide-range of tasks by a wide-range of people, particularly people who look like them and have interests similar to their own.

Here are a few sample surveys of student attitudes: 

• A wide range of examples are provided on this site from Pearson   
• Survey on student attitudes toward science and other STEM subjects (Faber, et al., 2013). This paper describes use and development of the survey, with the actual tool at the end. 
• Changes in Attitudes about the Relevance of Science (CARS) – with three versions of the instrument, it could be given at the beginning, middle of a school year, instead of being more spread out. 
• Test of Science Related Attitudes (TORSA) – see  final page for items and scoring information.

These surveys can provide teachers with data to evaluate their individual courses and the science program more generally.

While surveys of student outcomes are critical within a system of assessments, it’s also important to understand the views of parents/community members and teachers during the change process. Surveying parents and other community members can help ensure they’re aware of and meaningfully connecting to the school science vision and students’ science learning. Teacher surveys can ensure they’re comfortable teaching their content and the practices of science in accordance with the vision for science learning. Within results, you can look at trends by demographics, such as race and ethnicity, or differences between new and veteran teachers.

In a survey of parents and community members, you probably don’t want to get into the content being taught. Hearing about personal views of evolution and climate change isn’t necessary for these purposes. Questions could have a Likert-scale format, with selections from strongly agree to strongly disagree. Some examples include:

• Through the science courses, I believe my student is becoming a better scientific thinker (for the broader community that would be rephrased as “students are becoming”).  
• I am familiar with the district vision for science education. 
• I believe my student is receiving a quality foundation in his/her science classes to pursue science careers in the future. 
• I believe my student is being well-prepared for science classes at the college or university level.

There should also be an open-ended text box, asking survey takers to please share any comments or questions about the science education program at their school. Of course, even with community input, you’re not going to resort to poor instructional practice that isn’t research based, like lecture. Educators are the professionals in this setting. You may, however, decide to make more career linkages in your courses or bring in more guest scientists.

It’s also important to know where teachers are at in the change process. Are they getting the support they need in teaching science? Tools like the Survey of Enacted Curriculum (SEC) can also let educators and administrators know whether what they’re doing actually lines up with the intentions of the instructional program. It’s not a “gotcha” system, but an approach like Lesson Study that can lead to tremendous, collaborative professional learning.

A brief endnote… While it is true that for statistically-validated studies surveys need to undergo extensive testing, everyday school surveys can provide a useful piece of information for guiding instructional programs. Surveys linked above have largely undergone testing and include multiple item constructs, so using them or learning from them is a good step.

Some other tips for creating quality surveys include: 

• Use multiple questions to measure each idea or topic. Looking at several questions together provides a more valid picture of what people really think.  
• Have a student, parent, etc. verbally talk through their thoughts on the survey with you. They think aloud as they read and answer the questions. Are they understanding the questions in the way that was intended? Is there some confusing wording? Having people of different backgrounds do so helps ensure the questions are similarly interpreted by people. 
• A focus group, with a neutral facilitator (i.e., not your boss), can provide a different perspective and bring out ideas that a survey cannot. It can also inform survey development.  
• Pilot the survey before sending it out broadly. 
• Here are a few further tips for online surveys.

And, yes, it takes extra time and effort to know whether you’re actually making a long-term difference for your students and whether the large-scale changes improve classroom practice, but it’s worth it. 

Science Program Evaluation and a System of Science Assessment

Assuming your district/school has established a vision for science education and large-scale, specific goals aligned to that vision, you will next need to determine a system of assessments for evaluating progress toward those goals. As mentioned in my last post, many districts are working to adopt and implement new science standards. Strategically assessing science-related outcomes at multiple levels will provide ongoing evidence of effective change – after all, why make changes if you don’t know whether they actually make any difference?

While it might be obvious, an evaluation of a science program based on these goals will take more than one assessment! In other words, the annual state standardized test, often the only systematic science test used by a school, will not measure the full range of outcomes related to a meaningful vision for science education. That requires leaders to strategically implement a system of assessments. The Wisconsin DPI has a chart that illustrates some components of such a system, including formative, interim, and summative elements.

The majority of assessment will happen formatively at the classroom level. This level is where teachers see the day-to-day use of scientific practices by their students as they investigate, communicate, and ask questions about science. It will be critical for teachers to have the structures to discuss what they’re observing from their students, collaboratively determining next steps. Processes of informal formative assessment should drive instructional practice. If schools are moving toward the NGSS or NRC Science Framework, formative, as well as all levels of assessment, should be three-dimensional.

 Common, interim assessments and rubrics across classrooms and grade-levels can support collaborative understanding of students’ abilities. These types of assessments can provide a more formal view into student growth in relation to science content knowledge and practice. Quality performance tasks can potentially provide the clearest information for collaborative groups of teachers to reflect on progress toward their goals. They need to be implemented well, however, in order to be useful. Teachers must have the time to score papers together and come to an agreement on how particular examples of student work meet the rubric criteria.

Large-scale district summative tests (or state level tests) often afford the least amount of data for specific instructional guidance. They might, however, suggest areas for professional development or foci for revised student project rubrics. For example, a set of district end-of-course exams might all show that students across the district struggle with using data effectively. Often these types of tests are multiple-choice, which provide limited information in relation to authentic science practice, but they can be effectively paired with open-ended opportunities for students to describe their reasoning.

An often forgotten element in such an assessment system is an evaluation of student attitudes about science and their general scientific literacy. Do they see how science relates to their lives? Can they make sense of scientific evidence within popular media? Is science meaningful for them?

In summary, schools and districts reviewing and attempting to improve their science programs will have unclear success in that process if they haven’t defined what outcomes they want and how to measure them. A meaningful and strategic system of science assessment will be an essential part of this process. 

The next series of blog posts will discuss formative, interim, and summative assessments in more depth, as well as effective surveys of student attitudes. Each will provide examples of these assessment types and suggestions for classroom or school use. 

Science Program Objectives

After a focus on disciplinary literacy in the last couple of posts, I’m now returning to the theme of science program review/revision…

I’ve spoken to several districts in the last few months that have established a vision for science education. That excites me a lot! Talking to them further, I often ask how they’re going to measure whether or not they’re achieving that vision. They share their 3-5 year plan with me for revising their science program, and I ask, “How will you know in 3 to 5 years whether you’ve made progress in accomplishing your vision?” Many leaders have no answer to that. Supporting administrators and educators in establishing that evaluation plan is, therefore, the purpose of this post and others to come.

In order to create that evaluation plan, the school/district science leadership team will first need to translate their vision statement into specific and measurable objectives. These are the big picture goals of science for the students. They’re more concrete than the vision but less specific than the more content-related learning objectives that would be part of a standards-based report card (I’ll describe those objectives in a later post). Ideally, these goals will be written out as SMART goals, meaning they are:

• Specific: Clearly states what will be done. 
• Measurable: Links to a particular outcome using a specific test, noting a particular target. 
• Achievable: Want it to be a stretch, but realistic. 
• Results-focused: Should measure student outcomes, not program implementation. 
• Time-bound: Have a due date. 

To craft a couple examples of SMART goals, let’s take a few phrases from the initial vision I shared in this blog from the NRC Framework for K-12 Science:

“[By] the end of 12th grade, all students have some appreciation of the beauty and wonder of science; possess sufficient knowledge of science and engineering to engage in public discussions on related issues …” 

Starting with the first phrase, I’ll turn it into a SMART goal. Let’s say I’m working with a group of middle school teachers.

By the end of their 8th grade year, all of our students will express an appreciation of the importance of science in their lives and a sense of wonder in relation to science, as measured by answering “somewhat agree” or higher on the relevant questions of the Science Attitudes Survey.

The goal is specific. There is a particular outcome wanted for all students. The goal is measurable. The school will be using specific questions on the Science Attitudes Survey, with a ranking of at least “somewhat agree” on those questions (note: I’m not referencing a specific survey here, though there are several available). The goal might be achievable. Science programs and goals should be for ALL students, but will all students really agree with statements about the sense of wonder inherent in science? That’s less certain. After the first year of data collection, having an established baseline will allow for more realistic goals. The key will be continuing to have high expectations for all and pushing on what might be considered “realistic.” The goal is results-focused. It’s not just that teachers will have more engaging activities. It’s focused on an outcome, student engagement in science, where they’re seeing its meaning related to themselves. The goal is time-bound. It’s by the end of 8th grade. As a middle school team, goals could be annual, semi-annual, or by unit, but if it’s collaborative work as a department, having a goal for the end of their three years with you would also make sense.

Here’s another example of a SMART goal, linked to the second phrase in the vision statement:

By the end of the year, all of our students will increase performance task scores by at least one point in each category of the claims, evidence, and reasoning rubric. Three times each year we will use this rubric with performance tasks to measure their ability to communicate claims supported by evidence, with clear scientific reasoning.

A lot of important goals will not be measurable on a standardized test! Staff could create a series of performance tasks requiring students to make a substantiated claim for a particular action their community should take in relation to a particular phenomenon that they studied (pollution, erosion, habitat destruction, etc.).

The goal is specific. There is a clear outcome noted for all students. The goal is measurable. The school will be using common performance tasks, and students’ growth on those tasks, based on a rubric is spelled out. Again, it’s unclear whether the goal is achievable, but we want all students to learn through the year, and moving up one rubric category might not be rigorous enough (notably, such a goal might not be relevant to some students already scoring at the top). The goal is results-focused. It’s focused on an outcome, student performance on specific tasks requiring communicating and defending scientific ideas. The goal is time-bound. Each teacher would expect to see progress by the end of the year.

Of course, the science department will need to come together regularly to look at data in relation to these big-picture goals. Conversations should likely be happening at least weekly in relation to student work and how particular formative or interim assessment data could inform instruction. Those weekly conversations would focus on more particular goals, likely those in standards-based grading. But, at least a few times per year (beginning/middle/end), teachers should be coming together to talk about progress in relation to these big-picture goals. Thompson, et. al., describe a process for this type of collaborative work. Selecting one of these goals per year can provide a focus for teacher collaboration and professional development. A science program does not have to be reviewed in relation to every one of the goals every year.

The next blog post will provide more specific guidance on the evaluation of science program in relation to these goals through a “system” of science assessment.

Two Annotated Lessons to Show How Concepts of Literacy Might Continually Intersect in a Science Classroom

Building on my previous post (a must read before this one), I describe two lessons here, labeling the literacy components of each lesson. For reference, connections to literacy strategies (L), disciplinary literacy (DL), and scientific literacy (SL) are labeled within the descriptions of the lessons below. Because each reference to disciplinary literacy (DL) would also be a reference to scientific literacy, for clarity it’s only labeled as DL (not as SL too). 

Elementary

In a kindergarten classroom, a teacher puts a stuffed animal on a rolling chair in front of the room. The teacher asks, “How could we make ‘Stuffy’ move? Share an idea with a partner” [DL]. She then circulates to hear student talk [L]. She randomly asks a few students to come describe and demonstrate their method [DL]. As students share their method, she’ll be pointing out terms they use, particularly highlighting or prompting the terms push and pull. Next, she has students write in their science notebooks, “A force is a push or a pull” [L]. This writing may be scaffolded by having some students just trace these words on a worksheet glued into the notebook [L]. Above that writing, she asks students to draw a picture of their idea, or another pair’s idea, for how to move the animal [DL]. Some student pairs that haven’t shared yet are then given the opportunity to share and explain their drawing [DL]. Students are specifically asked to explain, “What is causing the force in your picture?” [SL]. 

For homework, students are asked to somehow show their parents a push and a pull and tell them that a push or a pull is a force [DL]. For accountability, parents could help students write or draw about what they did, or students would just know they would have to share the next day.
In class the next day, the teacher asks students to share some of the pushes and pulls they showed their parents, asking them to use the word force [DL]. She then asks students to talk with their partner about, “Why did the animal in the chair sometimes move far and sometimes not move as far when we added a force?” [SL]. She then asks some students to demonstrate and describe an idea for making the animal/chair more far or less far; ideally, students will push or pull with varying degrees of force [SL]. Students are then asked to write in their notebooks, “A big force makes it move more!” [L] With a teacher example as needed, they also draw an image of what this might look like [DL]. 

As a possible extension: how would a scientist decide for sure which went further? How would she measure it? The class could discuss and perform different means for measurement, standard and nonstandard [SL].

Middle School

Student groups begin with a set of rocks and crystals at their table. The teacher asks them to generate a set of science-related questions in relation to them, noting those questions in their notebooks [SL]. Each group shares one of their questions, looking for one that’s different from other groups and adding new ideas they like to their list [L]. The teacher then asks students to observe the rocks and crystals with a goal of generating a list of characteristics based on/related to those patterns [SL]. Groups share how they’ve characterized the rocks, adding interesting new ideas shared to their own lists [L]. Tools such as microscopes or magnifying lenses will be useful in their observations [SL]. 

Students next read an article on rock and mineral types, including characteristics of those rocks and minerals [L]. The teacher asks them to specifically find information on characteristics of the rock types that relate to or are similar to the list of characteristics (groupings) that they’ve generated [DL]. Finally, groups of students put their rocks into groups based on the categories and characteristics provided in the text, providing written rationale and evidence for why each rock would be placed within each group [DL]. 

Extension: Before reading the text, the teacher could also elicit background information and use characteristics generated by students to have them predict what types of processes would have been required to create these rocks and crystals, with the given characteristics [SL]. One possible question, “What do you observe about the rocks that makes you think different processes might have been involved in forming and shaping them?” Further, they could model and investigate where the energy involved in creating the rocks came from, linking processes with types of energy (such as gravity providing the energy for erosion due to water or the thermal energy of earth’s core providing energy for metamorphosis of rocks and creation of crystals) [DL and SL]. 

Concluding Thoughts
As I labeled the literacy facets of these lessons, I more fully realized how integrated these components should be in a classroom all the time. Effective science instruction constantly: 1) supports the literacy work that students do (L), 2) as it encourages them to interact with text like scientists (DL), 3) while they engage in meaningful scientific thinking and practice (SL).

Making Sense of Literacy in Science: Applying Literacy Strategies to Science, Disciplinary Literacy in Science, and/or Scientific Literacy

Taking a hiatus from the pathway of my previous posts, I've been thinking about disciplinary literacy this summer.

Notions of literacy in science run in several directions that do not always have clear definitions or delineations. Several resources describe how teachers can apply literacy strategies to support student understanding. Current literature emphasizes “disciplinary literacy” in science, while longstanding commentary also emphasizes the importance of a scientifically literate populace. Below, I propose definitions for these three ideas--applying literacy strategies in science, disciplinary literacy, and scientific literacy--that connect the terms science and literacy, providing examples to illustrate how they differ. In my next blog post, I will describe two science lessons, one from lower elementary and one from middle school, showing how these strategies and frameworks intersect. 

Applying Literacy Strategies to Science

Generally, when I hear about science and literacy, it involves helping students comprehend their science textbook or other science reading. It’s a series of strategies from the field of literacy that educators can apply in a science context. For example, teachers could ask students to do a “close reading” of a text, pulling out specific vocabulary, key ideas, and answers to text-based questions. Or, a teacher might pre-teach vocabulary, and have students write the words in sentences and draw pictures illustrating those words. Perhaps students provide one another feedback on the effectiveness of a presentation. Did you speak clearly and emphasize a few main points? Did you have good eye contact? Generally, these strategies are useful, but they’re not science specific. They could be applied to any disciplinary context. These types of strategies are often mislabeled as “disciplinary literacy.” I would advocate they are not. Disciplinary literacy is not just a new name for reading in a content area. 

Disciplinary Literacy

Scientists have a unique way of working with text and communicating ideas. They read an article or watch a video with a particular lens and a particular way of thinking about the material. Engaging with disciplinary literacy in science means approaching or creating a text with that lens. Notably, the a text is not just a book. The Wisconsin DPI defines text as any communication, spoken, written, or visual, involving language. Reading like a scientist is different from having strategies to comprehend a complex text, and the texts involved have unique characteristics. Further, if students themselves are writing like scientists, their own texts can become the scientific texts that they collaboratively interact with and revise over time. In sum, disciplinary literacy in science is the confluence of science content knowledge, experience, and skills, merged with the ability to read, write, listen, and speak, in order to effectively communicate about scientific phenomena. 

As a disciplinary literacy task in a classroom, students might be asked to write an effective lab report or decipher the appropriateness of a methodology explained in a scientific article. They might listen to audio clips, describing with evidence how one bird’s “song” differs throughout a day. Or, they could present a brief description of an investigation they’re conducting in order to receive feedback from peers.

Scientific Literacy

Scientists use a set of skills that is broader than text, moving beyond the realm of “literacy” in terms of language, to specific ways of thinking about and interacting with particular phenomena. Therefore, disciplinary literacy in science is a subset of the broader abilities described within the concept of scientific literacy. 

Building on ideas from the National Science Education Standards (p. 22, 1996), scientific literacy implies a base level  of scientific understanding and ability that enables a student to effectively “ask, find, or determine answers to questions derived from curiosity about everyday experiences.” Literacy skills generally connect with science understanding as a student researches a phenomenon of interest or even watches the news. Further, scientific literacy and disciplinary literacy merge in the ability to “read with understanding articles about science in the popular press and to engage in social conversation about the validity of the conclusions.”  Further, “a literate citizen should be able to evaluate the quality of scientific information on the basis of its source and the methods used to generate it”--a process that requires an understanding of scientific practices along with the literacy skill of evaluating sources. 

Clearly, disciplinary literacy and scientific literacy work together. Because “scientific literacy implies that a person can identify scientific issues underlying national and local decisions and express positions that are scientifically and technologically informed,” our students must develop the skills and understanding to clearly communicate complex, scientific information through various media. They must also learn “to pose and evaluate arguments based on evidence and to apply conclusions from such arguments appropriately.” Argumentation with evidence is a literacy skill that applies across all disciplines, but it requires an ability to apply unique disciplinary lenses to evidence and communication methods to effectively carry it out in a scientific context. 

Continuing with the brief bird song example mentioned above, scientific literacy skills would be necessary to generate meaningful questions, such as, “Why do birds sing?” Or, further, “Do variations in bird song correlate with particular behaviors?” Students might start with making observations of bird behaviors and songs through a set timeframe, then comparing and contrasting bird songs. Determining a proper timeframe, sufficient subjects for study, and other investigation parameters would be aspects of scientific literacy. Effectively collaborating with classmates to conduct the investigation and communicating explanations with evidence would involve essential disciplinary literacy skills. 

In other words, scientific literacy encompasses a broader range of work with scientific practices than disciplinary literacy. Students can show some ability and understanding of science practices through text, but scientific literacy also requires physical performance of these practices. Further, it requires background knowledge, whereas a disciplinary task in science, such as creating an effective data table, might not require such knowledge. Students planning and conducting a chemistry experiment moves beyond textual interaction to knowing when to use a fume hood and how to be safe more generally in a lab. Scientific literacy also requires mathematical thinking and a comprehension of scale for effective study of daily phenomena. These skills come into play in defining and modeling a particular system, with all relevant components.  

A late addition to this post - great article along these lines in Science!

In my next blog post, I’ll look more in depth at two lessons, one from elementary and one from middle school, showing the intersection of these three notions of literacy and science.

Conducting a Science Program Audit

With a mission and vision in place for the science program, teachers will need to personally and collaboratively decide whether or not they have a sufficient understanding of what instructional practice looks like under that vision. In future blog posts, I’ll be providing suggestions for professional development on research-based science instruction. For this post, I’ll be assuming (always a risky plan!) that teachers have the skills and understanding necessary to evaluate their work in relation to their vision.

Therefore, to determine progress towards a vision, a school or district will first need to determine where they’re at now. A science program “audit” is a strategy to do that. It asks, “How well does our instruction align with this mission? How well are we accomplishing our vision? How do we know?”

Forming a Committee

Again having an audit committee will be a valuable guide for this process. While there could be some overlap with the leadership committee that crafted the mission and vision statements, the auditors should consist of outsiders, not district employees. A school will need constructive, impartial outsiders to give an unbiased perspective. Some suggestions for members of this committee:

1. A recent graduate of the district currently studying science in college
2. Community members working in science-related fields (healthcare, high tech, university research, etc.)
3. Science education professors or coordinators (college, university, teacher professional organizations)
4. State education department or regional education service agency science education leaders
5. Parents
6. Educators from neighboring school districts (would be great to have districts across a region support each other in this audit process)

The size of the committee needed depends on the size of the district. The committee would ideally be able to visit every school in the district and interview a representative sample of teachers and students across the district that includes some from every school. One strategy would be to have two auditors work together, visiting one school in the morning and one in the afternoon. Thus, in a district with 40 schools, 20 auditors would then be needed to complete all of the visits and interviews in one day. Of course, fewer auditors would be needed if the process was spread over several days. More in-depth audits could be conducted by one or two auditors over the course of several months, though an audit can be a unique opportunity to engage a broad range of community members.

Data Gathering

Before the committee comes to the school, administrators and teachers should collect and share useful data and information about the school/district science program. This data should be broken down by all appropriate subgroups and could include:

• Assessment data: standardized test scores, district or grade level common assessments, classroom level assessment examples;
• Descriptions of courses taught and typical student pathways: for example, it will be important to note if students from particular backgrounds tend to be in “honors” classes to a lesser extent;
• Teachers’ license and longevity information;
• Postsecondary pursuits: career or college/university pathways;
• Any documentation of science program mission/vision and links to science department or teacher websites.

Information gathered before the auditing day(s) should also include broad-scale surveys to gather quantitative data on the science program. These could easily be created and conducted through Google forms or another free survey tool. These surveys could alternatively be developed, conducted, and analyzed by the auditor(s).

Student Surveys Example Questions

• Elementary: at lower elementary the teacher would need to read and explain the questions. A simple yes/no might be most appropriate K-2. 

1. I enjoy science class. [A lot, kind of, not really]
2. We do interesting experiments and investigations in science [A lot, sometimes, not much] 
3. We learn about scientists in class that look like me [yes, no]
4. We study science outside [A lot, sometimes, not much]
5. I get to study my own questions in science class [yes, no]
6. I learn about things in science class that I wonder about in my life.

• Secondary:

1. I enjoy science class. [A lot, sometimes, not really]
2. We do interesting experiments and investigations in science [A lot, sometimes, not much]
3. We learn about scientists in class that look like me [yes, no]
4. We study science outside [A lot, sometimes, not much]
5. I get to study my own questions in science class [A lot, sometimes, not much]
6. I learn about things in science class that I wonder about in my life or the world around me [A lot, sometimes, not much]

Parent/Community Survey Example Questions

1. I am satisfied with the science education that my child (or our community’s children) is receiving [yes, somewhat yes, somewhat no, no]   Explain: [open-ended]
2. Students are involved in science learning relevant to needs and issues in our community [A lot, sometimes, not enough]
3. Students are learning relevant 21st century skills through science classes, such as analyzing scientific studies and evidence, communicating technical information, conducting investigations, and collaborating with peers [A lot, somewhat, not adequately]
4. Students are adequately prepared for postsecondary careers or educational pathways [A lot, somewhat, not adequately] 
5. Community members and experts are invited into science classrooms and to support science projects [A lot, sometimes, not enough]

Teacher Survey Example Questions

• Select how much you agree with the sentence.

1. I am comfortable teaching science. [Yes; somewhat yes; somewhat no; no]
2. I have sufficient collaboration time with other teachers focused on science. [Yes; somewhat yes; somewhat no; no]
3. I receive sufficient science-related professional development from my district/school. [Yes; somewhat yes; somewhat no; no]
4. Our current curricular materials effectively support science instruction. [Yes; somewhat yes; somewhat no; no]
5. I have adequate lab and related science equipment for effectively teaching science. [Yes; somewhat yes; somewhat no; no]

• The next set of questions asks about frequency of various actions.

1. I take advantage of outside professional development opportunities (not school or district sponsored). [At least monthly, five to ten times per year, two to four times per year, about once per year, once every few years, never]
2. Students have opportunities to investigate their own questions in my science class [At least monthly, five to ten times per year, two to four times per year, about once per year, never].
3. Our class goes outside or to other relevant sites in the community (or beyond) for science learning and investigation [At least monthly, five to ten times per year, two to four times per year, about once per year, never].

To better understand the “why” for the survey answers, auditors should also conduct interviews and focus groups of students and teachers based on these questions. This back-and-forth will allow for follow up questions and clarification. Focus groups will allow for greater synergy of respondents and will typically be more appropriate for work with students, where one-on-one interviews are not advisable for student safety and comfort reasons.

Observations

As part of the day(s) that the audit committee is in schools, they should be observing classrooms. Ideally, this observation will happen naturally in the flow of instruction, not just be scheduled for the best lab of the year. The auditors need an authentic perspective on what is happening in the day-to-day work of the classroom. Notably, auditors without an education background will need further guidance on what they’ll be looking for in classrooms. In addition to general, objective observations of what happens in the class, some specific prompts/questions that auditors could be answering include:

1. Describe the classroom. How are desks arranged? What materials and space are available? 
2. What happens during the class? What is the duration of each segment of the class? [lecture, independent work, collaborative work, and whole-class discussion]
3. What types of questions is the teacher asking the students? What questions are students asking of each other and of the teacher? [Yes/no, clarification, one-right answer, logistical, deeper though required, etc.] 
4. What type of work are students doing? [step-by-step lab, worksheet w/ one right answer, open-ended problems, peer discussion, arguing with evidence, modeling or investigating a phenomena, 
5. How does the teacher establish whether or not the students understand the material? 
6. How does the teacher ensure all students are equitably engaging in the instruction?

Bringing It All Together

At the beginning of the day (week, month, or year) the audit committee would come together with administrators and science teacher leaders to establish their roles and the plan of attack. They would receive guidance on what’s expected of them and a schedule, maps, and other needed support in accomplishing those tasks. Ideally, they would have tasty snacks (I prefer cake donuts and seasonal fruit, in case you’re wondering) and lunch available during the day. At the end of the allotted timeframe, the auditors will come together to discuss the data and observations as a group and collaboratively review the day. They should have feedback templates where they can create and share consensus comments of the group and their individual feedback on the day, though individual feedback could come back a few days/weeks later after auditors have had a chance to fully analyze the data provided from the school, their observations, and the observations of their fellow auditors. Templates would include the specific questions that the school/district wants feedback on from the auditors, as well as space for open reflections on the wide-ranging pieces of data reviewed.

After conferring as a group, the auditors should present their initial findings to teachers and administrators, with the opportunity for auditors and district educators to ask clarifying questions of each other. This discussion will need an able facilitator, as it is a time for teachers to reflect on the observations shared by the auditors, not get defensive about perceived negative feedback.

Once final feedback is received by individual auditors, either a primary auditor or a school/ district leader should combine the data and feedback into a final report. Several science department and leadership committee meetings will need to be devoted to reviewing the feedback and determining the next steps in moving toward the vision and mission for science education. 

*Special thanks to Judy Singletary of Stoughton School District for ideas on this audit process.