Geoscience Education and Public Outreach Network (GEPON)
Report of a workshop
held at UNAVCO,
March 22-24, 2006
The Digital Library for Earth System Education Program Center (DPC) of the University Corporation for Atmospheric Research (UCAR), working with the University of California Museum of Paleontology (UCMP), convened the Geoscience Education and Public Outreach Network (GEPON) workshop. This workshop was in direct response to recommendations emerging from the NSF-funded Geoscience Outreach Workshop held at the UCMP in May 2005. At that time, participants proposed the formation of a formal network (GEPON) to support geoscience outreach efforts and a community meeting to articulate a coherent plan for a successful network effort.
The
primary goals of the 2006 workshop were to examine the feasibility of
establishing GEPON, to identify the services the network ought to provide, and
to formalize recommendations to the NSF and the geoscience community for
network implementation, support and sustainability. The workshop was co-hosted
by Susan Eriksson, Education and Outreach Director at UNAVCO, and John Taber, Education and Outreach Program Manager at IRIS, and was
held at the UNAVCO facility in
Rationale
In her 2005 AAAS Presidential Address, Shirley Ann Jackson called for the scientific community to foster better public understanding in order to overcome "mistrust of science, distrust of scientists, and a movement away from understanding the importance of science to modern life, of its role in addressing issues of human health and welfare" (Jackson, 2005). Multiple research studies (NRC, 1996; Loucks-Horsley, 2003), confirm that direct engagement of scientists in education and outreach is critical if positive and lasting outcomes are to be expected.
As with all of the sciences, geoscience research efforts have been generously supported through federal funding, and in turn, the geoscience community has the responsibility to demonstrate to the nation that these funds have been well-invested. NSF’s Broader Impacts review criterion provides an opportunity to fulfill that responsibility. Broader impacts requirements do not diminish or take away from research efforts; rather, the broader impacts of research programs enhance and add value to the overall research enterprise.
A commitment to broader impacts should be intrinsic and complementary to all geoscience research efforts in order to:
- Foster public understanding of the processes and products of the scientific enterprise, and of the relevance and importance of the geosciences to our communal lives
- Demonstrate that the geosciences contribute significantly to the nation’s collective health, security and economic well-being
- Support human resource development in the geosciences to help prepare the workforce for the 21st Century
- Promote technology transfer from fundamental research to applications in the private sector
- Demonstrate accountability to the public that the funds entrusted to the Directorate for Geosciences are well-managed and have provided measurable benefits to the nation.
Geoscience
research efforts have been generously supported through federal funding, and in
turn, the geoscience community has the responsibility of sharing that knowledge
with the nation. Beyond this responsibility, however, is the disturbing fact
that science knowledge in the
Prior Work
The
initial NSF-funded geoscience outreach workshop, organized by the University of
California Museum of Paleontology and the
- provide an infrastructure to facilitate collaboration between scientists, educators, and EPO professionals
- provide support systems for professional development and cross-disciplinary learning for all involved in EPO activities
- recognize and give value to current EPO activities, thereby encouraging involvement by others
- promote and help to institutionalize a cultural change that recognizes, values, and supports outreach as both a responsibility and an opportunity.
Workshop participants agreed that it would be necessary to convene an additional workshop to articulate the characteristics, strategies, and management models for a successful network effort.
Workshop Goals and Participants
The
primary goals of the 2006 workshop were to examine the feasibility of
establishing GEPON, to identify the services the network ought to provide, and
to formalize recommendations to the NSF and the geoscience community for
network implementation, support and sustainability. Thirty-eight scientists and
EPO professionals from 17 states and the
Workshop Outcomes and Recommendations
An initial discussion of an informal survey of NSF Program Officers in the Directorate for Geosciences (Mayhew, 2006; see Appendix A) provided a fresh perspective on the importance of BI and on the potential role(s) of GEPON. The survey revealed the need for clarity regarding the purpose of Broader Impacts, consistency in the application of Criterion 2 in the review process, and better understanding among principal investigators of the array of possible activities that satisfy the BI criterion. These survey results and outcomes from the initial workshop informed the discussions that followed, which resulted in recommendations for five key initiatives that target two audiences: the geoscience research community and those who assist that community with their broader impacts efforts. Working groups were formed to pursue the following initiatives:
- Provide community assistance to NSF. The purpose of this initiative is to clarify and integrate BI into the peer review process. It was recommended that a small working group be established to approach high visibility scientists to discuss the importance of BI, to develop a series of PI meetings with NSF on the subject of BI, and to establish guidelines for reviewing BI within proposals. (See page 3)
- Showcase Broader Impacts exemplars. Such a showcase would provide ideas for models to be adopted or modified and would give recognition to those who participate in BI. It was recommended that a collection of examples be identified to demonstrate best practices and to provide guidelines for implementing exemplary BI activities. (See page 4)
- Provide a venue for information exchange. It was recommended that a community database be developed that would link people to people and provide access to information about broader impacts efforts. (See page 5)
- Support an annual meeting. An annual BI meeting would provide professional development for all BI stakeholders and participants and provide an opportunity to build partnerships and share experiences in BI activities. (See page 6)
- Increase appreciation of broader impacts programs. This was deemed to be a long-term initiative that will build on the previous initiatives. As such, it will focus on activities that more effectively engage geoscientists in broader impacts efforts and will instill a positive perspective of such efforts within the scientific culture.
The following section reflects the discussion and resulting recommendations from the working groups that focused on the first four initiatives.
GEPON Initiatives
1. Provide community assistance to NSF
A significant part of the geoscience community recognizes the importance of demonstrating the broader impacts of our research to engender broader public awareness and support. The participants of this workshop are prepared to assist the Directorate for Geosciences by providing information services, working groups, and collections of best practices and examples; identifying key elements of success; and evaluating and documenting the impacts. More specifically, participants recommend and are willing to assist with the initiation and implementation of (a) a BI workshop for principal investigators and program officers and (b) the development of a BI guide for PIs and reviewers.
(a) Broader Impacts Workshop for Geoscience Principal Investigators (PIs) and Program Officers (POs)
To help the geoscience research community understand the full range of opportunities encompassed by “broader impacts,” a series of workshops for PIs and POs in all GEO-sponsored programs is proposed. This recommendation follows the lead of a similar program sponsored by the Directorate for Biological Sciences and the Division of Undergraduate Education in which all PIs (and their EPO lead staff) from the Biocomplexity program were assembled at the National Academy of Sciences. The workshop and ensuing volume are entitled Integrating Research and Education, Biocomplexity Researchers Explore the Possibilities (NRC, 2003), http://www.nap.edu/books/0309088712/html/. At this workshop, experts in science education and outreach presented a series of “primers” on topics such as “Principles of Research Applied to Education Projects”, “Forming Collaborations”, and “What Constitutes an Effective Undergraduate Research Project?” The Biocomplexity workshop was focused on many dimensions specific to education, but a wider range of possible broader impacts approaches could also be developed for diverse applications in the geosciences.
As all-PI meetings are a customary event in many programs across the NSF, the structure for such a BI workshop is already in place and could be both informative and collaborative. In the case of the geosciences, these meetings could be organized by program, section, or division. Although the focus of the proposed workshops is on broader impacts, this event could support the development of networks among researchers and contribute to a significant exchange of ideas leading to additional collaborations in broader impacts and in research.
(b) Guide for Developers and Reviewers of Broader Impacts Activities in NSF Proposals
To help the geoscience community understand how to design and evaluate a quality broader impacts activity, workshop participants recommended the development of a guide that outlines essential elements and best practices for a full range of broader impacts activities. The guide would assist PIs in development of project concepts, and if formally endorsed by NSF, could inform reviewers of proposals.
The contents of the guide would include suggestions and examples of how to align and scale the broader impacts effort to the primary research component of the project. Additional topics might include the importance of knowing and involving target audiences in the planning and implementation of the activity, how to explore the relevant educational research, identifying related projects and collaborations upon which to build or connect activities for greater impact, and the need for evaluation. Best practices and key elements could be culled from advice given in many existing education-related NSF program solicitations and from community experience. The final product would include resources such as checklists for project development and best practices and references to other resources of interest. Additionally, these best practices and existing guidelines in relevant NSF solicitations could be assembled into a matrix to identify the common elements across all broader impacts activities as well as those that are unique to certain types of activities. The guide could be disseminated on the NSF site, if approved, or referenced in NSF solicitations.
2. Showcase Broader Impacts exemplars
The variety of public audiences mandates different approaches to sharing our work with these publics. To foster this understanding among researchers, it was recommended that a collection of exemplars be identified to demonstrate best practices and to provide guidelines for implementing BI activities. These examples, or case studies, would also give recognition to high quality BI initiatives and to the researchers involved, and would provide ideas for adoption or modification by others (see Figures 1 and 2, and Appendix B). Toward the establishment of such a collection, it was recommended that a working group be established. This group would bring together critical stakeholders willing to develop the case study collection of exemplary BI projects. Ideally this group would consist of individuals with diverse expertise in areas such as geosciences research, K-12 education, teacher professional development, minority partnerships, and evaluation. This working group would determine the criteria by which to select BI cases and would design the system for data collection. This group would also be responsible for curation and growth of the collection and its broad dissemination within the geoscience community.
Figure 1: A sample BI activity Submitted by Sharon Franks, Scripps Institution of Oceanography |
Figure 2: A sample BI activity Submitted by Anupma Prakash, Geophysical Institute, University of Alaska Fairbanks |
3. Provide a venue for information exchange
Collecting and disseminating information about broader impacts activities, professionals, and evaluations is a keystone activity of the network. Much of this information has been collected by DLESE, and should continue. We envision developing a database to support this effort and to facilitate ongoing communication within the community. This database will contain information about professionals and researchers who conduct BI activities and will serve as the primary tool for connecting PIs with broader impacts professionals.
Several steps were recommended to meet this goal:
- Development of the
database: A
prototype for the database was begun, by Mark Chandler at
- Verification of entries: Because individuals will enter their own data, verification prior to public access is necessary. Meeting participants recognized that quality assurance for these entries will be a critical and time-consuming task.
- Populating the database: The attendees of the two GEPON workshops are a small subset of the community that participates in BI activities. However, as GEPON attendees belong to a diverse set of existing network(s), these networks can be mined to populate the database.
- Dissemination of the database: Once the database contains a critical mass of entries, it should be marketed to the target audience: researchers and PIs. It may be most effective to conduct this promotion through NSF, its website, and program solicitations. Additionally, it is important to seek out additional dissemination channels, such as NASA Space Science Support Network.
- Communication within the network: Additional tools are required to facilitate broad communication within the community. Examples include an edited email newsletter, threaded discussion forums, and a “news and opportunities” area within the database.
- Maintaining the database: Maintenance includes monitoring inactive accounts, event calendars, and discussion forums. The use of automated tools to perform these tasks currently exist within the community, such as those established by DLESE.
4. Support an annual meeting
In
support of the above initiatives, scientists and EPO professionals must work
together to make meaningful progress toward addressing the NSF Broader Impacts
criterion. Communication within this community is essential and can best be
supported by annual meetings. Several participants at this workshop, as well as
the previous
Both the scientific and EPO communities will benefit by these meetings: the science community will be more informed about the needs and options for BI planning, and the EPO community will gain knowledge and skills that can be more effectively applied to BI projects. New science/EPO partnerships will naturally form, and over time, the overall awareness, appreciation, understanding and application of BI will be enhanced.
Summary
Why should scientists care about the broader impacts of their research? Workshop participants articulated several reasons. Most significantly, when research is funded by public dollars, scientists have an obligation to return the results and implications of that research to the public. Obligation, however, is a poor motivator. Instead, scientists will be motivated when they see clear, direct benefits to conducting thoughtfully designed, integrated broader impacts activities – benefits that may include additional funding to sustain research, more and better students in their classes, and greater recognition in traditional venues such as tenure decisions.
Evidence is growing that these benefits are, in fact, real. Increasingly, colleges and universities include education-related activities in promotion and tenure decisions. Many scientists who have become involved in broader impacts activities report highly rewarding experiences. Still, many scientists are unprepared to make this leap. Too frequently, they view the broader impacts criterion as a burden and hastily tack on a selection of activities in an attempt to adhere to proposal guidelines. Poorly integrated broader impacts activities defeat the purpose of the broader impacts criterion. They do little or nothing to alleviate the public perception that science is difficult, inaccessible, and, even worse, irrelevant to our daily lives.
Direct, personal benefits do result from scientists’ engagement in high-quality broader impacts activities. However, this engagement should be regarded as both a responsibility of and a benefit to the entire scientific community. Scientists have a critical role to play in the public understanding of science and in nurturing the next generation of scientists. Now is the time for scientists to recognize that they can communicate the excitement of the scientific endeavor and the relevance of their research to a broad audience – an audience that includes teachers, students, parents, the media, voters, legislators, museum-goers, national park visitors, people in rural communities and urban centers – in short, everyone.
Participants also believed that a significant amount of work will need to be done to make the science community aware of the GEPON concept and to buy into a collaborative plan. At this point, it is not entirely clear what that commitment is, although many believed that positive signs of a cultural change were being exhibited on university campuses and in national venues such as AGU, GAS, and AMS conferences.
This
workshop provided an important step toward recognition of the importance of and
opportunities from scientist’s engagement in education and public outreach
activities. The proposed initiatives will add clarity and consistency at the
level of proposal submission and review, will build bridges among those who are
engaged in or interested in outreach activities, and will facilitate
communication to sustain current collaborative efforts. Participants agreed
that ultimately a more formalized network would be needed to maintain these
activities and to help scientists move into this new territory with enthusiasm
and confidence. Participants also agreed that some NSF support would be needed
for a central structure, but that much of this activity would have to be
community based.
Editors
Mary
Marlino and Judy Scotchmoor
Contributing Authors
Anne
Egger
Susan
Eriksson
Sharon
Franks
Michelle
Hall
Mary
Marlino
David
Mogk
Martin
Ruzek
Judy
Scotchmoor
Sophie
Warny
Organizing Committee
Mary
Marlino,
Judy
Scotchmoor,
Steve
Ackerman, University of Wisconsin-Madison
Anne
Egger,
Susan
Eriksson, UNAVCO
Sharon
Franks, Scripps Institution of Oceanography, University of California, San Diego
Jack Hehn, American Institute of Physics
Ian
Macgregor, National Association of Geology Teachers and
Bibliography
Jackson, Shirley A. (2005). The Nexus of Science
and Society. Keynote address February 17, 2005 to the AAAS 2005
Annual Meeting (American Association for the Advancement of Science):
Loucks-Horsley, S., N. Love, K. E. Stiles, S.
Mundry, and P.W. Hewson. (2003). Designing Professional Development for
Teachers of Science and Mathematics.
Mayhew, Mike. (2006). Responses to questions
put to GEO program managers concerning Broader Impacts. Survey in March
2006.
National Research Council. (1996). The Role of Scientists in the
Professional Development of Science Teachers.
National Research Council. (2003). Integrating
Research and Education, Biocomplexity Researchers Explore the Possibilities:
Summary of a Biocomplexity Workshop.
National Science Board (2006) Science and Engineering Indicators – 2006. Arlington, VA: National Science Foundation, Foundation (volume 1, NSB 06-01; volume 2, NSB 06-01A).
Scotchmoor, J., M. R. Marlino, and Russanne Low.
(2005). Making a Broader Impact: Geoscience Education, Public Outreach, and
Criterion 2. Report of a workshop sponsored by the National Science
Foundation: University of California Museum of Paleontology, Berkeley, and the
Digital Library for Earth System Education (DLESE) Program Center, University
Corporation for Atmospheric Research (UCAR). http://gepon.org/
Appendix: Related resources
Appendix
A: NSF Broader Impacts survey (Mayhew,
2006)
Appendix
B: “Why Earth Science?” flyer
Acknowledgements
This workshop was a joint project of the Digital Library for Earth System Education (DLESE) Program Center and the University of California Museum of Paleontology. Funding from the National Science Foundation was received under grant 0622010 to DLESE. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
Special thanks to:
Hildy
Kane, Mary Beth Reece, Mic Stremel, and all workshop participants for their
contributions to this document.
Appendix A: Responses to questions put to GEO program managers concerning Broader Impacts (Mayhew, 2006)
1. What in your view constitutes a high-quality response to the Broader Impacts requirement? Can you characterize the Broader Impacts elements of one or two awards from your own program that you regard as exemplary?
A high quality response is one that considers the whole range of broader impacts that the work might have, not just "Oh, and the project has a graduate student". The really good ones are usually about a half page with short paragraphs elaborating each broader impact, for example, impact on the subdiscipline, or research infrastructure, or international collaboration, or societal impact, etc... Good responses are becoming more common.
In my view, BI comprises two important elements. Scientific impact outside the immediate field, and public impact. Simply supporting the education of students already involved in the field is no longer sufficient - this is the standard proposed activity to satisfy the BI requirement. If a project will involve some possibilities of applications/discoveries in other fields, and/or a good public outreach or even a policy application, then this will usually review well against the BI criteria.
Most of
our large facilities have E&O Programs that are for the most part effective
but need additional resources in my opinion if we want them to do
more. Certainly, IRIS, UNAVCO and
COMPRES have small but effective responses in this area, and given their
limited resources (compared to other activities of the facilities) are
doing well. I think the new
IRIS effort to work with a school system is a very effective model
but also a difficult one to break in to. They are beginning in
Any number of things from outreach in National Parks, building GPS networks, support of PIs and students from underrepresented groups, etc. Didn't a division in MPS do a workshop/document where they highlighted all sorts of BI? I saw this and thought it did an excellent job of presenting examples. There are some good examples on the NSF web site, too.
The best response should address why the project would impact somebody else but the PI. Many PIs do not understand that broader impacts include more than education and outreach. It also includes dissemination of previous results (i.e., your publication record), supporting infrastructure for use by the community, international collaborations, and potential societal impacts of the research.
I would need to spend some time looking through several summaries to find one that is truly exceptional (they are rare), and I am pressed for time right now…Here are a couple of very good but not outstanding ones.
Example 1 – addresses collaborations
Broader impacts of this project include training of a Ph.D. student in a variety of ignimbrite correlation techniques and modern 40Ar/39Ar geochronology methods. Undergraduate students will be involved in paleomagnetic analytical work and fieldwork. Our findings will be disseminated to K-12 educators, non-traditional students, and the general public through the VolcanoWorld web site (http://volcano.und.nodak.edu/vw.html) and synergistic activities. Scientific results will be disseminated in a timely fashion to professional peer-groups and should have societal impact in furthering our understanding of episodicity in large, explosive volcanic systems that are of concern for hazard mitigation programs. The close collaboration with a Bolivian colleague and his student ensures training, data and knowledge transfer, and consolidation of international cooperative research. Collaboration with an Argentine colleague will aid the process of correlation across the international border.
Example 2 – Mostly education and outreach
The broader impacts of the proposed studies center around an integrated plan for scientific research and human resource development. Five undergraduates, three graduate students, and eight teachers will participate in research over the five year award period. A research experience program for K-12 teachers in volcano-petrology (RET/VP) will provide an opportunity for teachers to participate in research activities for six weeks during the summer. The goal is to achieve a collaborative lab group wherein undergraduate students pursue senior thesis research projects, graduate students gain leadership experience mentoring the undergraduates and teachers, teachers deepen their confidence and mastery of science content, and the ensemble benefits from exposure to differing perspectives on effective teaching and learning.
In my view BI refers to benefits stemming from a funded project that go beyond the primary scientific objectives, and more specifically related to the human resources side – including training, outreach, preparation of materials to educate and spread the significance of the research to a broad audience.
One that integrates the BI activity with the research, demonstrates that the PI is truly committed to the activity (indeed, enthusiastic about it!), and one that will be catalytic. The activity may be one large scale effort that fits into any one of the 5 strands of BI activities, or may include several efforts that are of smaller scale.
A high quality response to the EPO requirement is difficult at best to describe, and one rarely had the sense that it is at all a priority in the development of the proposal. When we do see one, it does seem to have certain components which are common: involvement of the PI(s), involvement of the public, incorporation of some sort of internet resource. Far too many of our proposals have Broader Impact sections which are tacked on at the end, and actually come at the end of the proposal, and are obviously simply an effort to conform to the letter of the requirement.
A high-quality response is one that goes beyond a simple statement of student support and includes a response in which the PI stands back from the proposed research and asks "So what?" in terms of who would/should care about the implications of the research beyond a small group of scientists or in terms of a PI's own career goals.
I
consider broad impact in two categories: what project would do for science or
society, and in the training of students or outreach to other groups. In the
case of exemplary projects, the broad impact has dealt with training. One of
them runs a summer program that would take graduate students and teachers from
under represented groups to a field site to teach evolution by studying
specimens in a stratigraphic sequence. This was part of many more activities,
but this feature made it unique. The other involved taking minority students
from
A high-quality response to BI generally brings some level of innovation to the standard, though acceptable, elements used to address the BI requirements, and is usually well-integrated into the research component of the project. This may mean bringing in new researchers from other scientific communities or engaging underrepresented groups in a novel way, or creating a new and innovative educational tool or class. Some projects with excellent BI:
CAREER EAR-0448871 : Geophysics "garage". This was a CAREER proposal, so we generally expect the BI to be strong in these proposals. But this concept was unique for our community and went beyond even the high standards we hold for CAREER awards. The PI would engage the Visual Arts community at his institution for the creation of what he termed, the "garage." This facility would be a work space that would help students gain a physical understanding of geodynamics through the construction of demonstrations for the general public. This work was tied to his research, because his geodynamics modeling would inform the physical concepts for the demos. These displays would then be used as teaching tools, and the construction methods would be made available to other researcher/teachers who were interested. The PI also developed workshops for high school teachers in the local area to help them make use of the demos that came from the project. The work was noted because it engaged graduate, undergraduate and high school students, it created collaborations with academics outside the sciences and had a unique and broad vision for making complex concepts in geodynamics more accessible.
Panero, the PI, an early-career female researcher, developed collaborations with researchers in material sciences to create a new and innovative experimental tool that if successful would be broadly utilized in both mineral physics and material science. In addition she would start up a high temperature and high pressure lab at OSU and provide laboratory training for undergrads and grads.
One that actually addresses the NSF broader impacts in a serious way. Some proposals treat the BI criteria as an afterthought to their overall research goals, whereas other try in meaningfully integrate the BI criteria in to the work plan. An exemplary example would be that of Michele Cooke's CAREER proposal and her work with deaf students and schools.
A high-quality BI constitutes something beyond the normal "graduate student education and dissemination of results that will be fruitful to a broad audience" that I often see. What "beyond" means is more difficult to describe, but it might involve the direct participation of an educator or education specialist. This serves two valuable purposes: it somewhat frees the PI from the direct responsibility of planning things that are outside of her or his experience, and it brings in someone who really is an expert and that can offer cutting-edge ideas. A few PIs have made connections to National Parks, some going as far as wanting to build physical analog experiments that demonstrate, in this case, glacial processes. Earth science is generally inadequately presented in National Parks, so this connection is a good idea. Another example is the transfer of a potentially useful but hard-to-learn technology or method to the broader community. Specifically, a PI is proposing to use a new web-based service to manage and better communicate the status and progress of a project that has multiple interested parties (0617557). Some PIs make a cogent case for the relevance of their focused research on related fields both within EAR and outside, although there is also a lot of gilding the lily here, too.
Most of our BI components are nearly implicit; improve weather and climate predictions [which provide] societal benefits, etc. In general, I have never really thought of the review criterion and evaluation beyond that. There are some notable exceptions, but they are easy, e.g., young scientist workshops and the like. New techniques of measuring important climate related changes.
A high-quality response to the broader impacts requirement, should not be a response at all. The weakest descriptions of broader impacts are those that appear to be thrown in at the end to fill in a requested field in the proposal format. In some cases, the proposals that are strongest in broader impacts just say what is going to be done in EPO, diversity, data dissemination, international collaborations, etc. The most convincing proposals provide details about the actual activities to be undertaken, a plan and schedule for performing those activities, a description of expected outcomes, and a metric by which the effectiveness of the program will be evaluated. I seldom see proposals that are this thorough in terms of activities producing broader impacts.
2. To what extent do you take account of the Broader Impacts component in your proposal review process and in your funding decisions?
We explicitly consider what the mail reviewers say about broader impacts, and what the panel says about them as well. Our review analysis template has a section for each of these. In most cases, for "normal" research proposals, intellectual merit is weighed more heavily than broader impacts, but not in all cases. And then there are some proposals, like database ones, that are mostly "broader impacts" and are funded for that reason. So I guess the general answer is "it depends". Not much help here....
The BI criteria are an integral part of the review process. They are stressed for the ad hoc and panel portions of the review, and they are used in programmatic decisions. However, if a proposal has weak intellectual merit, the broader impacts are irrelevant.
All awards made by [my program] are by definition contributing to broader impacts (enhancing the infrastructure at the institution). However, reviewers, panel, and the program expect more in having concrete management plans that include EPO. In some cases (usually within the MRI competition for non-PhD granting institutions) we weigh the BI's criteria heavily and make awards where the IM of the proposal is not as compelling.
Both the panel and myself do pay attention to this, and are always delighted to see an unusual and innovative plan for BI. Having said that, the top priority is always the science. The best BI in the world will not overcome a poor intellectual merit, but BI does become a factor in proposals that are otherwise judged of equal intellectual merit.
We make sure BI are discussed in panel, ask them to give us a qualitative ranking (strong, typical, weak), and include a discussion in the panel summary. We include a discussion of BI in the review analysis. They are a factor in our decisions.
I think that I know the broader impacts of the activity better than the PIs, and I take that into consideration when making awards at the waterline. Panel seems to recognize when somebody has made a serious attempt to respond to the broader impacts criteria in a meaningful way. They acknowledge the ones who are exceptional but do not seem to harp too much on those that have not done an adequate job.
We certainly look to see if BIs are addressed, and if so how. It is expected that all fundable proposals will at least address this issue. However, I would say that this is a secondary issue to the intellectual merit, innovativeness, and importance of the science proposed. Without the latter, we would not consider funding at all. So, BIs can be influential in discriminating added values that could determine whether or not an award is likely be recommended.
It is important, but obviously secondary to the research merit itself. We do not actively encourage excellence in broader impacts through any specific instructions on our website or through our communications with the PIs. It is something which winds up coming through when the PI and the institution are both involved in the preparation of the proposal.
I take them into account as secondary to Intellectual Merit. If a proposal has low intellectual merit, strong broader impacts will not rescue it from a decline. If the proposal rates good-very good as to intellectual merit but very good-excellent as to broader impacts, then that proposal will be funded over a higher intellectual merit but lower broader impacts proposal.
If a project has a poor broad impact, it would not be funded. Nevertheless, the quality of the science is about 60%-75% of the decision. If a project has an excellent scientific component and students will be trained while working on the project, I consider that the students will be exposed to superb science training and the project is building human capacity.
When the BI are exceptional or unusual, they are noted by the panel and can be weighted more than the Intellectual Merit. We make sure that the panel discussion includes a section on the BI for each proposal. Sometimes it's easier than other times. When a collaboration that broadens participation, say with minority serving institutions or foreign partners, is integral to the research being conducted, it is very easy to have the BI discussion. Other times, if they are not well articulated in the proposal or are exaggerated or difficult to discern, it is harder to have that discussion. We account for BI in funding decisions differently depending on the case. CAREER proposals, of course, need a very strong educational component to be competitive. For other proposals we view the BI as one of handful of important criteria that determine funding, intellectual merit, BI, high-risk, PI history, geographical and programmatic balance.
The BI criteria are extremely important, especially in judging proposals that are essentially even with respect to reviews, ranking, etc. In these cases, strong BIs can be the reason to fund one proposal over another.
Both the panel and myself do pay attention to this, and are always delighted to see an unusual and innovative plan for BI. Having said that, the top priority is always the science. The best BI in the world will not overcome a poor intellectual merit, but BI does become a factor in proposals that are otherwise judged of equal intellectual merit.
This depends on the proposal. Even though NSF requires a description of broader impacts in every proposal, each is different in terms of how important broader impacts are relative to scientific merit. We fund some proposals on the basis of broader impacts alone; others have weak broader impacts, but we recommend them solely on the basis of their scientific merit.
3. Is it clear to your PIs what NSF expects of them with regard to Broader Impacts? Is it clear to you?
Yes, and Yes, and becoming clearer to the community. Even those that initially hated the idea.
No, and no. NSF has discussed, and has been advised to post, clearer guidance on this for as long as it has had these two general criteria.
I believe so now. The community seems entirely prepared to do an effective job on the most part. I have never been asked how to respond to this proposal section, but again it may be that this is easier to do for the infrastructure proposals.
It is clear to me. It is variable among PIs. Many don't recognize that their proposals have many BI because they have not read the GPG or have the mistaken notion that it is the "education component" because that is what they heard from their colleague down the hall. I fear that GEPON will help promulgate this urban legend. In general, reviewers, panelists, and POs often do a better job of teasing the BI out of a proposal than does the PI.
I think that it is much easier for us in the inside than for those on the outside. Most PIs still address broader impacts by making statements that should be placed in the intellectual merit of the proposal…for instance, many comment only on the impact of the results on the collective knowledge about the science.
I think it is clear to me, and to most PIs (and reviewers), but there are certainly differences in how the guidelines should be interpreted. For example, in a project summary, a statement to the effect that there are NO broad impacts, could be viewed as having addressed the issue – even though the desired content is not there. Most commonly, where the BI content is weak, this is because the PIs have actually focused on the broader implications of the research they are proposing to carry out. In my view, this could reflect ignorance of what is expected or misunderstanding of the guidelines. In such cases, we may have failed to reach or educate all PIs as to the expectations. However, before using such criteria to decide if a proposal should or should not be considered, we need to make sure that the guideline criteria are absolutely clear and explicit.
No, it is not very clear to the PIs, and more importantly, to the reviewers. PIs want to know how the BI proposed activities are weighted in the award decisions. They also want to know what types of activities are likely to help them get funded. Yes, it is very clear to me, but that is because I'm writing a report on the topic!
The overall goal of broader impacts is laudable, but somewhat works against the whole thrust of the competitive bent of our PIs. They are under very intense pressure to present and defend the very best research ideas and methods in their field to be at all competitive within the program solicitation. With this in mind, it is the Intellectual Merit under which most proposals are funded. I believe it is possible for a proposal with excellent research and intellectual merit, but little to no broader impacts, to be funded. I do not believe the converse is true. So de facto, the broader impacts become somewhat artificial to most of our PIs and they tend to be a final thought, which they normally pawn off to graduate students, or try to involve a local school.
No. They are confused, or they understand, but don't have the time/expertise to do anything beyond graduate and sometimes undergraduate education efforts.
No on both counts. The NSF does a lousy job of explaining what it sees as appropriate broader impacts. The lame statement that the NSF does have suggests to my PIs that it is all about student support and outreach to K-12 and the public. This situation needs to be remedied with a clear statement about broader impacts if the NSF and the NSB are serious about broader impacts. The consequence of this inattention by the NSF and the NSB has left the research community that I serve jaded about what could be a useful service to the wider science and civil communities through implementation of well-articulated broader impacts.
It is not clear to all PIs, because they think it is more difficult than it is. I have interpreted Broad Impact loosely, but I try to be consistent. I think I know what NSF wants but again, I follow my own common sense in making decisions.
It is not clear to either the PIs or us what is expected in terms of BI. PIs and reviewers don't know to what extent activities that they normally engage in, such as paper publication, professional talks, and offering data to community collaborations and on the web, count towards broader impacts.
My own feeling is that the way it currently works is for each program and community to decide what it considers acceptable BI. It is clear what broad categories the GPG states can be considered BI, but it is unclear (There also seems to be part of a section missing in the GPG (pg 39 only discusses 2 of the 5 criterion)) what constitutes a compliant response to those criterion. Different people in the community have different ideas about this, too. So a single proposal can have both reviews that praise the BI and reviews that pan them. Also our reviewers have a better handle on how to evaluate certain BI. It is easier to quantify how many female or minority participants are involved in a project. Sometimes this ignores or undervalues other aspects of the BI criteria.
There seems to be considerable confusion regarding this topic. To some, the NSF BI criteria are confused with the broader implications of the proposed science to the discipline as a whole. In effect, some of the PIs and reviewers are confusing the BI criteria with the intellectual merit. Some PIs treat the BIs as a throw-away, mentioning it in the project summary, but not mentioning it again in the proposal text. We do see that reviewers and PIs are taking the NSF BIs much more seriously and do base at least part of their review score based on the NSF criteria. In some cases, reviewers are much better at identifying the NSF BIs than the PIs. However, other reviewers appear to ignore it completely.
No, probably not. Thus the plethora of standard graduate student education stuff. I think it's clear to me, and I try to pass this along to the community.
There was a learning curve on this, but most of our PIs understand what we want now. What they worry most about is what reviewers will think. This is where most of the stress over broader impacts arises. Even when a PI calls and asks us about how to address broader impacts, we still can’t guarantee that what they write will meet reviewer expectations. This inconsistency in reviewer attitudes was picked up by our COV and led to the comments I’m attaching below.
Recommendation [by ATM UARS COV, 2005]: (a) The NSF UARS should clarify the relative weight of the two merit criteria for each proposal opportunity. This information should be disseminated to the community in an easily accessible manner so that each PI and reviewer is clearly aware of what is expected by the NSF UARS. (b) The COV recommends that the “intellectual merit” criterion generally be given greater weight than “broader impact”. The COV also recognizes that there may be opportunities where the relative weight of the “broader impact” criterion should be given greater weight
4. What do you hear from your PIs concerning the obstacles they face in attempting to satisfy the Broader Impacts requirement?
Hardly anything these days. And for large projects, the response is usually outstanding.
Nothing substantive - most of my feedback comes from panel discussions and a rare comment in an ad-hoc review.
Very little actually.
Not much. We talk about it a lot when we do outreach. It is well covered in the Regional Grants Conferences and NSF Days. I think people are getting the message, slowly. I see a profound shift in this over the last 5 years.
Our PIs seem to accept that they have to address broader impacts in some way. However, few take the time to really learn what is expected of them. Some are reluctant to go there, except in the most minimal way possible. I do not think the majority of our research PIs feel that broader impacts is taken seriously enough in the review of their proposals, so they do not try hard enough to learn about it and do not see this as an obstacle at all.
I think it is extremely valuable, although most of the individuals that we fund feel that dissemination should be mostly about communicating their work with their peers. The tide is changing but only slowly. It will not change fast unless the community of researchers starts valuing it more than they currently do, and start putting weight on that. Many scientists that we fund are actually shocked when their accomplishments receive a truly broad audience (such as interviews in the NY Times). In fact, I don’t think they think it is important enough that others would care.
I think most PIs have little problem identifying some BIs related to their work. Many are actually involved in extended or outreach education efforts, and of course nearly all train and teach students. However, I think that some may feel that they have to initiate new efforts to stand out in this arena, and most academic scientists are already feeling stretched far in meeting all the expectations placed on them. Another subgroup feels that the BI component is not particularly important, and they treat it with a degree of capriciousness. This view also has come across in some of the reviews I have seen.
They tend to feel unprepared professionally to address BI, and to think too much is expected of them in this area. Guidance on what is a reasonable level of effort and the amount of $ to ask for these efforts is needed.
Most PIs have little if any experience developing an educational and public outreach program, and are unwilling to spend the resources or time it takes to develop a truly defensible and assessment-driven EPOP. It winds up being simply an add on since it is required. Also, panels are rarely knowledgeable in pedagogy requirements or what is truly novel or testable. Therefore, most PIs simply state that they are training one or two graduate students, attempt to recruit "underrepresented groups" and have involved their undergraduate students in the development of a "special course" in the topic of the investigation.
They comment that NSF program officers don't give consistent instructions about how to deal with BI.... sometimes they get the advice that Criteria 2 is not important and should only be addressed in a superficial manner.
The PIs are frustrated that unless they institute a K-12 or public outreach program, they cannot satisfy broader impacts. Several reviewers are openly hostile to being asked to consider broader impacts in peer review and the PI receives such written feedback and thinks that they are caught in the middle and getting screwed as a result.
Not really, they just ask whether specific activities could be considered part of the broad impact.
I have also heard of departments hiring personnel specifically to help researcher answer BI requirements. Some of this is okay, if that personnel works to facilitate outreach activities. The problem arises for PIs at institutions without this help. It can also become less of an activity related to a project and a PI, thus defeating the purpose of the BI criteria. There is the belief that BI are bringing down reviews for otherwise solid research proposals, so sometimes PIs go out of their way to invent things that sound like they fit into the BI requirements, instead of simply describing the efforts they are involved in.
I've not heard of any obstacles in attempting to satisfy BI requirement. I suppose that those that do not understand them may feel at a disadvantage, but we do try to give programmatic feedback regarding whether the BIs are average, below average, excellent, etc.
I don't hear much directly, largely because I tend to be proactive and tell them to do more than the standard fare. PIs tend to put greater energy into the intellectual merit aspect, because they see this as the higher hurdle to clear. Nonetheless, they understand that BI is not simply a hoop to jump through, but that it is a genuine and credible goal to shoot for. Sorry for the mixed metaphors, but at least they're all sports!
5. GEPON conceives of a community-based mechanism for showcasing best practices and networking the Broader Impacts efforts of our scientist organizations, and in general nurturing a culture that values the sharing of the output of geoscience research with a broader audience. Of what value do you imagine a mechanism of this sort may be to our community?
I think that sort of activity could be very beneficial, especially if people are creating useful products that only need distribution, and marketing efforts to be widely used. I think that an effort like that would probably require some cultural changes, but that is possible- For some of the data base efforts I see in the community, as an example, cultural change is actually happening. Amazing!
Of great value, since we do not seem able to put out useful guidance ourselves.
I think that having some means for disseminating best practices is a good idea, but I don't think the "network" proposal is really necessary. Or, if something like this is funded it should be a part of an existing effort rather than creating something new (like an office to do just this). This to me seems like overkill, particularly in constrained budget times.
Of limited value - mostly to those who want to do some educational component and don't have a ready way to do it, but that is only 1 out 5 categories to satisfy the BI.
I think such an effort would be educational for scientists as well as the broader public who might be interested in knowing the relevance of the science funded by NSF. However, there are many layers of significance within this concept. Underlying this there may be a feeling by some that we should democratize science such that it is accessible to everyone who is interested. On the other hand, I do not believe that everyone is created with equal abilities to be an effective or successful scientist – even if they are afforded the opportunities to do so. Science to me is like art – some people are gifted in unusual ways, and they are arguably the ones most likely to make the greatest contributions. However, many people can put science to use in practical ways. Thus, it is desirable to have an informed public that appreciates the benefits of funding a cutting edge scientific effort. This is needed at all levels – from basic education, to business and government. I see BIs as covering this side of the equation - helping everyone better understand how the world works, what the problems are, and how to find solutions to them. Also, making science broadly accessible will help attract talented individuals into scientific careers.
My understanding is that GEPON is Education and Public Outreach, which is only one area of BI activity. To help the GEO community with Criterion 2, all BI areas need to be addressed. I also think a focused effort needs to be preceded by data gathered by NSF: Where are we with BI Criterion 2? How has Criterion 2 influenced awards? What is the impact of BI activities on the public? More specifically, we need to know what PIs propose to do to meet the Criterion, and compare the data with what they actually accomplished (Annual and Final Project Reporting requirements need to be re-vamped to help gather this info.), before giving them help with how to address the Criterion.
Very high value, since almost no one knows what constitutes "best practices" in this far too broadly-defined requirement.
Limited unless GEO program officers begin to apply Criterion 2 in a consistent way.
This would be very valuable – I use examples from innovative projects to suggest to other what is possible to do.
Yes. I think this would be very valuable. The mechanism would have to work with existing outreach efforts in the community, such as the SCEC E&O, etc. I think PIs would appreciate having a way to improve and extend their efforts, learn about new opportunities and resources as well as keep abreast of new efforts.
I would have to know more about it to assess it.
I think this would be very welcome.
I think this would be very helpful, but we shouldn’t mislead people into thinking that EPO is the only way for a proposal to have broader impacts. On the other hand, if GEPON is restricted only to EPO in geosciences, will it overlap with AGU activities?
If you would like to share any views concerning Broader Impacts beyond the limits of the questions above, please do so right here....
Overall, I think that what is most needed is some coordination, at many levels, to maximize the benefit of the whole spectrum of BI activities that are going on. This is a big challenge.
I do not think so. It is not only the Broader Impacts merit review criteria that are misunderstood by most people, but also those criteria that pertain to the Intellectual Merit of the proposal. I’ve seen proposals in other programs that focus only on the broader impacts because they view this as the main group of evaluation criteria (for instance in ADVANCE). NOTE THAT --- NSF is again undergoing the process of reevaluating the Merit Review process, like they did in the 90’s. It is possible that we may see those two sets of criteria be reformulated into something else at some stage in the near future.
A final thought concerns incentives for people to take up scientific careers. Over the years I have heard talk and seen efforts to recruit this or that group of students into the science “pipeline’. Very rarely have I seen as much concern regarding how to retain them in the “pipeline”. This is a complicated issue, but one that needs to be addressed if we want to keep the most talented people working as scientists.
[Muller-Parker, Gisele T.] I will be happy to share the results of our study of OCE proposals in a month or two, after OCE program officers have a chance to comment. I will be preparing an EOS article also.... Gisele
In some ways there is hostility to BI that is hard to alleviate because there are so many ways to look at it and some researchers feel like they are being forced to do things they aren't interested in. More information on what qualifies as BI for both PIs and the reviewer community could help this situation.
Appendix B: “Why Earth Science?” flyer
