RFCs for HDF5 and HDF-EOS5
Status Update
Richard Ullman
Chair ES-DSWG - Standards
November 29, 2006
Overview
•
What is the NASA Earth Science Data System Working Group,
Standards Process Group?
•
•
HDF in the SPG
Invit...
Motivation
•
One initiative after another has stressed the need for interoperability
standards.
•
Many standards initi...
EOSDIS Evolution 2015 Vision Tenets
Vision Tenet
Archive Management
EOS Data
Interoperability
Future Data Access and
Pro...
A Shared Vision for Earth Observation
Articulated by 34 Nations in an Earth Observation Summit (July 31, 2003)
An interna...
Insights from “SEEDS” Analysis
•
Interoperability does not require homogeneous systems, but rather
coordination at the i...
The Request For Comment Process
•
•
•
Modeled after example of Internet “IETF RFC”.
•
The Standards Process Group forms...
The Review Process
Initial Screening
Initial review of the RFC
Provide RFC submission support
Form TWG; set schedule
R...
The Endorsement Process
•
SPG will send recommended standards to NASA HQ Program
Executive for Data Systems with the fol...
RFC criteria:
•
Are there components (technologies practices) that if documented and
more widely used would promote:
–
...
HDF5 and HDF-EOS RFCs
in the NASA SPG Process
•
•
•
•
Related RFCs handled by same TWG
One review request for both RFCs
...
HDF RFC “Implementation” Reviewers
•
•
Total of 17 people commented on one or both RFCs
Backgrounds include:
–
–
–
Data...
Summary of comments from
Implementation Review phase
•
HDF5 comments from 10 reviewers were generally positive
–
•
HD...
Other Comments Received
•
Comments on HDF in general from 4 reviewers
–
–
General comments, not addressing implementat...
Summary of Comments So Far
from Usability Survey
•
•
Total of 13 people have commented on HDF5
•
•
5 are international...
of 15

RFCs for HDF5 and HDF-EOS5 Status Update

Published on: Mar 3, 2016
Published in: Technology      
Source: www.slideshare.net


Transcripts - RFCs for HDF5 and HDF-EOS5 Status Update

  • 1. RFCs for HDF5 and HDF-EOS5 Status Update Richard Ullman Chair ES-DSWG - Standards November 29, 2006
  • 2. Overview • What is the NASA Earth Science Data System Working Group, Standards Process Group? • • HDF in the SPG Invitation to comment.
  • 3. Motivation • One initiative after another has stressed the need for interoperability standards. • Many standards initiatives, both formal and grass roots have put forward specifications or demonstrated various ways to enable access to data. • NASA, or NASA funded projects are often in the forefront of these activities. • However, NASA participation in a standards development activity does not imply that NASA projects endorse the results of that activity. • Need a way to identify the “standards that work” in the context of NASA’s Earth science research and applications data systems under the operational loads that NASA typically experience.
  • 4. EOSDIS Evolution 2015 Vision Tenets Vision Tenet Archive Management EOS Data Interoperability Future Data Access and Processing Data Pedigree Cost Control User Community Support IT Currency Vision 2015 Goals  NASA will ensure safe stewardship of the data through its lifetime.  The EOS archive holdings are regularly peer reviewed for scientific merit.  Multiple data and metadata streams can be seamlessly combined.  Research and value added communities use EOS data interoperably with other relevant data and systems.  Processing and data are mobile.  Data access latency is no longer an impediment.  Physical location of data storage is irrelevant.  Finding data is based on common search engines.  Services invoked by machine-machine interfaces.  Custom processing provides only the data needed, the way needed.  Open interfaces and best practice standard protocols universally employed.  Mechanisms to collect and preserve the pedigree of derived data products are readily available.  Data systems evolve into components that allow a fine-grained control over cost drivers.  Expert knowledge is readily accessible to enable researchers to understand and use the data.  Community feedback directly to those responsible for a given system element.  Access to all EOS data through services at least as rich as any contemporary science information system.
  • 5. A Shared Vision for Earth Observation Articulated by 34 Nations in an Earth Observation Summit (July 31, 2003) An international comprehensive, coordinated and sustained Earth observation system • Comprehensive: meeting the needs of a variety of science and applications disciplines • Coordinated: multinational satellite, suborbital and in situ observing capabilities strategically coordinated via agreed standards and data exchange • Sustained: long-term, continued financial and in-kind support from funding authorities Group on Earth bservations
  • 6. Insights from “SEEDS” Analysis • Interoperability does not require homogeneous systems, but rather coordination at the interfaces. • Management can judge success based upon program goals rather than dictated solutions. – • • example: degree of interoperability rather than use of particular data format. Communities of practice have solutions. Published practices that demonstrate benefit can grow … – – – successful practice in specific community broader community adoption community-recognized “standards”
  • 7. The Request For Comment Process • • • Modeled after example of Internet “IETF RFC”. • The Standards Process Group forms a Technical Working Group (TWG) to coordinate evaluation. Tailored for responsiveness to NASA. Proposed standards are documented as specifications acording to SPG guidelines and submitted by practitioners within the NASA community. – What does “implementation” of this specification mean in the context of NASA Earth Science Data Systems? – What constitutes successful “operational” experience? • The community is invited by means of email announcement to comment on the specification and particularly to address questions formulated by the TWG. • The TWG also identifies key stakeholders that are likely to have particular experience with the technology and solicits their opinion. • The TWG reports to the SPG and the SPG makes recommendations for final status of the RFC.
  • 8. The Review Process Initial Screening Initial review of the RFC Provide RFC submission support Form TWG; set schedule RFC Proposed Standard Stakeholder/TWG Review Stakeholders Respond to TWG TWG Develop review criteria Evaluate known experiences Evaluate community response Specification Operational Readiness Suitability for Use SPG Recommendation Recommended Standard
  • 9. The Endorsement Process • SPG will send recommended standards to NASA HQ Program Executive for Data Systems with the following: – – Strengths/ Weaknesses Applicability/ Limitations • • Endorsement is briefed to HQ Earth Science Steering Committee • “Core” standards will be required of NASA Earth Science programs, projects and awards. • “Community” standards will be applied at discretion of program or studies managers. HQ will disseminate endorsement through NASA CIOs and to general announcement email to community
  • 10. RFC criteria: • Are there components (technologies practices) that if documented and more widely used would promote: – – – – – • Easier sharing or exchanging of data among distributed partners and users. Reducing the cost of developing or maintaining a system. Distributed systems development and sharing of software and technical expertise. Increasing the use of scientific data products and bringing more funding. Interoperability and enhancing innovation, collaboration, and computing performance. For identified technologies/practices, Is there a community of use that: – – Has experience in implementation and operation . Has leadership necessary to promote the advantage of wider use.
  • 11. HDF5 and HDF-EOS RFCs in the NASA SPG Process • • • • Related RFCs handled by same TWG One review request for both RFCs Community members asked to comment on either or both of the RFCs Target community includes – – – – • NASA instrument teams NASA data systems managers Earth science investigators Developers of software tools for Earth science data Comment period for “implementation experience” April 26 – July 31
  • 12. HDF RFC “Implementation” Reviewers • • Total of 17 people commented on one or both RFCs Backgrounds include: – – – Data providers (AIRS, HIRDLS and MLS instrument teams) – Earth science data users (CERES instrument team, DSCOVR project) NASA data systems managers (ORNL and NSIDC DAACs, TSDIS/GPM, OMI SIPS ) Developers of software tools for Earth science data (RSS – developers of IDL, UAH Subsetting team, Unidata, MathWorks, NetCDF, HDF Group (reviewing HDF-EOS5), HDF-EOS team (reviewing HDF5)) • All using HDF and/or HDF-EOS software from original developers, perhaps with minor modifications • Tools used include FORTRAN, IDL, Mathematica, Scilab, HDFLook, HDFView, Matlab • Operating systems include Windows (32 and 64 bit), Linux (32 and 64), Solaris (32 and 64), Mac (PPC & Intel), SGI, IRIX, HP-UX
  • 13. Summary of comments from Implementation Review phase • HDF5 comments from 10 reviewers were generally positive – • HDF-EOS5 comments from 8 reviewers were somewhat mixed, still overall positive – – – • 9 of 10 recommending endorsement 4 recommending endorsement 2 neutral 2 recommending improvements to RFC before endorsement Primary concerns: – – – Complexity of data model, API and software libraries Availability of tools to improve ease of use Long term support
  • 14. Other Comments Received • Comments on HDF in general from 4 reviewers – – General comments, not addressing implementation questions Not specific to either flavor of HDF being considered • Maybe these should be considered for “operations” rather than “implementation” phase • 3 of 4 of these comments were negative – Both Science Data Users fell in this group
  • 15. Summary of Comments So Far from Usability Survey • • Total of 13 people have commented on HDF5 • • 5 are international • Should HDF be a recommended format for NASA Earth Science data? 12 are software tools developers (among other roles); 1 is a data user only; 7 classify themselves as both US respondents from military, national labs, high performance computing, NOAA, commercial companies – – – – 10 selected “strongly agree” 1 selected “disagree” 1 selected “neutral” 1 selected “not applicable”

Related Documents