Taming the data deluge with the new open source iRODS data grid system

Users can apply the growing set of existing rules or write new ones. Rules can also be developed as community-wide policies to manage data.

“One reason policy-based data management is important is that it lets communities integrate across different types of collection structures,” said Moore. “What this means is that iRODS lets one community talk to any other community independent of what data management system the other community is using. No matter which technology you pick you aren’t isolated.”

iRODS is designed to be flexible, growing seamlessly from small to very large needs.

“You can start using it as a single user who only needs to manage a small stand-alone data collection,” said Arcot Rajasekar, who leads the iRODS development team. “The same system lets you grow into a very large federated collaborative system that can span dozens of sites around the world, with hundreds or thousands of users and numerous data collections containing millions of files and petabytes of data – it’s a true full-scale distributed data system.” A petabyte is one million gigabytes, about the storage capacity of 10,000 of today’s PCs.

At SDSC alone iRODS and its predecessor SRB technology are already managing one petabyte of data and two hundred million files for 5,000 users.

“It’s an advantage that the new iRODS system is open source,” added Rajaseker. “This is bringing in collaborators from the US and as far away as France, the UK, Japan, and Australia who are contributing code, so iRODS will quickly add more features.”

“We also find that users like the open source approach and have more confidence in adopting the new technology. Open source software makes it possible to assemble a larger development team and interact with a wider range of user communities. This increases user confidence that the iRODS system will be around in the future.”

Currently the iRODS team is current working with partners to help a number of projects apply the technology, including the National Archives and Records Administration (NARA), the Ocean Observatories Initiative (OOI), the National Science Digital Library, the Temporal Dynamics of Learning Center (TDLC), the UC Humanities, Arts and Social Sciences (HASS) grid and the Testbed for the Redlining Archives of California’s Exclusionary Spaces (T-RACES) project, and numerous others.

Version 1.0 of iRODS is supported on Linux, Solaris, Macintosh, and AIX platforms, with Windows coming soon. The iRODS Metadata Catalog (iCAT) will run on either the open source PostgreSQL database (which can be installed via the iRODS install package) or Oracle. And iRODS is easy to install -- just answer a few questions and the install package automatically sets up the system.