Ingest Freeze for Islandora 7

Last Updated, May 31 2023.

The migration process is underway and below are the tasks which have been completed thus far related to the migration. We will continue to provide updates as milestones are achieved via Slack and a summary is added to this page.

Migration Progress 

Research (January 3, 2022 to May 26, 2023) ✓

100%
Review existing CTDA content, map fields to Modern Islandora, review/address any issues with mapping fields.

Build (October 1, 2022 to May 31, 2023) ✓

100%
Create new Modern Islandora servers on UConn infrastructure.

Test (May 1, 2023 to Present)

95%
Utilize small batches of CTDA content to test migration workflow, make adjustments, and prepare for larger migration.

Migrate (May 31, 2023 to Present )

Connect existing CTDA content to the new Modern Islandora infrastructure.

Review (TBA)

0%

Review the migration of content, verify settings, and configure facets and display settings. Add logos and other information for groups.

Release (TBA)

0%

Release site to CTDA members and after final review launch to public.

May 31, 2023

On Wednesday May 31, 2023, the next phase of the migration began with the original content being mapped to the new infrastructure. This process involves reindexing and remapping content from the existing CTDA infrastructure (i7) to the new series of Modern Islandora servers which have been allocated additional processors and resources to expedite the indexing process for the 3.7 million objects. This process will likely take a few weeks to complete and once complete, the next phase will be to review everything and prepare the system for production. This is a huge milestone that has been months (and years) in the making and we are very appreciative of everyone's work and patience as we move forward into the next generation of the CTDA with Modern Islandora.

May 26, 2023

As of Friday May 26, 2023, the migration process continues with additional testing, production system configuration, and general troubleshooting. Included below are items which are either in-progress or completed as of May 26, 2023

Production Systems

• Kubernetes for load balancing of the production system is being configured with documentation being developed to provide to UITS for implementation on future projects.
• CTDA Storage for the production system mapping is being planned and tested along with an allocation of 30TB of additional storage to accommodate growth in FY24.
• Multiple staging servers have been configured, additional processors added, all to facilitate the migration of the 3.7 million objects.
• Planning for future growth and expansion of the system was discussed and infrastructure is capable of scaling to well beyond a petabyte of data.
• Mirador viewer is in development and is on target with being implemented in the final production system.

Sandbox Updates

• Group theming has been temporarily disabled to enable faster loading for test data. This functionality will return after these tests are completed.
• Some configuration and performance tests may be conducted with the sandbox so users may notice changes with fields and other configurations temporarily as the system is being tested with migration content. 

May 19, 2023

As of Friday May 19, 2023, the migration process is in the metadata review, testing, and system configuration stage of the process. Included below are the items which have been completed from May 1, 2023 to May 19, 2023:

Metadata

• Review of namespaces and mapping any objects not associated with an official namespace to the correct namespace - 57 objects (completed)
• Review of datastreams which are not official datastreams and mapping to official datastreams - 44 datastreams with 739 objects (completed)
• Review of objects with more than one content model and map to correct content model - 81 objects with 173 total content models (completed)
• RELS typo/error review - 13 objects (completed)
• Empty content model review - 62 objects (completed)
• Not assigned content model review - 28 objects (completed)
• Embargo list for migration - 11,510 objects (completed)
• Scholar embargo list for migration - 411 objects (completed)
• Review of creative commons licenses to ensure valid terminology is utilized in the records to be migrated (completed)

Production System

• Configuration of on premises system (underway)
• Load balancing options for on premises system (information gathering phase)
• Storage space mapping (information gathering and testing phase)
• Adding more system resources to DEV box for migration (underway)

Sandbox Updates

• IP embargo configuration and testing (completed)
• Configuration of Sandbox synced to Production and frozen (completed)

May 1, 2023

The Ingest Freeze has begun. No new content may be added to the CTDA repository until the migration is complete in 6-8 weeks. The last elements of the migration process are being tested and test migration batches will begin shortly. The public CTDA server will continue to operate as usual until the migration is complete. 

April 27, 2023

After over a year and a half of work in preparing for the migration to the new Modern Islandora system, the time has finally come where we will begin to prepare for the migration of the content from the Islandora 7 system to the new Modern Islandora system. To prepare for this migration, an ingest freeze will need to take place starting on May 1, 2023 to prepare for migrating the content from Islandora 7 to Modern Islandora.  This process will take multiple weeks to complete as content is mapped to the new system, indexed, validated, and reviewed.

The Ingest Freeze will likely take place for 6-8 weeks. The CTDA will provide regular updates on this process including any changes to this time period.

The status of the Ingest Freeze will be updated regularly via Slack and via banners on the CTDA manage and the new CTDA sandbox sites. These updates will include both text and a visual identifier (with alt text) to indicate the status of the Ingest Freeze.

Ingest Freeze Status

As the ingest freeze progresses the CTDA logo will begin to be covered in ice and on May 1, 2023 until the ingest freeze is complete, the CTDA logo will be frozen in an ice cube and as the freeze begins to come to an end, the CTDA logo will begin to thaw as the ice melts. Once the freeze is complete the CTDA logo will return have a puddle of water under it.

FAQs about the Ingest Freeze

Below are additional details on the ingest freeze process for contributing institutions. 

• When does the Ingest Freeze start?
  The Ingest Freeze will start on Monday May 1, 2023. 
• What does an Ingest Freeze mean?
  During the Ingest Freeze no new content can be added to the Connecticut Digital Archive (CTDA) manage site (manage.ctdigitalarchive.org) and no updates/edits can be made to content already ingested.
• Why do we have to have an Ingest Freeze?
  The Ingest Freeze allows a snapshot of all data in the Connecticut Digital Archive to be taken and enables the work of mapping all the 3.5+ million objects in the CTDA to be mapped and indexed in the new system. This freeze ensures the integrity of the data is preserved and enables a faster migration process.
• What do I need to do to help with the Ingest Freeze?
  Up until Monday May 1, 2023 you can continue to edit and ingest new content via the CTDA manage site. Other than being prepared to not ingest content as of May 1, 2023 there is no additional effort needed from CTDA contributing institutions.
• How long will the Ingest Freeze take?
  The ingest freeze will likely take place for 6-8 weeks. The CTDA will provide regular updates on this process including any changes to this time period.
• Can the public discover and access content within the CTDA during the Ingest Freeze?
  Yes! The content in the CTDA will be discoverable and accessible during the Ingest Freeze and migration process through the public site https://ctdigitalarchive.org .
• How will updates on the Ingest Freeze be provided to the CTDA contributing institutions?
  We will provide updates via Slack to the CTDA community, include a banner on both the CTDA manage and the new CTDA Sandbox sites on the progress of the CTDA Ingest Freeze and migration process. 
• Can I use the new CTDA Sandbox site during the Ingest Freeze?
  Yes! The CTDA Modern Islandora Sandbox site will remain available for the CTDA contributing institutions to practice ingests, updates, and explore features that will be part of the new CTDA once the migration is complete. Keep in mind the sandbox is a place to practice and none of the content within the sandbox will be migrated to the new system. 
• What can I do during the Ingest Freeze to prepare for the new system?
  During the Ingest Freeze is a great time to familiarize yourself with the new CTDA system, review training materials, review documentation, and prepare for ingests in the new system. The CTDA Content Manager Help Center in the sandbox is being continually updated with new information, documentation, and training materials on working with content in the New CTDA. 
• Can I login to manage.ctdigitalarchive.org during the Ingest Freeze?
  No, during the Ingest Freeze manage.ctdigitalarchive.org will not be accessible for login in and will likely show a time-out or error when you go to this page. This is to ensure the data remains unchanged during the migration process.
• Can I access content via manage.ctdigitalarchive.org during the Ingest Freeze?
  No, during the Ingest Freeze manage.ctdigitalarchive.org will be offline to ensure the integrity of the data. If you need to download an object from the repository during the freeze, CTDA staff can assist, just slack us or email us at ctda@uconn.edu .
• I need a JPEG/JPG version of an image how can I get a better quality image during the Ingest Freeze?
  From the production site https://ctdigitalarchive.org navigate to the object you need to download, and in the URL string add the following /datastream/MEDIUM_SIZE to get access to the JPG or if you need the higher resolution version add /datastream/OBJ to download that object. These links are only to be used during this migration phase and are not persistent.

We thank the entire CTDA community for their patience and support during this Ingest Freeze and migration process.