Data Transfer Test Campaigns
| Campaign | Focus | Start Date | End Date/Frequency | Software/Data Resources | Infrastructure Resources | Activities |
|---|
| Network and Storage Performance Dress Rehearsal | Network and storage performance | April 1 | April 15 | Rucio task manager, various sized data sets, monitoring dashboards | | Resource | Percentage |
|---|
| Storage | 50% | | Network | 50% | | Computing | 10% |
| - Conduct full network mesh tests with data of various sizes using Rucio task manager.
- Monitor data transfer and storage performance through dashboards.
|
| Network and Storage Performance | Network and storage performance | May 1 | May 7/first week of every month | Rucio task manager, various sized data sets, monitoring dashboards | | Resource | Percentage |
|---|
| Storage | 100% | | Network | 100% | | Computing | 10% |
| - Conduct full network mesh tests with data of various sizes using Rucio task manager.
- Monitor data transfer and storage performance through dashboards.
|
| Scalability and Stress Tests | System scalability and stress tolerance | July 15 | July 31 | Predefined workload profiles with increasing data and user demands, monitoring tools | | Resource | Percentage |
|---|
| Storage | 100% | | Network | 100% | | Computing | 100% |
| - Gradually increase data volumes and user concurrency to test system capacity.
- Monitor performance metrics for resource usage, data transfer, and response times. Identify potential bottlenecks and areas for improvement.
|
| Wrap-up and Reporting | Finalize test results and prepare reports | September 1 | September 30 | Consolidated test data, reporting tools | N/A | - Consolidate results from all previous campaigns. Identify overall system performance and potential areas for improvement.
- Generate reports summarizing test results, findings, and recommendations.
|
Data Lifecycle Test campaigns
| Campaign | Focus | Start Date | End Date | Software/Data Resources | Infrastructure Resources | Activities |
|---|
| Ingestion and Dissemination | Data ingestion, replication, and transfer | April 1 | April 30 | Precursor/synthetic data, ingestion service, data transfer tools, monitoring dashboards | | Resource | Percentage |
|---|
| Storage | 50% | | Network | 50% | | Computing | 10% |
| - Ingest precursor and synthetic test data (Karabo) via the ingestion service.
- Replicate data to a designated set of sites.
- Monitor data transfer performance through dashboards.
|
| Integration, User Workflows, and Data Lifecycle | Integration, workflows, data lifecycle | June 15 | June 30 | Workloads repository, data for lifecycle testing, Jupyter Notebooks, visualization tools | | Resource | Percentage |
|---|
| Storage | 50% | | Network | 50% | | Computing | 75% |
| - Run pre-defined workflows from a common repository using public precursor or synthetic data.
- Document workflow execution details (local environments, resources, sites involved).
- Test interactive workflows involving visualization tools like VisIVO or CARTA.
- Conduct end-to-end data lifecycle tests with all v0.1 sites participating.
- Process data using Jupyter Notebooks, and visualise results.
|
| Wrap-up and Reporting | Finalize test results and prepare reports | September 1 | September 30 | Consolidated test data, reporting tools | N/A | - Consolidate results from all previous campaigns. Identify overall system performance and potential areas for improvement.
- Generate reports summarizing test results, findings, and recommendations.
|
Miro view (see comment), snapshot taken 23 Jan 2025, please delete this image if the plans change. (and consider editing the miro and updating here)
1 Comment
Bolton, Rosie
I made a miro view of this table, for presentation purposes: https://miro.com/app/board/uXjVLrIQbrM=/, password srcnet0.1test