OCP_2_5_Group_3

DSSD_25_Composite_Temperature_Variation_Test_Case.py

The test involves the comparison of the Composite Temperature reported on 2 separate drives from the same supplier, model, capacity, FW revision, and under identical workload conditions in the same slot to ensure a fair apples-to-apples comparison.

The objectives of this test case are to verify the following:

  1. Device to device composite temperature variation shall be +/- 1 C under the same environmental conditions, slot location and workload.

  2. Single device.s composite temperature variation shall not vary by more than +/- 1C once it is in steady state under the same environmental condition.

Note that environmental thermal conditions must be controlled by the supplier when testing.

NOTE: The user is required to run this script from the CLI as Yes/No questions will need to be answered

Pass/Fail: Script passes if all tests pass

Open Compute Project Datacenter NVMe SSD Spec v2.5 requirements referenced by this script:

TRPT-1/2/3/4

add_error_silent()

Silently increment the error count for the current test

Add 1 to the count in the errors file in the rest tree. This number is not reset between passes.

Returns:

Current total errors after incrementing

Return type:

int

DSSD_25_Data_Integrity_Test_Case.py

This script will run the following tests:

  1. Issue Identify Controller and verify it’s successful

  2. Check RTD3 Entry Latency field value

  3. Issue a Sequential Write IO for 128kiB, QD=128

  4. Issue Sequential Read IO for 128kiB, QD=128

  5. Execute normal shutdown process and verify CC.SHN and CSTS.SHST values

  6. Execute abrupt shutdown process and verify CC.SHN and CSTS.SHST values

Pass/Fail: Script passes if all tests pass

Open Compute Project Datacenter NVMe SSD Spec v2.5 requirements referenced by this script:

PLP-1

check_performance_logs(dut, perf_metrics, io_bytes, shutdown_type, no_path_timeout)

Checks the performance log using the perf_metrics dict to determine if a “Performance Not Met” warning needs to be issued

Parameters:

  • dut (XML_API) – Device object under test

  • perf_metrics (list | dict) – Performance metrics of Read and Write

  • io_bytes (dict) – dict for read/write bytes

  • shutdown_type (int) – Original Abrupt value

  • no_path_timeout (int) – Original no path timeout value

Return type:

None

collect_performance_logs(dut, io_test, perf_field)

Collects logs from the ADP and prints the performance logs for a given IO command

Parameters:

  • dut (XML_API) – Device object under test

  • io_test (dict) – Custom IO dict

  • perf_field (str | list) – Performance field to be collected. Either “TestReadBytes” | “TestWriteBytes” | “TestLatencyQoS” | “TestWriteIOs” | “TestReadIOs”

Returns:

perf_val_dict – Parsed QoS latency values or dict for Number of read/write bytes

Return type:

dict

execute_IO_Profile(dut, test_type, io_profile, io_runtime=-1, io_pattern=17)
Executes each of the IO Profiles listed in Table 4

  1. Sequential Read (MiB/s) (128kiB, QD = 128)

  2. Sequential Write (MiB/s) (128kiB, QD = 128)

  3. Random Read (KIOPS) (4kiB, QD = 128)

  4. Random Writes (KIOPS) (4kiB, QD = 128)

  5. Random 70R:30W mix (KIOPS) (4kiB, QD = 128)

Parameters:

  • dut (XML_API) – Device object under test

  • test_type (str) – Either ‘Sequential’ | ‘Random’

  • io_profile (str) – IO Profile to be executed

  • io_runtime (int) – Time in seconds for IO to execute

  • io_pattern (int) – Pattern for io to be executed

Return type:

dict

get_runtime_seconds(dut, test_id)

Get the starting or ending timestamp and format it for use in sb_filter

Parameters:

  • dut (XML_API) – Device object under test

  • test_id (str) – I/O testname

Returns:

timestamp – Timestamp formatted in seconds

Return type:

int

restore_orig_abrupt_value(dut, shutdown_type, no_path_timeout)

Restore original value of Abrupt shutdown

Parameters:

  • dut (XML_API) – Device object under test

  • shutdown_type (int) – Original Abrupt value

  • no_path_timeout (int) – Original no path timeout value

Return type:

None

verify_ios(io_tests)

Verifies the succession of an I/O command

Parameters:

io_tests (list) – list of I/O Command dicts

Returns:

io_failed – Returns failed if any io test failed

Return type:

bool

DSSD_25_Device_Panic_Type_10_No_More_NAND_Spares_Read_Only_Mode_Test_Case.py

This script will run the following tests:

  1. Perform sequential write/read for five minutes

  2. Inject Panic Type 10 (No More NAND Spares)

  3. Detect injected Panic Type 10 (No More NAND Spares)

  4. Read log page data

  5. Execute the workflow to attempt recovery

  6. Determine whether the workflow has resolved the injected panic

  7. Verify that error injection is disabled, and device is no longer in an injected panic state

  8. Perform final I/O performance test

Pass/Fail: Script passes if all tests pass

Open Compute Project Datacenter NVMe SSD Spec v2.5 requirements referenced by this script:

CRASH-1/2/3/4

DSSD_25_Device_Panic_Type_1_CPU_Controller_Hang_Test_Case.py

This script will run the following tests:

  1. Perform sequential write/read for five minutes

  2. Inject Panic Type 1 (CPU/Controller Hang)

  3. Detect injected Panic Type 1 (CPU/Controller Hang)

  4. Execute the workflow to attempt recovery

  5. Verify whether the workflow has resolved the injected panic

  6. Verify that error injection is disabled, and device is no longer in an injected panic state

Pass/Fail: Script passes if all tests pass

Open Compute Project Datacenter NVMe SSD Spec v2.5 requirements referenced by this script:

CRASH-1/2/3/4

DSSD_25_Device_Panic_Type_2_NAND_Hang_Test_Case.py

This script will run the following tests:

  1. Perform sequential write/read for five minutes

  2. Inject Panic Type 2 (NAND Hang)

  3. Detect injected Panic Type 2 (NAND Hang)

  4. Execute the workflow to attempt recovery

  5. Determine whether the workflow has resolved the injected panic

  6. Verify that error injection is disabled, and device is no longer in an injected panic state

  7. Perform final I/O test

Pass/Fail: Script passes if all tests pass

Open Compute Project Datacenter NVMe SSD Spec v2.5 requirements referenced by this script:

CRASH-1/2/3/4

check_performance_logs(perf_metrics, io_bytes, io_type)

Checks the performance log using the perf_metrics dict to determine if a “Performance Not Met” warning needs to be issued

Parameters:

  • perf_metrics (list) – Performance metrics of Read and Write

  • io_bytes (float) – Either the read or write bytes to be compared to io metric

  • io_type (int) – 0: Read, 1: Write

Return type:

None

collect_performance_logs(dut, seq_read, seq_write)

Collects and prints the performance logs for both a Read and a Write command

Parameters:

  • dut (XML_API) – Device object under test

  • seq_read (dict) – Custom Read IO dict

  • seq_write (dict) – Custom Write IO dict

Returns:

  • read_bytes (int) – Number of bytes read

  • write_bytes (int) – Number of bytes written

execute_IO_Profile(dut, test_type, io_profile, io_runtime=-1, io_pattern=0)
Executes each of the IO Profiles listed in Table 4

  1. Sequential Read (MiB/s) (128kiB, QD = 128)

  2. Sequential Write (MiB/s) (128kiB, QD = 128)

  3. Random Read (KIOPS) (4kiB, QD = 128)

  4. Random Writes (KIOPS) (4kiB, QD = 128)

  5. Random 70R:30W mix (KIOPS) (4kiB, QD = 128)

Parameters:

  • dut (XML_API) – Device object under test

  • test_type (str) – Either ‘Sequential’ | ‘Random’

  • io_profile (str) – IO Profile to be executed

  • io_runtime (int) – Time in seconds for IO to execute

  • io_pattern (int) – Pattern for io to be executed

Return type:

dict

run_read_and_write_IOs(dut, verify_step_num, io_runtime=-1, io_pattern=0)

Executes the Sequential Read and Write for 128kiB, QD = 128 from Table 4, simultaneously

Parameters:

  • dut (XML_API) – Device object under test

  • io_runtime (int) – Time in seconds for IO to execute

  • io_pattern (int) – Pattern for io to be executed

  • verify_step_num (int) – Step number for the Verification of both Read and Write IOs

Return type:

List of dicts

DSSD_25_Device_Panic_Type_3_PLP_Defect_Read_Only_Mode_Test_Case.py

This script will run the following tests:

  1. Perform sequential write/read for five minutes

  2. Inject Panic Type 3 (PLP Defect)

  3. Detect injected Panic Type 3 (PLP Defect)

  4. Read log page data

  5. Execute the workflow to attempt recovery

  6. Determine whether the workflow has resolved the injected panic

  7. Verify that error injection is disabled, and device is no longer in an injected panic state

  8. Perform final I/O performance test

Pass/Fail: Script passes if all tests pass

Open Compute Project Datacenter NVMe SSD Spec v2.5 requirements referenced by this script:

CRASH-1/2/3/4

DSSD_25_Device_Panic_Type_4_Logical_Firmware_Error_Test_Case.py

This script will run the following tests:

  1. Perform sequential write/read for five minutes

  2. Inject Panic Type 4 (Logical Firmware Error)

  3. Detect injected Panic Type 4 (Logical Firmware Error)

  4. Execute the workflow to attempt recovery

  5. Determine whether the workflow has resolved the injected panic

  6. Verify that error injection is disabled, and device is no longer in an injected panic state

  7. Perform final I/O performance test

Pass/Fail: Script passes if all tests pass

Open Compute Project Datacenter NVMe SSD Spec v2.5 requirements referenced by this script:

CRASH-1/2/3/4

DSSD_25_Device_Panic_Type_5_DRAM_Corruption_Critical_Path_Test_Case.py

This script will run the following tests:

  1. Perform sequential write/read for five minutes

  2. Inject Panic Type 5 (DRAM Corruption Critical Path)

  3. Detect injected Panic Type 5 (DRAM Corruption Critical Path)

  4. Execute the workflow to attempt recovery

  5. Determine whether the workflow has resolved the injected panic

  6. Verify that error injection is disabled, and device is no longer in an injected panic state

  7. Perform final I/O performance test

Pass/Fail: Script passes if all tests pass

Open Compute Project Datacenter NVMe SSD Spec v2.5 requirements referenced by this script:

CRASH-1/2/3/4

DSSD_25_Device_Panic_Type_6_DRAM_Corruption_Non_Critical_Path_Test_Case.py

This script will run the following tests:

  1. Perform sequential write/read for five minutes

  2. Inject Panic Type 6 (DRAM Corruption Non-Critical Path)

  3. Detect injected Panic Type 6 (DRAM Corruption Non-Critical Path)

  4. Execute the workflow to attempt recovery

  5. Determine whether the workflow has resolved the injected panic

  6. Verify that error injection is disabled, and device is no longer in an injected panic state

  7. Perform final I/O performance test

Pass/Fail: Script passes if all tests pass

Open Compute Project Datacenter NVMe SSD Spec v2.5 requirements referenced by this script:

CRASH-1/2/3/4

DSSD_25_Device_Panic_Type_7_NAND_Corruption_Test_Case.py

This script will run the following tests:

  1. Perform sequential write/read for five minutes

  2. Inject Panic Type 7 (NAND Corruption)

  3. Detect injected Panic Type 7 (NAND Corruption)

  4. Execute the workflow to attempt recovery

  5. Determine whether the workflow has resolved the injected panic

  6. Verify that error injection is disabled, and device is no longer in an injected panic state

  7. Perform final I/O performance test

Pass/Fail: Script passes if all tests pass

Open Compute Project Datacenter NVMe SSD Spec v2.5 requirements referenced by this script:

CRASH-1/2/3/4

DSSD_25_Device_Panic_Type_8_SRAM_Corruption_Test_Case.py

This script will run the following tests:

  1. Perform sequential write/read for five minutes

  2. Inject Panic Type 8 (SRAM Corruption)

  3. Detect injected Panic Type 8 (SRAM Corruption)

  4. Execute the workflow to attempt recovery

  5. Determine whether the workflow has resolved the injected panic

  6. Verify that error injection is disabled, and device is no longer in an injected panic state

  7. Perform final I/O performance test

Pass/Fail: Script passes if all tests pass

Open Compute Project Datacenter NVMe SSD Spec v2.5 requirements referenced by this script:

CRASH-1/2/3/4

DSSD_25_Device_Panic_Type_9_HW_Malfunction_Test_Case.py

This script will run the following tests:

  1. Perform sequential write/read for five minutes

  2. Inject Panic Type 9 (HW Malfunction)

  3. Detect injected Panic Type 9 (HW Malfunction)

  4. Execute the workflow to attempt recovery

  5. Determine whether the workflow has resolved the injected panic

  6. Verify that error injection is disabled, and device is no longer in an injected panic state

  7. Perform final I/O performance test

Pass/Fail: Script passes if all tests pass

Open Compute Project Datacenter NVMe SSD Spec v2.5 requirements referenced by this script:

CRASH-1/2/3/4

DSSD_25_Device_Stability_And_Data_Integrity_Test_Case.py

This script will run the following tests:

  1. Precondition the namespace twice with 128KiB sequential writes

  2. Perform sequential reads and writes on the device for 5 minutes

  3. Do a normal shutdown on the device

  4. Power device back on

  5. Perform sequential reads and verify no there’s no data corruption

  6. Perform sequential reads and writes on the device for 5 minutes and check performance

  7. Do an abrupt shutdown on the device

  8. Repeat steps 4-6

  9. Start a sequential write test

  10. Do an ungraceful shutdown on the device

  11. Repeat steps 4-6

Pass/Fail: Script passes if all tests pass

Open Compute Project Datacenter NVMe SSD Spec v2.5 requirements referenced by this script:

TTR-7/8/12/15/16

check_performance_logs(perf_metrics, io_bytes, io_type)

Checks the performance log using the perf_metrics dict to determine if a “Performance Not Met” warning needs to be issued

Parameters:

  • perf_metrics (list) – Performance metrics of read and write expected throughput

  • io_bytes (float) – Either the read or write bytes to be compared to I/O metric

  • io_type (str) – Read, Write

Returns:

performance_met – If measured performance was less than 5% below the performance requirement

Return type:

boolean

collect_performance_logs(dut, seq_read, seq_write, io_runtime)

Collects and prints the performance logs for reads and writes in an I/O test

Parameters:

  • dut (XML_API) – Device object under test

  • seq_read (dict) – Custom read I/O test dict

  • seq_write (dict) – Custom write I/O test dict

  • io_runtime (int) – Time in seconds for IO to execute

Returns:

  • read_bytes (int) – Number of bytes read

  • write_bytes (int) – Number of bytes written

DSSD_25_Garbage_Collection_Test_Case.py

The objectives of this test case are as follows:

  1. Verify that the device supports Garbage Collection during periods of no I/O (Idle GC).

  2. Verify that Read latency does not change more than -5% from baseline when the host is issuing Dataset Management commands with AD = 1.

  3. Verify that Read latency does not change more than -5% from baseline after the device has been performing Idle GC.

Open Compute Project Datacenter NVMe SSD Spec v2.5 requirements referenced by this script:

TRIM-5/6/7

collect_performance_logs(test_id)

Collects logs from the ADP and prints the performance logs for a given IO command

compare_qos_performance_logs(test_id, performance_logs, baseline_qos)

Determine if the measured QoS values are within 5% of the baseline QoS values.

precondition(dut, INDIRECT_UNIT_SIZE)

Precondition the DUT by performing 1x128KiB sequential writes and 2x(Indirection Unit Size)KiB random writes.

DSSD_25_IO_Commands_Test_Case.py

This script will run the following tests:

  1. Verify Read/Write/Flush commands are successful

  2. Verify random write I/O performance with FUA=0 and FUA=1 is within 5% of each other

  3. Execute Flush during random write I/O and verify both pass

  4. Do random write/read I/O with FUA=0 for both; then do random write/read I/O with FUA=0 for write and FUA=1 for read, and verify performance is within 5% of each other

  5. If supported, disable volatile write cache, do random write I/O, and verify performance is within 5% of the value in step 2

  6. If supported, enable volatile write cache and verify it was successful

Pass/Fail: Script passes if all tests pass

Open Compute Project Datacenter NVMe SSD Spec v2.5 requirements referenced by this script:

NVMe-IO-1/3/4/5/6/7/8/9/10/11/12/13/14, GETF-2

collect_performance_logs(dut, io_test, perf_field, runtime_io)

Collects logs from the ADP and prints the performance logs for a given IO command

Parameters:

  • dut (XML_API) – Device object under test

  • io_test (dict) – Custom IO dict

  • perf_field (str | list) – Performance field to be collected. Either “TestReadBytes” | “TestWriteBytes” | “TestLatencyQoS” | “TestWriteIOs” | “TestReadIOs”

  • runtime_io (int) – Length of time I/O test ran for

Returns:

perf_val_dict – Parsed QoS latency values or dict for Number of read/write bytes

Return type:

dict

compare_performance_logs(base_io_bytes, other_io_bytes, compare_msg)

Checks the performance log using the perf_metrics dict to determine if a “Performance Not Met” warning needs to be issued

Parameters:

  • base_io_bytes (dict) – baseline values, dict of either the read, write or flush bytes to be compared to io metric

  • other_io_bytes (dict) – actual values from I/O, dict of either the read, write or flush bytes to be compared to io metric

  • compare_msg (List) – List tuple of two strings respectively associated with the base_io and other_io to print the params run with that io, such as [(‘with FUA’, ‘without FUA’)]

Return type:

None

get_test_timestamp(dut, test_id, timestamp_type)

Get the starting or ending timestamp and format it for use in sb_filter

Parameters:

  • dut (XML_API) – Device object under test

  • test_id (str) – I/O testname

  • timestamp_type (str) – “StartTime” or “EndTime”

Returns:

timestamp – Timestamp formatted in seconds

Return type:

int

issue_io(dut, seektype, io_profile, threads, blocks, iosperpass=0, io_runtime=-1, io_pattern=0, extra_config=None)

Executes an IO Test

Parameters:

  • dut (XML_API) – Device object under test

  • seektype (str) – Either ‘Sequential’ | ‘Random’

  • io_profile (str) – IO Profile to be executed

  • threads (int | str) – Threads (QD) to execute IO on

  • blocks (int | str) – Number of Logical Blocks

  • iosperpass (int | str) – Param to determine what the IO does on the drive ‘0’: unlimited, ‘-1’: Iterate once over drive, ‘-2’: Iterate twice over drive

  • io_runtime (int) – Time in seconds for IO to execute

  • io_pattern (int) – Pattern for io to be executed

  • extra_config (dict | None) – Optional parameter for running IO test with additional configurations

Return type:

dict

verify_ios(io_tests, is_verify_step=True)

Verifies the successful execution and wait for the completion of an I/O command

Parameters:

  • io_tests (list) – list of I/O test dicts

  • is_verify_step (bool) – Determines whether to execute resting action in method or not (Special case for if the IO verify logging comes in a later step)

Returns:

  • result_msg (tuple(func, msg) | none) – msg for later fail_action, or none if passed

  • bool – boolean to determine whether any I/O test failed

DSSD_25_IO_Latency_And_Performance_Test_Case.py

This script will run the following tests:

  1. Issue Get-Feature FID=C6h for PLP Health Check Interval data

  2. Issue Set-Feature FID=C6h to set PLP Health Check Interval data for the following values: 05h, 19h, 0Fh

  3. Issue both a Read and Write IO at 128kiB, QD=128, for 5 minutes

  4. Compare IO tests performance to the performance logs in the ADP

Pass/Fail: Script passes if all tests pass

Open Compute Project Datacenter NVMe SSD Spec v2.5 requirements referenced by this script:

PLP-2/7

check_performance_logs(perf_metrics, io_bytes)

Checks the performance log using the perf_metrics dict to determine if a “Performance Not Met” warning needs to be issued

Parameters:

  • perf_metrics (list | dict) – Performance metrics of Read and Write

  • io_bytes (dict) – dict for read/write bytes

Return type:

None

collect_performance_logs(dut, io_test, perf_field)

Collects logs from the ADP and prints the performance logs for a given IO command For a Sequential IO: MiB/s, for a Random IO: IOPS

Parameters:

  • dut (XML_API) – Device object under test

  • io_test (dict) – Custom IO dict

  • perf_field (str | list) – Performance field to be collected. Either “TestReadBytes” | “TestWriteBytes” | “TestLatencyQoS” | “TestWriteIOs” | “TestReadIOs”

Returns:

perf_val_dict – Parsed QoS latency values or dict for Number of read/write bytes

Return type:

dict

execute_IO_Profile(dut, test_type, io_profile, io_runtime=-1, io_pattern=17)
Executes each of the IO Profiles listed in Table 4

  1. Sequential Read (MiB/s) (128kiB, QD = 128)

  2. Sequential Write (MiB/s) (128kiB, QD = 128)

  3. Random Read (KIOPS) (4kiB, QD = 128)

  4. Random Writes (KIOPS) (4kiB, QD = 128)

  5. Random 70R:30W mix (KIOPS) (4kiB, QD = 128)

Parameters:

  • dut (XML_API) – Device object under test

  • test_type (str) – Either ‘Sequential’ | ‘Random’

  • io_profile (str) – IO Profile to be executed

  • io_runtime (int) – Time in seconds for IO to execute

  • io_pattern (int) – Pattern for io to be executed

Return type:

dict

get_runtime_seconds(dut, test_id)

Get the starting or ending timestamp and format in seconds

Parameters:

  • dut (XML_API) – Device object under test

  • test_id (str) – I/O testname

Returns:

timestamp – Timestamp formatted in seconds

Return type:

int

verify_ios(io_tests)

Verifies the succession of an I/O command

Parameters:

io_tests (list) – list of I/O Command dicts

Returns:

io_failed – Returns failed if any io test failed

Return type:

bool

DSSD_25_Precondition_Drive.py

This script will run the following tests:

  1. Precondition the entire drive once with 128KiB sequential writes

  2. Precondition the entire drive twice with INDIRECT_UNIT_SIZE_BYTES random writes

Pass/Fail: Script passes if all tests pass

Open Compute Project Datacenter NVMe SSD Spec v2.5 requirements referenced by this script:

N/A

get_current_queue_depth(dut)

Get the current queue depth on the controller

Parameters:

dut (XML_API) – Device object under test

Returns:

Current queue depth

Return type:

int

set_queue_depth(dut, value)

Set the queue depth on the controller

Parameters:

  • dut (XML_API) – Device object under test

  • value (int) – Queue depth to set

Return type:

None

DSSD_25_Temperature_Sensors_And_Composite_Temperature_Calculation_Test_Case.py

The objectives of this test case are to verify the following:

  1. That the device supports the number of sensors indicated by the vendor parameter data.

  2. That the equation, settings, thresholds and artifacts provided by the vendor, and the raw sensor data, can be used to calculate a composite temperature value that matches the value reported in the SMART Log data.

NOTE: The user is required to run this script from the CLI as Yes/No questions will need to be answered

Pass/Fail: Script passes if all tests pass

Open Compute Project Datacenter NVMe SSD Spec v2.5 requirements referenced by this script:

TRPT-1 to TRPT-4

get_user_input_value(question)

Ask user a question and wait for input, and verify it’s an integer or float value

Parameters:

question (str) – Question to ask the user

Returns:

input_value – Value entered by the user

Return type:

int or float