JUSTIFICATION FOR REAL WORLD TESTING APPROACH

Consistent with the ONC’s recommendation that “Real World Testing verify that deployed Certified Health IT continues to perform as intended by conducting and measuring observations of interoperability and data exchange,” this test plan focuses on capturing and documenting the number of instances that certified capability is successfully utilized in the real world. In instances where no evidence exists due to zero adoption of a certified capability or the inability to capture evidence of successful use for other reasons, we will demonstrate the required certified capability in a semi-controlled setting as close to a “real world” implementation as possible.

It is important to note that Real World Testing is only one component of the Health IT Certification program used to demonstrate compliance with the program requirements. Real World Testing should augment and support testing that was conducted prior to certification being granted. It is not intended to duplicate the methods or results previously demonstrated. Instead, this test plan was developed to demonstrate that the certified capabilities have been successfully deployed for providers to use at their discretion in live settings.

We are using a 2-fold approach to demonstrate successful real-world implementations.

STANDARDS UPDATES

(INCLUDING STANDARDS VERSION ADVANCEMENT PROCESS-SVAP AND USCDI)

MedInformatix has not updated MedInformatix EHR to any new standards as part of SVAP or the Cures Update criteria as of this date.

CARE SETTINGS

MedInformatix EHR specializes in Radiology settings and is marketed primarily to Radiology. Our secondary market are multiple ambulatory specialties. Even though our primary sales focus is Radiology, our EHR is successfully used by any ambulatory clinician, and approximately 50% of our user base are non-radiology ambulatory specialties. Our solutions are comprised of Certified EHR/PHR, RIS, Revenue Cycle Management, and Enterprise Practice Management.

While the clinical workflow may be different between different specialty providers, use of the certified criteria is the same. 

MEASURES USED IN OVERALL APPROACH

For each measurement/metric, describe the elements below:

• ❯ Description of the measurement/metric
• ❯ Associated certification criteria
• ❯ Care setting(s) that are addressed
• ❯ Justification for selected measurement/metric
• ❯ Expected Outcomes

ADOPTION RATES

The following metrics are applicable to all criteria and all care settings. These metrics will not be used directly to demonstrate interoperability or conformance to certification criteria. Instead, they will primarily be used to help determine the participants that will be in scope for this evaluation. They can also aid with the justification for other metrics by providing additional context (i.e., extremely low adoption rates for certain certified capabilities will necessitate a different approach to testing).

The following metrics are applicable to all criteria that are licensed separately from the base license and all care settings.

SUMMATIVE ASSESSMENT METRICS

The following metrics will be measured by viewing audit logs and reporting systems available to track the behavior of the certified Health IT module during a given time frame. All metrics are designed to reflect the core elements of the criteria, demonstrate interoperability, and demonstrate the success rate of the certified capability being used. In most cases, we elected to record these metrics over a 90-day period to reflect the reporting periods typically required for compliance with the federal incentive programs.

The continued measurable use of certified capabilities will provide implicit evidence of successful implementation of the required certified capability. This is especially meaningful when demonstrating interoperability with outside systems. When it is impossible to determine “success” via an explicit confirmation by a receiving system, success is defined as a transmission made where no error was received from the destination system or its intermediaries. Additionally, we will review internal customer and vendor issue tracking systems for reports of failures or unsatisfactory performance in the field.

None of the following criteria were updated to the Cures Update version of the criteria prior to August 31, 2021. As a result, all testing is scheduled to be conducted against the 2015 Edition version of the criteria.

SCHEDULE OF KEY MILESTONES

Real World test planning will commence in first quarter of 2022. We expect each phase to take 90 days to complete, with report writing to occur end of 2022/early 2023.

ATTESTATION

This Real World Testing plan is complete with all required elements, including measures that address all certification criteria and care settings. All information in this plan is up to date and fully addresses the Health IT Developer’s Real World Testing requirements.

ONC Real World Testing Results Report

Under the ONC Health IT Certification Program, Health IT Developers are required to conduct Real World Testing of their Certified Health IT (45 CFR 170.556 and 170.523(i)). This report provides a summary of the results of the execution of the MedInformatix Real-World Testing plan submitted to ONC for 2022.

Introduction

This document contains a list of the steps taken to conduct the annual Real World Testing requirements for ONC certification. The Results within this document were reviewed as Screenshots and spreadsheets for their compliance with the criteria defined in the test plan. These artifacts will be maintained by the health IT developer, MedInformatix for audit purposes or further requests.

CHANGES TO ORIGINAL PLAN

SUMMARY OF TESTING METHODS AND KEY FINDINGS

MedInformatix developed a Real-World Test plan and measures for Real World Testing to demonstrate interoperability and functionality of our certified health IT in real world settings and scenarios. We conducted and measured observations of interoperability and data exchange for the following applicable certification criteria:

Real World Testing Approach

Our approach for Real World Testing focuses on capturing and documenting the number of instances that deployed certified capability is successfully utilized in a provider's production environment as per the test plan. In instances where no evidence exists due to low or zero adoption of a certified capability or the inability to capture evidence of successful use for other reasons, we tested and demonstrated the required certified capability in a semi-controlled setting as close to a “real world” implementation as possible.

We are using a 3-fold approach to demonstrate successful real-world implementations.

• ❯ Adoption Rate
• ❯ Summative Testing
• ❯ Interactive Testing

Adoption rate determines if/when certified capability is being used in the real world and helps identify differences in care settings. Evidence of high rates of implementation and usage indicate (but don't by themselves prove) a certified capability's usefulness and practical value. Evidence of low rates of implementation and usage might indicate a potential problem, of which there could be several different causes and can be accounted for by patient volume, location, or provider preference among other reasons. Note, it is not the goal of this exercise to identify the individual causes of why a given certified capability may have a high or low adoption rate, but rather to identify the users and care settings for which a given test is relevant.

We identified all patient facing users that participate in the QPP program (Primary Care and Other specialties) as well as non-patient facing users like Radiology Specialty. Of our total users that have our CEHRT version we took a 10% sample for our RWT participation.

Summative assessments measure which certified actions were performed at the conclusion of a given time frame (90 days). We conducted the assessments by running reports and examining audit logs from within the certified health IT module to help demonstrate the frequency of actions within the given time frame, and where possible, whether those actions were successful or unsuccessful. High success rates should be an indicator of a successful implementation of a given certified capability in a real-world setting.

Interactive testing was used to demonstrate conformance to requirements where the adoption rate of a given certified capability is zero and to demonstrate ongoing compliance with updated standards and code sets (SVAP). Interactive tests were live tested as opposed to examining historical usage statistics. The goal being to demonstrate the certified Health IT module being used in a way consistent within a practice or care setting.

This approach allowed for the successful testing and obtaining results for each criterion. Detailed below in the Metrics and Outcomes section the reader will find evidential data in the form of a Summative result(s) or Interactive test outcome for each certified criterion for MedInformatix EHR V 7.7

STANDARDS UPDATES (INCLUDING STANDARDS VERSION ADVANCEMENT PROCESS (SVAP) AND UNITED STATES CORE DATA FOR INTEROPERABILITY (USCDI))

Both required and voluntary standards updates must be addressed in the Real-World Testing plan. Real World Testing plans must include all certified health IT updated to newer versions of standards prior to August 31 of the year in which the updates were made. Indicate as to whether optional standards, via SVAP and/or USCDI, are leveraged as part of the certification of your health IT product(s).

• Yes, I have products certified with voluntary SVAP or USCDI standards.
• No, none of my products include these voluntary standards.

Care Setting(s)

MedInformatix EHR specializes in Radiology settings and is marketed primarily to Radiology. Our secondary market are multiple ambulatory specialties. Even though our primary sales focus is Radiology, our EHR is successfully used by any ambulatory clinician, and approximately 50% of our user base are non-radiology ambulatory specialties. Our solutions are comprised of Certified EHR/PHR, RIS, Revenue Cycle Management, and Enterprise Practice Management. While the clinical workflow may be different between different specialty providers, use of the certified criteria is the same.

Care Settings Tested:

Metrics and Outcomes

The following metrics were measured during a 90-day period of June 1 through August 31st of 2022, using a 10% sample of all MedInformatix CEHRT users. The care settings tested are Primary Care, Radiology and Other Specialties. This section details the outcomes from our testing that successfully demonstrate that MedInformatix certified health IT is compliant with the certification criteria, is exchanging electronic health information (EHI) in the care and practice settings for which we market for use; and/or, EHI is received by and used in the certified health IT.

MedInformatix collected audit logs to support spreadsheets that demonstrate proof of Real-World Testing for each criterion in Summative Testing. These files are referenced and remain on file with MedInformatix.

For measures requiring interactive testing this testing was performed during the month of January 2023.

Within this section, we describe how the specific data collected in the client's live production system, of our Real-World Testing measures demonstrate the results. We also calculated adoption % rates for most measures based on our sample RWT customers and included those as part of the outcomes.

Outcome Details - Results Analysis

The following sections contain additional descriptions and test results supporting documentation to provide more context for the testing outcomes defined in the Metrics and Outcomes table above for each certified criterion.

Please contact MedInformatix for any Results spreadsheets if needed.

Summative Test

The purpose of this test was to show that CDA documents can be created and exported. A query on historical audit logs for 90-day periods was performed for the 170.315(b)(1) criterion. The resulting totals show that this module was active throughout the period and therefore demonstrates a compliant result.

Justification

This criterion requires the ability of a certified Health IT module to create CCDAs according to specified standards and vocabulary code sets, as well as send and receive CCDAs via edge protocols. However, it is not possible to consistently and reliably demonstrate that all required standards and code sets were used because not all CCDAs created in a real-world setting contain all the necessary data elements. Furthermore, it is not feasible to obtain copies of CCDA documents from “outside” developers or providers who have no incentive to participate in this exercise. Therefore, we intend to demonstrate the required certified capabilities by demonstrating how often CCDAs are created and exchanged with other systems to demonstrate the certified capability is available and effective, regardless of the frequency it is used.

Assumptions for 170.315(b)(1) Transitions of care

Our expectation or assumption was that there will be moderate utilization by providers with a high success rate.

Results found

The data shows a low frequency count in Primary Care Physicians (PCP), the Radiology and “Other” specialties meets our expectation with a moderate utilization.

Summative Test

The purpose of this test was to show that CDA documents can be imported, matched to a patient, reconciled and new CDA documents created and exported. A query on historical audit logs for 90-day periods was performed for the 170.315(b)(2) criterion. The resulting totals show that this module was active throughout the period and therefore demonstrates a compliant result.

Justification

This criterion requires the ability of a certified Health IT module to take a CCDA received via an outside system and match it to the correct patient; reconcile the medication, allergy, and problem lists; and then incorporate the lists into the patient record. The expectation is each of these steps is done electronically within the certified Health IT module. While this certified capability is available to our users, most providers in the real world typically prefer to perform these steps manually and elect to save any outside received CCDAs as attachments to the patient record. Therefore, we intend to record the frequency that providers are electronically reconciling and incorporating CCDAs that were received from outside providers to demonstrate the certified capability is available and effective, regardless of the frequency it is used.

Assumptions for 170.315(b)(2) Clinical information reconciliation and incorporation

Our expectation or assumption was that there will be low utilization by providers with a high success rate.

Results found

The data shows a low frequency count in “Other” specialties. Primary Care Physicians (PCP), and Radiology (RIS) had zero utilization. This indicates that our clients are resistant to import information from other systems and populate with “meaningful” data

Summative Test

The purpose of this test was to show that an active connection from EHR customer sites to an ePrescribing solution was deployed. A query on historical audit logs for 90-day periods was performed for the 170.315(b)(3) criterion. The resulting totals show that this module was active throughout the period and therefore demonstrates a compliant result.

Justification

This criterion requires the ability of a certified Health IT module to perform prescription-related electronic transactions (eRx) using required standards. However, it is not possible to demonstrate the correct standards were used because it is not feasible to obtain copies of eRx documents from “outside” companies or pharmacies who have no incentive to participate. Therefore, we intend to demonstrate the required certified capabilities are effective by demonstrating how often eRx transactions are performed by examining reports from our eRx partner. This will demonstrate that not only are the eRx transactions sent from the certified Health IT module, but that the transactions are successfully received by the eRx clearinghouse.

Assumptions for 170.315(b)(3) Electronic prescribing:

Our expectation or assumption was that there will be high utilization by providers with a high success rate.

Results found

The data shows a high utilization by providers for the required prescriptions Created and prescriptions Renewed programs outside of the Radiology specialty. Usage of Cancel and Change are optional programs and do not have as high utilization.

Summative Test

The purpose of this test was to show that our customers can export patient data from our HER without any assistance from MedInformatix. A query on historical audit logs for 90-day period was performed for the 170.315(b)(6) criterion. The resulting totals show that this module was active throughout the period and therefore demonstrates a compliant result.

Justification

This criterion requires the ability of a certified Health IT module to export a summary of a patient's record in CCDA format according to specified standards and vocabulary code sets. However, it is not possible to consistently and reliably demonstrate that all required standards and code sets were used because not all CCDAs created in a real-world setting contain all the necessary data elements. Therefore, we intend to demonstrate the certified capability is available and effective, regardless of the frequency it is used.

Assumptions for 170.315(b)(6) Data export

Our expectation or assumption was that there will be very low utilization by providers with a high success rate.

Results found

The data shows a moderate utilization across our sampled client base of data export for a single patient with a high success rate. This result was higher than expected for this measure.

There was one site with very high usage but overall, the usage was very low for batch export. Radiology specialty had no usage reported.

Our specialty providers are utilizing this capability more than PCP and Radiology.

Summative Test

The purpose of this test was to show that the EHR can create, receive, and sequester restricted summary of record (CCDA) documents. A query on historical audit logs for 90-day periods was performed for the 170.315(b)(7) and 170.315(b)(8) criterion. The resulting totals show that this module was active throughout the period and therefore demonstrates a compliant result.

Justification

170.315(b)(7) criterion requires the ability of a certified Health IT module to create a CCDA document according to specified standards and vocabulary code sets and tagged as restricted. However, it is impossible to consistently and reliably demonstrate that all required standards and code sets were used because not all CCDAs created in a real-world setting contain the necessary data elements.

Therefore, we intend to demonstrate the certified capability is available and effective, regardless of the frequency it is used. 170.315(b)(8) criterion requires the ability of a certified Health IT module to receive a CCDA document tagged as restricted and to sequester the document from other unrestricted documents. We intend to record the frequency that providers receive a CCDA tagged as restricted and how often they are sequestered as a result to demonstrate the certified capability is available and effective, regardless of the frequency it is used. Numbers are duplicate because for every restricted document we receive it's immediately sequestered.

Assumptions for 170.315(b)(7) Security tags - summary of care - send and 170.315(b)(8) Security tags - summary of care - receive

Our expectation or assumption was that there will be very low utilization by providers with a high success rate.

Results found

The data shows a low utilization by providers as expected. There was one outlier with high usage but overall, the usage was very low for both send and receive summary of care. Radiology and PCP specialty had no usage reported. We saw high utilization in our specialty practices where the restriction was more commonly used.

Summative Test

The purpose of this test was to show that the EHR provides patients access to a patient portal with the ability to view, download, and send their health care records. A query on historical audit logs for 90-day periods was performed for the 170.315(e)(1) criterion. The resulting totals show that this module was active throughout the period and therefore demonstrates a compliant result.

Justification

This criterion requires the ability of a certified Health IT module to provide patients access to a patient portal with the ability to view, download, and send their health care records to other providers via encrypted or unencrypted transmission methods in CCDA format. We intend to record the frequency that patients are viewing, downloading, and transmitting their records from the portal using the certified capabilities to demonstrate the certified capability is available and effective, regardless of the frequency it is used.

Assumptions for 170.315(e)(1) View, download, and transmit to 3rd party

Our expectation is there will be moderate utilization by patients for view and lower utilization for download and transmit with a high success rate for all certified capabilities.

Results found

The data shows a moderate utilization by patients with a high success rate. This technology is self-taught to the patient when they sign in. Usage of Download and Transmission to another practitioner by a patient is nonexistent. The responsibility of transmitting records to other practitioners still falls on the practice. This is why we believe there is low utilization in this area.

Summative Test

The purpose of this test was to show that the EHR can fulfill an API request that enables external applications to request a unique patient identifier from the certified Health IT module that can be used to request additional patient data. A query on historical audit logs for 90-day periods was performed for the 170.315(g)(7) criterion. The resulting totals show that this module was active throughout the period and therefore demonstrates a compliant result.

Justification

This criterion requires the certified Health IT module to provide an API and supporting documentation that enable external applications to request a unique patient identifier from the certified Health IT module that can be used to request additional patient data. We intend to record the frequency that patient ID requests are received by providers via API to demonstrate the certified capability is available and effective, regardless of the frequency it is used.

Assumptions for 170.315(g)(7) Application access — patient selection - Results Analysis

Our expectation is there will be very low utilization by users with a high success rate.

Results found

Currently there is one third-party entity connected to our public FHIR API with only one customer utilizing the application during the October-December 90-day period.

We were surprised with utilization being moderate when we expected it to be low with just one customer. The data shows a moderate utilization with a moderate success rate. This indicates that users are eager to try new technologies.

Market has yet to penetrate our client base to consume FHIR API solutions. Overall, there is very low utilization/adoption of FHIR/API solutions.

Summative Test

The purpose of this test was to show that the EHR can fulfill an API request that enables external applications to request patient data categories from the certified Health IT module. A query on historical audit logs for 90-day periods was performed for the 170.315(g)(8) criterion. The resulting totals show that this module was active throughout the period and therefore demonstrates a compliant result.

Justification

This criterion requires the certified Health IT module to provide an API and supporting documentation that enable external applications to request patient data by category from the certified Health IT module. We intend to record the frequency that patient data requests by category are received by providers and fulfilled via API to demonstrate the certified capability is available and effective, regardless of the frequency it is used.

Assumptions for 170.315(g)(7) Application access — patient selection - Results Analysis

Our expectation is there will be very low utilization by users with a high success rate.

Results found

The data shows a Moderate utilization by users with a moderate/high success rate.

When monitoring the RWT we observed an incompleteness in the report. We promptly investigated and reported internally the matter. The issue was resolved on December 22. Data on follow up requests made before 12/22/2022 is incomplete due to this misconfiguration of our logging middleware that caused audit events to be generated only for authentication / authorization related requests.

In our report you can see what would have been considered a normal success rate during the 12/22 through 12/31-time frame. This report is for a 90-day period, we addressed our technical issue and expect to see improvement in the next reporting period.

Summative/Interactive Test

The purpose of this test was to show that the EHR can fulfill an API request that enables external applications to request all categories of patient data defined in the CCDS from the certified Health IT module. A query on historical audit logs for 90-day periods was performed for the 170.315(g)(9) criterion. Due to no results found during this summative test, MedInformatix demonstrated the module function in our system as an interactive test demonstrating a compliant result.

Assumptions for 170.315(g)(9) Application access — all data request — patient selection - Results Analysis

Our expectation is there will be very low utilization by users with a high success rate.

Results found

The data shows no utilization by third-party entities with no success rate. To demonstrate the module functions as certified we performed an interactive test to demonstrate compliance.

The current Third-party application that accesses our FHIR API does not request patient summary details. It does request other individual FHIR endpoints which is contained in this report. We haven't seen requests to access our public FHIR API from other external vendors.

We believe this is due to a low market adoption of this technology.

KEY MILESTONES

Below is a list of key milestones that were met during the Real-World Testing process.

REPORT OVERVIEW

This report outlines MedInformatix's plan to measure the real world use of interoperability features for the year 2023 for their MedInformatix EHR 7.7 Certified health IT. The certification criteria to be observed in this plan include § 170.315(b)(1)-(3), (b)(6)-(8), (e)(1), and (g)(7)-(9).

JUSTIFICATION FOR REAL WORLD TESTING APPROACH

Consistent with the ONC's recommendation that "Real World Testing verify that deployed Certified Health IT continues to perform as intended by conducting and measuring observations of interoperability and data exchange,"this test plan focuses on capturing and documenting the number of instances that certified capability is successfully utilized in the real world. In instances where no evidence exists due to zero adoption of a certified capability or the inability to capture evidence of successful use for other reasons, we will demonstrate the required certified capability in a semi-controlled setting as close to a "real world"implementation as possible.

It is important to note that Real World Testing is only one component of the Health IT Certification program that demonstrates compliance with the program requirements. Real World Testing should augment and support testing conducted before certification was granted. It does not duplicate the methods or results previously demonstrated. Instead, this test plan demonstrates that the certified capabilities have been successfully deployed for providers to use at their discretion in live settings.

We are using a 2-fold approach to demonstrate successful real-world implementations.

Participants - To evaluate a representative sample of organizations, MedInformatix will conduct Real World Testing on 10% of our (users or organizations) of the Certified Health IT version. These participants will represent all care settings supported. The sample will include small, medium, and large organizations.

STANDARDS UPDATES

As of August 31, 2022, MedInformatix has not updated MedInformatix EHR 7.7 to any new standards as part of SVAP or the Cures Update criteria.

CARE SETTINGS

MedInformatix EHR specializes in Radiology settings and is marketed primarily to Radiology. Our secondary market includes multiple ambulatory specialties. Even though our primary sales focus is Radiology, our EHR is successfully used by any ambulatory clinician, and approximately 50% of our user base are non-radiology ambulatory specialties. Our solutions comprise Certified EHR/PHR, RIS, Revenue Cycle Management, and Enterprise Practice Management.

While the clinical workflow may differ between different specialty providers, the use of the certified criteria is the same.

MEASURES USED IN THE OVERALL APPROACH

For each measurement/metric, describe the elements below:

• ❯ Description of the measurement/metric
• ❯ Associated certification criteria
• ❯ Care setting(s) that are addressed
• ❯ Justification for selected measurement/metric
• ❯ Expected Outcomes

ADOPTION RATES

The following metrics are applicable to all criteria and all care settings. These metrics will not be used directly to demonstrate interoperability or conformance to certification criteria. Instead, they will primarily be used to help determine the participants that will be in scope for this evaluation. They can also aid with the justification for other metrics by providing additional context (i.e., meager adoption rates for specific certified capabilities will necessitate a different approach to testing).

The following metrics apply to all criteria licensed separately from the base license and all care settings.

SUMMATIVE ASSESSMENT METRICS

The following metrics will be measured by viewing audit logs and reporting systems available to track the behavior of the certified Health IT module during a given time frame. All metrics are designed to reflect the core elements of the criteria, demonstrate interoperability, and demonstrate the success rate of the certified capability being used. In most cases, we elected to record these metrics over 90 days to reflect the reporting periods typically required for compliance with the federal incentive programs.

The continued measurable use of certified capabilities will provide implicit evidence of the successful implementation of the required certified capability. This is especially meaningful when demonstrating interoperability with external systems. When it is impossible to determine success via an explicit confirmation by a receiving system, success is defined as a transmission made where no error was received from the destination system or its intermediaries. Additionally, we will review internal customer and vendor issue tracking systems for reports of failures or unsatisfactory performance in the field.

MedInformatix did not update any of the following criteria to the Cures Update version of the criteria prior to August 31, 2022. As a result, all testing is scheduled to be conducted against the 2015 Edition version of the criteria.

RELIED UPON SOFTWARE

MedInformatix uses the following relied upon software to demonstrate conformity for the following certification criterion:

• 170.315(b)(3) Electronic prescribing: We use NewCrop, an e-prescribe solution that is integrated with our EHR and other vendors.
• 170.315(e)(1) View, download, and transmit to 3rd party: For this criterion we use the following relied upon software:

  MedInformatix Patient portal

  A MedInformatix module that strengthens engagement between patients and providers.

  GEMMS Patient portal

  A module for generation of CCDA content.

  Updox

  A healthcare communication platform to ensure transmission of CCDAs to any third party.

SCHEDULE OF KEY MILESTONES

Real World test planning will commence in first quarter of 2023. We expect each phase to take 90 days to complete, with report writing to occur end of 2023/early 2024.

ATTESTATION

This Real World Testing plan is complete with all required elements, including measures that address all certification criteria and care settings. In addition, all information in this plan is up to date and fully addresses the Health IT Developer's Real World Testing requirements.

ONC Real World Testing Results Report

Under the ONC Health IT Certification Program, Health IT Developers are required to conduct Real World Testing of their Certified Health IT (45 CFR 170.556 and 170.523(i)). This report provides a summary of the results of the execution of the MedInformatix Real-World Testing plan submitted to ONC for 2023.

Introduction

This document contains a list of the steps taken to conduct the annual Real World Testing requirements for ONC certification. The Results within this document were reviewed as Screenshots and spreadsheets for their compliance with the criteria defined in the test plan. These artifacts will be maintained by the health IT developer, MedInformatix for audit purposes or further requests.

CHANGES TO ORIGINAL PLAN

SUMMARY OF TESTING METHODS AND KEY FINDINGS

MedInformatix developed a Real-World Test plan and measures for Real World Testing to demonstrate interoperability and functionality of our certified health IT in real world settings and scenarios. We conducted and measured observations of interoperability and data exchange for the following applicable certification criteria:

Real World Testing Approach

Our approach for Real World Testing focuses on capturing and documenting the number of instances that deployed certified capability is successfully utilized in a provider’s production environment as per the test plan. In instances where no evidence exists due to low or zero adoption of a certified capability or the inability to capture evidence of successful use for other reasons, we tested and demonstrated the required certified capability in a semi-controlled setting as close to a “real world” implementation as possible.

We are using a 3-fold approach to demonstrate successful real-world implementations.

• ❯ Adoption Rate
• ❯ Summative Testing
• ❯ Interactive Testing

Adoption rate determines if/when certified capability is being used in the real world and helps identify differences in care settings. Evidence of high rates of implementation and usage indicate (but don’t by themselves prove) a certified capability’s usefulness and practical value. Evidence of low rates of implementation and usage might indicate a potential problem, of which there could be several different causes and can be accounted for by patient volume, location, or provider preference among other reasons. Note, it is not the goal of this exercise to identify the individual causes of why a given certified capability may have a high or low adoption rate, but rather to identify the users and care settings for which a given test is relevant.

We identified all patient facing users that participate in the QPP program (Primary Care and Other specialties) as well as non-patient facing users like Radiology Specialty. Of our total users that have our CEHRT version we took a 10% sample for our RWT participation.

Summative assessments measure which certified actions were performed at the conclusion of a given time frame (365 days). We conducted the assessments by running reports and examining audit logs from within the certified health IT module to help demonstrate the frequency of actions within the given time frame, and where possible, whether those actions were successful or unsuccessful. High success rates should be an indicator of a successful implementation of a given certified capability in a real-world setting.

Interactive testing was used to demonstrate conformance to requirements where the adoption rate of a given certified capability is zero and to demonstrate ongoing compliance with updated standards and code sets (SVAP). Interactive tests were live tested as opposed to examining historical usage statistics. The goal being to demonstrate the certified Health IT module being used in a way consistent within a practice or care setting.

This approach allowed for successful testing and obtaining results for each criterion. Detailed below in the Metrics and Outcomes section the reader will find evidential data in the form of a Summative result(s) or Interactive test outcome for each certified criterion for MedInformatix EHR V 7.7

STANDARDS UPDATES (INCLUDING STANDARDS VERSION ADVANCEMENT PROCESS (SVAP) AND UNITED STATES CORE DATA FOR INTEROPERABILITY (USCDI))

Both required and voluntary standards updates must be addressed in the Real-World Testing plan. Real World Testing plans must include all certified health IT updated to newer versions of standards prior to August 31 of the year in which the updates were made. Indicate as to whether optional standards, via SVAP and/or USCDI, are leveraged as part of the certification of your health IT product(s).

• Yes, I have products certified with voluntary SVAP or USCDI standards.
• No, none of my products include these voluntary standards.

Care Setting(s)

MedInformatix EHR specializes in Radiology settings and is marketed primarily to Radiology. Our secondary market is multiple ambulatory specialties. Even though our primary sales focus is Radiology, our EHR is successfully used by any ambulatory clinician, and approximately 50% of our user base are non-radiology ambulatory specialties. Our solutions are comprised of Certified EHR/PHR, RIS, Revenue Cycle Management, and Enterprise Practice Management. While the clinical workflow may be different between different specialty providers, use of the certified criteria is the same.

Care Settings Tested:

Metrics and Outcomes

The following metrics were measured during a 365-day period of January 1 through December 31st of 2023, using a 10% sample of all MedInformatix CEHRT users. The care settings tested are Primary Care, Radiology and Other Specialties. This section details the outcomes from our testing that successfully demonstrate that MedInformatix certified health IT is compliant with the certification criteria, is exchanging electronic health information (EHI) in the care and practice settings for which we market for use; and/or, EHI is received by and used in the certified health IT.

MedInformatix collected audit logs to support spreadsheets that demonstrate proof of Real-World Testing for each criterion in Summative Testing. These files are referenced and remain on file with MedInformatix.

For measures requiring interactive testing this testing was performed during the month of January 2023.

Within this section, we describe how the specific data collected in the client’s live production system of our Real-World Testing measures demonstrate the results. We also calculated adoption % rates for most measures based on our sample RWT customers and included those as part of the outcomes.

Outcome Details - Results Analysis

The following sections contain additional descriptions and test results supporting documentation to provide more context for the testing outcomes defined in the Metrics and Outcomes table above for each certified criterion.

Please contact MedInformatix for any Results spreadsheets if needed.

Summative Test

The purpose of this test was to show that CDA documents can be created and exported. A query on historical audit logs for 365-day period was performed for the 170.315(b)(1) criterion. The resulting totals show that this module was active throughout the period and therefore demonstrates a compliant result.

Justification

This criterion requires the ability of a certified Health IT module to create CCDAs according to specified standards and vocabulary code sets, as well as send and receive CCDAs via edge protocols. However, it is not possible to consistently and reliably demonstrate that all required standards and code sets were used because not all CCDAs created in a real-world setting contain all the necessary data elements. Furthermore, it is not feasible to obtain copies of CCDA documents from “outside” developers or providers who have no incentive to participate in this exercise. Therefore, we intend to demonstrate the required certified capabilities by demonstrating how often CCDAs are created and exchanged with other systems to demonstrate the certified capability is available and effective, regardless of the frequency it is used.

Assumptions for 170.315(b)(1) Transitions of care

Our expectation or assumption was that there will be moderate utilization by providers with a high success rate.

Results found

The data shows a low frequency count in Primary Care Physicians (PCP), the Radiology and “Other” specialties meets our expectation with a moderate utilization.

Summative Test

The purpose of this test was to show that CDA documents can be imported, matched to a patient, reconciled and new CDA documents created and exported. A query on historical audit logs for 365-day period was performed for the 170.315(b)(2) criterion. The resulting totals show that this module was active throughout the period and therefore demonstrates a compliant result.

Justification

This criterion requires the ability of a certified Health IT module to take a CCDA received via an outside system and match it to the correct patient; reconcile the medication, allergy, and problem lists; and then incorporate the lists into the patient record. The expectation is each of these steps is done electronically within the certified Health IT module. While this certified capability is available to our users, most providers in the real world typically prefer to perform these steps manually and elect to save any outside received CCDAs as attachments to the patient record. Therefore, we intend to record the frequency that providers are electronically reconciling and incorporating CCDAs that were received from outside providers to demonstrate the certified capability is available and effective, regardless of the frequency it is used.

Assumptions for 170.315(b)(2) Clinical information reconciliation and incorporation

Our expectation or assumption was that there will be low utilization by providers with a high success rate.

Results found

The data shows a low frequency count in “Other” specialties and Primary Care Physicians (PCP), and Radiology (RIS) had zero utilization. This indicates that our clients are resistant to importing information from other systems and populate with “meaningful” data points, despite providing a mid-year webinar training during the 2023 performance year.

Summative Test

The purpose of this test was to show that an active connection from EHR customer sites to an ePrescribing solution was deployed. A query on historical audit logs for 365-day period was performed for the 170.315(b)(3) criterion. The resulting totals show that this module was active throughout the period and therefore demonstrates a compliant result.

Justification

This criterion requires the ability of a certified Health IT module to perform prescription-related electronic transactions (eRx) using required standards. However, it is not possible to demonstrate the correct standards were used because it is not feasible to obtain copies of eRx documents from “outside” companies or pharmacies who have no incentive to participate. Therefore, we intend to demonstrate the required certified capabilities are effective by demonstrating how often eRx transactions are performed by examining reports from our eRx partner. This will demonstrate that not only are the eRx transactions sent from the certified Health IT module, but that the transactions are successfully received by the eRx clearinghouse.

Assumptions for 170.315(b)(3) Electronic prescribing:

Our expectation or assumption was that there will be high utilization by providers with a high success rate.

Results found

The data shows a high utilization by providers for the required prescriptions Created program. Usage of Cancel, Renewed and Change are optional programs and we saw an increase in usage after a mid-year webinar training. Radiology specialty have an inherent exception, so utilization meets our expectations.

Summative Test

The purpose of this test was to show that our customers can export patient data from our EHR without any assistance from MedInformatix. A query on historical audit logs for 365-day period was performed for the 170.315(b)(6) criterion. The resulting totals show that this module was active throughout the period and therefore demonstrates a compliant result.

Justification

This criterion requires the ability of a certified Health IT module to export a summary of a patient’s record in CCDA format according to specified standards and vocabulary code sets. However, it is not possible to consistently and reliably demonstrate that all required standards and code sets were used because not all CCDAs created in a real-world setting contain all the necessary data elements. Therefore, we intend to demonstrate the certified capability is available and effective, regardless of the frequency it is used.

Assumptions for 170.315(b)(6) Data export

Our expectation or assumption was that there will be low utilization for export of a single patient by providers and moderate utilization of batch export by providers with a high success rate.

Results found

The data shows a low utilization across our sampled client base of data export for a single patient with a high success rate. For batch export the data shows a high utilization across our sampled client base with a high success rate. This result was higher than expected for this measure. Radiology specialty had no usage reported. Our specialty and PCP providers are utilizing this capability more. The data indicates that offices prefer the batch processes over using the transition of care workflow.

Summative Test

The purpose of this test was to show that the EHR can create, receive, and sequester restricted summary of record (CCDA) documents. A query on historical audit logs for a 365-day period was performed for the 170.315(b)(7) and 170.315(b)(8) criterion. The resulting totals show that this module was active throughout the period and therefore demonstrates a compliant result.

Justification

170.315(b)(7) criterion requires the ability of a certified Health IT module to create a CCDA document according to specified standards and vocabulary code sets and tagged as restricted. However, it is impossible to consistently and reliably demonstrate that all required standards and code sets were used because not all CCDAs created in a real-world setting contain the necessary data elements.

Therefore, we intend to demonstrate the certified capability is available and effective, regardless of the frequency it is used. 170.315(b)(8) criterion requires the ability of a certified Health IT module to receive a CCDA document tagged as restricted and to sequester the document from other unrestricted documents. We intend to record the frequency that providers receive a CCDA tagged as restricted and how often they are sequestered as a result to demonstrate the certified capability is available and effective, regardless of the frequency it is used. Numbers are duplicate because for every restricted document we receive it’s immediately sequestered.

Assumptions for 170.315(b)(7) Security tags - summary of care - send and 170.315(b)(8) Security tags - summary of care - receive

Our expectation or assumption was that there will be very low utilization by providers with a high success rate.

Results found

The data shows a low utilization by providers as expected. There was one outlier with high usage but overall, the usage was very low for both send and receive summary of care. Radiology and PCP specialty had no usage reported.

Summative Test

The purpose of this test was to show that the EHR provides patients access to a patient portal with the ability to view, download, and send their health care records. A query on historical audit logs for 365-day period was performed for the 170.315(e)(1) criterion. The resulting totals show that this module was active throughout the period and therefore demonstrates a compliant result.

Justification

This criterion requires the ability of a certified Health IT module to provide patients access to a patient portal with the ability to view, download, and send their health care records to other providers via encrypted or unencrypted transmission methods in CCDA format. We intend to record the frequency that patients are viewing, downloading, and transmitting their records from the portal using the certified capabilities to demonstrate the certified capability is available and effective, regardless of the frequency it is used.

Assumptions for 170.315(e)(1) View, download, and transmit to 3rd party

Our expectation is there will be moderate utilization by patients for view and lower utilization for download and transmit with a high success rate for all certified capabilities.

Results found

The data shows low (Radiology), moderate (Other), and high (PCP) utilization by patients depending on the specialty with a high success rate in viewing their records. This technology is self-taught to the patient when they sign in. Usage of Download and Transmission to another practitioner by a patient is very low. Patients still expect practitioners to provide courtesy copy to other practices. This is why we believe there is low utilization in this area.

Summative Test

The purpose of this test was to show that the EHR can fulfill an API request that enables external applications to request a unique patient identifier from the certified Health IT module that can be used to request additional patient data. A query on historical audit logs for a 265-day period was performed for the 170.315(g)(7) criterion. The resulting totals show that this module was active throughout the period and therefore demonstrates a compliant result.

Justification

This criterion requires the certified Health IT module to provide an API and supporting documentation that enable external applications to request a unique patient identifier from the certified Health IT module that can be used to request additional patient data. We intend to record the frequency that patient ID requests are received by providers via API to demonstrate the certified capability is available and effective, regardless of the frequency it is used.

Assumptions for 170.315(g)(7) Application access — patient selection - Results Analysis

Our expectation is there will be Moderate utilization by radiology users with a high success rate. Our expectation for PCP and Other specialty will be low utilization with a high success rate.

Results found

Currently there is one third-party entity connected to our public FHIR API with thirteen customers utilizing the application during the April-December 265-day period.

Utilization increased as the number of customers increased. The data shows high utilization with a moderate success rate in our Radiology care setting while utilization in the other areas was low because they adopted the technology later in the year. The market has yet to penetrate our client base to consume FHIR API solutions. Overall, there is an increase in utilization of our FHIR/API solution.

Summative Test

The purpose of this test was to show that the EHR can fulfill an API request that enables external applications to request patient data categories from the certified Health IT module. A query on historical audit logs for 265-day period was performed for the 170.315(g)(8) criterion. The resulting totals show that this module was active throughout the period and therefore demonstrates a compliant result.

Justification

This criterion requires the certified Health IT module to provide an API and supporting documentation that enable external applications to request patient data by category from the certified Health IT module. We intend to record the frequency that patient data requests by category are received by providers and fulfilled via API to demonstrate the certified capability is available and effective, regardless of the frequency it is used.

Assumptions for 170.315(g)(7) Application access — patient selection - Results Analysis

Our expectation is there will be moderate utilization by radiology users with a high success rate. Our expectation for PCP and Other specialty will be low utilization with a high success rate.

Results found

The data shows high utilization with a high success rate in our Radiology and other care setting while utilization in the PCP area was low because they adopted the technology later in the year.

Summative/Interactive Test

The purpose of this test was to show that the EHR can fulfill an API request that enables external applications to request all categories of patient data defined in the CCDS from the certified Health IT module. A query on historical audit logs for a 265-day period was performed for the 170.315(g)(9) criterion. Due to no results found during this summative test, MedInformatix demonstrated the module function in our system as an interactive test demonstrating a compliant result.

Assumptions for 170.315(g)(9) Application access — all data request — patient selection - Results Analysis

Our expectation is there will be very low utilization by users with a high success rate.

Results found

The data shows no utilization by third-party entities with no success rate. To demonstrate the module functions as certified we performed an interactive test to demonstrate compliance.

The current Third-party application that accesses our FHIR API does not request patient summary details. It does request other individual FHIR endpoints which is contained in this report.

We have seen a few requests to access our public FHIR API from other external vendors but no engagement.

KEY MILESTONES

Below is a list of key milestones that were met during the Real-World Testing process.

REPORT OVERVIEW

This report outlines the MedInformatix plan to measure the Real World use of interoperability features for the year 2024 for their MedInformatix EHR 7.7 and 7.8 Certified health IT. The certification criteria to be observed in this plan include § 170.315(b)(1)-(3), (b)(6)-(8), (e)(1), (g)(7), (9) and (g)(10).

JUSTIFICATION FOR REAL WORLD TESTING APPROACH

Consistent with the ONC's recommendation that "Real World Testing verify that deployed Certified Health IT continues to perform as intended by conducting and measuring observations of interoperability and data exchange," this test plan focuses on capturing and documenting the number of instances that certified capability is successfully utilized in the real world. In instances where no evidence exists due to zero adoption of a certified capability or the inability to capture evidence of successful use for other reasons, we will demonstrate the required certified capability in a semi-controlled setting as close to a "real world" implementation as possible.

It is important to note that Real World Testing is only one component of the Health IT Certification program that demonstrates compliance with the program requirements. Real World Testing should augment and support testing conducted before certification is granted. It does not duplicate the methods or results previously demonstrated. Instead, this test plan demonstrates that the certified capabilities have been successfully deployed for providers to use at their discretion in live settings.

We are using a 2-fold approach to demonstrate successful real-world implementations.

Participants - MedInformatix will conduct Real World Testing on 10% of our customer organizations using the Certified Health IT version to evaluate a representative sample of organizations. These participants will represent all care settings supported. The sample will include small, medium, and large organizations.

STANDARDS UPDATES

As of August 31, 2023, MedInformatix has not updated MedInformatix EHR 7.7 and 7.8 to any new standards as part of SVAP or the Cures Update criteria.

CARE SETTINGS

MedInformatix EHR specializes in Radiology settings and is marketed primarily to Radiology. Our secondary market includes multiple ambulatory specialties. Even though our primary sales focus is Radiology, our EHR is successfully used by any ambulatory clinician, and approximately 50% of our user base are non-radiology ambulatory specialties. Our solutions comprise Certified EHR/PHR, RIS, Revenue Cycle Management, and Enterprise Practice Management.

While the clinical workflow may differ between different specialty providers, the use of the certified criteria is the same.

MEASURES USED IN THE OVERALL APPROACH

For each measurement/metric, describe the elements below:

• ✅ Description of the measurement/metric
• ✅ Associated certification criteria
• ✅ Care setting(s) that are addressed
• ✅ Justification for selected measurement/metric
• ✅ Expected Outcomes

ADOPTION RATES

The following metrics are applicable to all criteria and all care settings. These metrics will not be used directly to demonstrate interoperability or conformance to certification criteria. Instead, they will primarily be used to help determine the participants that will be in scope for this evaluation. They can also aid with the justification for other metrics by providing additional context (i.e., meager adoption rates for specific certified capabilities will necessitate a different approach to testing).

The following metrics apply to all criteria licensed separately from the base license and all care settings.

SUMMATIVE ASSESSMENT METRICS

The following metrics will be measured by viewing audit logs and reporting systems available to track the behavior of the certified Health IT module during a given time frame. All metrics are designed to reflect the core elements of the criteria, demonstrate interoperability, and demonstrate the success rate of the certified capability being used. In most cases, we elected to record these metrics over 180 days to reflect the reporting periods typically required for compliance with the federal incentive programs.

The continued measurable use of certified capabilities will provide implicit evidence of the successful implementation of the required certified capability. This is especially meaningful when demonstrating interoperability with external systems. When it is impossible to determine success via an explicit confirmation by a receiving system, success is defined as a transmission made where no error was received from the destination system or its intermediaries. Additionally, we will review internal customer and vendor issue tracking systems for reports of failures or unsatisfactory performance in the field.

MedInformatix updated the following criteria to the Cures Update version of the criteria prior to August 31, 2023. As a result, all testing is scheduled to be conducted against the Cures Update version of the criteria.

RELIED UPON SOFTWARE

MedInformatix uses the following relied upon software to demonstrate conformity for the following certification criterion:

• 170.315(b)(3) Electronic prescribing: We use NewCrop, an e-prescribe solution that is integrated with our EHR and other vendors.
• 170.315(e)(1) View, download, and transmit to 3rd party: For this criterion we use the following relied-upon software:

  MedInformatix Patient portal

  A MedInformatix module that strengthens engagement between patients and providers.

  GEMMS Patient portal

  A GEMMS application that strengthens engagement between patients and providers

  Updox

  A healthcare communication platform that allows patients to communicate with their providers via Direct Secure Messaging, including sending C-CDA documents.

SCHEDULE OF KEY MILESTONES

Real World test planning will commence in the first quarter of 2024. We expect each phase to take 90 days to complete, with report writing to occur end of 2024/early 2025.

ATTESTATION

This Real World Testing plan is complete with all required elements, including measures that address all certification criteria and care settings. In addition, all information in this plan is up to date and fully addresses the Health IT Developer's Real World Testing requirements.

REPORT OVERVIEW

This report outlines MedInformatix's plan to measure the real-world use of interoperability features for the year 2025 for their MedInformatix EHR 7.7 and 7.8 Certified health IT. The certification criteria to be observed in this plan include § 170.315(b)(1)-(3), (b)(7)-(8), (b)(10), (e)(1), (g)(7), (9) and (g)(10).

JUSTIFICATION FOR REAL WORLD TESTING APPROACH

Consistent with the ONC's recommendation that "Real World Testing verify that deployed Certified Health IT continues to perform as intended by conducting and measuring observations of interoperability and data exchange," this test plan focuses on capturing and documenting the number of instances that certified capability is successfully utilized in the real world. In instances where no evidence exists due to zero adoption of a certified capability or the inability to capture evidence of successful use for other reasons, we will demonstrate the required certified capability in a semi-controlled setting as close to a "real world" implementation as possible.

It is important to note that Real World Testing is only one component of the Health IT Certification program that demonstrates compliance with the program requirements. Real World Testing should augment and support testing conducted before certification is granted. It does not duplicate the methods or results previously demonstrated. Instead, this test plan demonstrates that the certified capabilities have been successfully deployed for providers to use at their discretion in live settings.

We are using a 2-fold approach to demonstrate successful real-world implementations.

Participants – To evaluate a representative sample of organizations, MedInformatix will conduct Real World Testing on 10% of our (users or organizations) of the Certified Health IT version. These participants will represent all care settings supported. The sample will include small, medium, and large organizations.

STANDARDS VERSION ADVANCEMENT PROCESS (SVAP) STANDARDS UPDATES

As of August 31, 2024, MedInformatix has not updated MedInformatix EHR 7.7 and 7.8 to any new standards as part of SVAP.

CARE SETTINGS

MedInformatix EHR specializes in Radiology settings and is marketed primarily to Radiology. Our secondary market includes multiple ambulatory specialties. Even though our primary sales focus is Radiology, our EHR is successfully used by any ambulatory clinician, and approximately 50% of our user base are non-radiology ambulatory specialties. Our solutions comprise Certified EHR/PHR, RIS, Revenue Cycle Management, and Enterprise Practice Management.

While the clinical workflow may differ between different specialty providers, the use of the certified criteria is the same.

MEASURES USED IN THE OVERALL APPROACH

For each measurement/metric, describe the elements below:

• ✅ Description of the measurement/metric
• ✅ Associated certification criteria
• ✅ Care setting(s) that are addressed
• ✅ Justification for selected measurement/metric
• ✅ Expected Outcomes

ADOPTION RATES

The following metrics are applicable to all criteria and all care settings. These metrics will not be used directly to demonstrate interoperability or conformance to certification criteria. Instead, they will primarily be used to help determine the participants that will be in scope for this evaluation. They can also aid with the justification for other metrics by providing additional context (i.e., meager adoption rates for specific certified capabilities will necessitate a different approach to testing).

The following metrics apply to all criteria licensed separately from the base license and all care settings.

SUMMATIVE ASSESSMENT METRICS

The following metrics will be measured by viewing audit logs and reporting systems available to track the behavior of the certified Health IT module during a given time frame. All metrics are designed to reflect the core elements of the criteria, demonstrate interoperability, and demonstrate the success rate of the certified capability being used. In most cases, we elected to record these metrics over 180 days to reflect the reporting periods typically required for compliance with the federal incentive programs.

The continued measurable use of certified capabilities will provide implicit evidence of the successful implementation of the required certified capability. This is especially meaningful when demonstrating interoperability with external systems. When it is impossible to determine success via an explicit confirmation by a receiving system, success is defined as a transmission made where no error was received from the destination system or its intermediaries. Additionally, we will review internal customer and vendor issue tracking systems for reports of failures or unsatisfactory performance in the field.

RELIED UPON SOFTWARE

MedInformatix uses the following relied upon software to demonstrate conformity for the following certification criterion:

• 170.315(b)(3) Electronic prescribing: We use NewCrop, an e-prescribe solution that is integrated with our EHR and other vendors.
• 170.315(e)(1) View, download, and transmit to 3rd party: For this criterion we use the following relied-upon software:

  MedInformatix Patient portal

  A MedInformatix module that strengthens engagement between patients and providers.

  GEMMS Patient portal

  A GEMMS application that strengthens engagement between patients and providers

  Updox

  A healthcare communication platform that allows patients to communicate with their providers via Direct Secure Messaging, including sending C-CDA documents.

SCHEDULE OF KEY MILESTONES

Real World test planning will commence in the first quarter of 2025. We expect each phase to take 90 days to complete, with report writing to occur end of 2025/early 2026.

ATTESTATION

This Real World Testing plan is complete with all required elements, including measures that address all certification criteria and care settings. In addition, all information in this plan is up to date and fully addresses the Health IT Developer's Real World Testing requirements.