Monitoring and evaluation is an integral part of TB programme management and provides information on the scope, quality, scale/coverage, and success of programmes.
Monitoring generally refers to the routine collection of information across time and sites in order to track a programme’s ongoing activities. Policy-makers use monitoring to track key health-related indicators, often without attributing change to any particular programme or set of programmes.
Evaluation involves the assessment of programme implementation in order to determine the worth or value of a programme in terms of its success in achieving predetermined outcomes/goals. Evaluation is usually achieved through a detailed analysis of the programme’s process and outcomes or impacts. Evaluation lends itself to the linkage of outcomes to the programme process, as well as rule-out non-programme effects on outcomes.
Monitoring and evaluation framework
Adequate monitoring and evaluation can only be achieved through a well-defined plan incorporating a framework of the goal, service delivery areas and activities. Indicators are needed to measure the performance and progress achieved.
It is crucial that the framework indicators can be critically analyzed and interpreted at all levels and corrective action taken. Guidelines and standard operating procedures for data management, analysis and quality assurance should also be developed and adhered to.
Role of the Stop TB programme
The Stop TB programme monitors and evaluates the implementation of the different components of the Stop TB Strategy through a quarterly web-based online system, DOTS quarterly online (or DQ online).
The Stop TB programme provides technical support in the following areas:
Development of a monitoring and evaluation plan as an integral part of the national strategic plan
Adaptation and use of revised recording and recording system-based hard and electronic tools. The latter is web-based version of TB surveillance, known as WEB TBS and its Access/Excel interfaces. These tools cover drug-sensitive and drug-resistant TB as well as national TB programme (NTP) and public and private non-NTP providers.
Capacity-strengthening in data management and data quality assurance including the general framework and tools for quarterly meetings and supervisory visits.
Capacity-strengthening in data analysis at national and subnational levels including the calculation of programme indicators, their interpretation, evaluating the epidemiological situation for TB control and taking the right public health actions based on the interpretation of programmatic data.
Monitoring and evaluation tools
Electronic Nominal Recording Reporting System (ENRS)
ENRS is the electronic version of the main four registers in the TB recording reporting system, where data entry is done on a nominal bases at TB management unit level using Excel. Nominal data are then processed to produce routine reports to the NTP and to calculate indicators using Excel and Access. Registers are linked using Access to produce a patient profile and back-up copy. TB drug consumption is calculated as well. The MDR register is used to generate MDR management reports. The system has data quality assurance sections and management sections.
The main characteristics of the ENRS are that it is:
Flexible: updating, adaptation and modification of the tool as per country need is feasible. Training of health staff is easy.
Available: Excel is everywhere.
Reliable: high quality data with tools of data verification, cleaning and confidentiality.
Comprehensive: the tool covers the four main TB registers.
Low cost: no need for special software, programming and human resources.
WEB-TBS
WEB TBS is a web-based surveillance system for TB. The online forms and generated reports are identical to the revised recording and reporting system in terms of the four registers (suspect, laboratory, TB and contacts) and their quarterly reports with the standard indicators.
Additional registers have also been introduced for multidrug resistant tuberculosis (MDR-TB), risk factors of TB, the reporting of TB suspects or notification of TB cases from non-NTP providers.
Further analysis can also be done using the exported data to Excel. The system covers the NTP and other sectors dealing with TB cases. The data submitted online are subjected to stringent data quality assurance criteria using internal and external data quality assurance mechanisms. Quality-assured data can be kept in the same database and as a back-up copy by exporting data to another Access database.
The system allows offline data entry on Access or Excel for settings with poor internet connectivity which will be uploaded on the system once net accessibility has been restored.
DOTS Online
DOTS Online, also known as DQ-online, is a quarterly report composed of five parts:
Introduction
DOTS coverage
Case notification
Sputum conversion
Treatment results.
The DOTS Online form can be used at all levels working in TB control from district level to Regional Office level.
The reports should reflect cohort analysis regarding the main two activities in TB control (case finding and case holding), and they should cover specific periods each quarter based on the duration of treatment, the treatment regimens used and the sequence of follow-up examination.
DOTS Online
Impact measurement
The goal of a TB surveillance system is to be able to provide data of sufficient quality and coverage so that:
TB incidence can be measured directly from TB notifications
TB mortality can be measured directly from vital registration records.
A global task force on impact measurement was established in 2006. The objectives of this task force are to:
produce robust, rigorous and widely-endorsed assessment of whether the 2015 targets for reduction in TB incidence, prevalence and mortality are achieved
regularly report on the progress towards these targets up to 2015
strengthen national capacity in monitoring and evaluation of TB control.
Areas of work of the global task force on impact measurement include:
Surveillance: the use of routine surveillance data to measure TB burden with particular emphasis on the certification of surveillance systems. Standards and benchmarks for certification of TB surveillance systems are under development by the task force.
Prevalence surveys: implementation of TB disease prevalence surveys in 21 focus countries. Pakistan is the only country in the Region included in this group and the survey results will be reported in 2012.
Estimates: periodic review of the methods used to produce estimates.Capture TB surveys were conducted in several countries of the Region: Djibouti, Egypt, Pakistan (pilot phase in 2008 and expanded study in 2011–2012)Syrian Arab Republic, and Yemen (pilot phase in 2008 and expanded study in 2010). These surveys aim to determine the proportion of under-reporting of TB cases in a representative sample of public and private non-NTP facilities. This information is then used to estimate the total number of existing TB cases using capture-recapture analysis. A capture TB guide is under development by the task force.
Monitoring and evaluation is an integral part of TB programme management and provides information on the scope, quality, scale/coverage, and success of programmes.
Monitoring generally refers to the routine collection of information across time and sites in order to track a programme’s ongoing activities. Policy-makers use monitoring to track key health-related indicators, often without attributing change to any particular programme or set of programmes.
Evaluation involves the assessment of programme implementation in order to determine the worth or value of a programme in terms of its success in achieving predetermined outcomes/goals. Evaluation is usually achieved through a detailed analysis of the programme’s process and outcomes or impacts. Evaluation lends itself to the linkage of outcomes to the programme process, as well as rule-out non-programme effects on outcomes.
Monitoring and evaluation framework
Adequate monitoring and evaluation can only be achieved through a well-defined plan incorporating a framework of the goal, service delivery areas and activities. Indicators are needed to measure the performance and progress achieved.
It is crucial that the framework indicators can be critically analyzed and interpreted at all levels and corrective action taken. Guidelines and standard operating procedures for data management, analysis and quality assurance should also be developed and adhered to.
Role of the Stop TB programme
The Stop TB programme monitors and evaluates the implementation of the different components of the Stop TB Strategy through a quarterly web-based online system, DOTS quarterly online (or DQ online).
The Stop TB programme provides technical support in the following areas:
Development of a monitoring and evaluation plan as an integral part of the national strategic plan
Adaptation and use of revised recording and recording system-based hard and electronic tools. The latter is web-based version of TB surveillance, known as WEB TBS and its Access/Excel interfaces. These tools cover drug-sensitive and drug-resistant TB as well as national TB programme (NTP) and public and private non-NTP providers.
Capacity-strengthening in data management and data quality assurance including the general framework and tools for quarterly meetings and supervisory visits.
Capacity-strengthening in data analysis at national and subnational levels including the calculation of programme indicators, their interpretation, evaluating the epidemiological situation for TB control and taking the right public health actions based on the interpretation of programmatic data.
Monitoring and evaluation tools
Electronic Nominal Recording Reporting System (ENRS)
ENRS is the electronic version of the main four registers in the TB recording reporting system, where data entry is done on a nominal bases at TB management unit level using Excel. Nominal data are then processed to produce routine reports to the NTP and to calculate indicators using Excel and Access. Registers are linked using Access to produce a patient profile and back-up copy. TB drug consumption is calculated as well. The MDR register is used to generate MDR management reports. The system has data quality assurance sections and management sections.
The main characteristics of the ENRS are that it is:
Flexible: updating, adaptation and modification of the tool as per country need is feasible. Training of health staff is easy.
Available: Excel is everywhere.
Reliable: high quality data with tools of data verification, cleaning and confidentiality.
Comprehensive: the tool covers the four main TB registers.
Low cost: no need for special software, programming and human resources.
WEB-TBS
WEB TBS is a web-based surveillance system for TB. The online forms and generated reports are identical to the revised recording and reporting system in terms of the four registers (suspect, laboratory, TB and contacts) and their quarterly reports with the standard indicators.
Additional registers have also been introduced for multidrug resistant tuberculosis (MDR-TB), risk factors of TB, the reporting of TB suspects or notification of TB cases from non-NTP providers.
Further analysis can also be done using the exported data to Excel. The system covers the NTP and other sectors dealing with TB cases. The data submitted online are subjected to stringent data quality assurance criteria using internal and external data quality assurance mechanisms. Quality-assured data can be kept in the same database and as a back-up copy by exporting data to another Access database.
The system allows offline data entry on Access or Excel for settings with poor internet connectivity which will be uploaded on the system once net accessibility has been restored.
DOTS Online
DOTS Online, also known as DQ-online, is a quarterly report composed of five parts:
Introduction
DOTS coverage
Case notification
Sputum conversion
Treatment results.
The DOTS Online form can be used at all levels working in TB control from district level to Regional Office level.
The reports should reflect cohort analysis regarding the main two activities in TB control (case finding and case holding), and they should cover specific periods each quarter based on the duration of treatment, the treatment regimens used and the sequence of follow-up examination.
DOTS Online
Impact measurement
The goal of a TB surveillance system is to be able to provide data of sufficient quality and coverage so that:
TB incidence can be measured directly from TB notifications
TB mortality can be measured directly from vital registration records.
A global task force on impact measurement was established in 2006. The objectives of this task force are to:
produce robust, rigorous and widely-endorsed assessment of whether the 2015 targets for reduction in TB incidence, prevalence and mortality are achieved
regularly report on the progress towards these targets up to 2015
strengthen national capacity in monitoring and evaluation of TB control.
Areas of work of the global task force on impact measurement include:
Surveillance: the use of routine surveillance data to measure TB burden with particular emphasis on the certification of surveillance systems. Standards and benchmarks for certification of TB surveillance systems are under development by the task force.
Prevalence surveys: implementation of TB disease prevalence surveys in 21 focus countries. Pakistan is the only country in the Region included in this group and the survey results will be reported in 2012.
Estimates: periodic review of the methods used to produce estimates.Capture TB surveys were conducted in several countries of the Region: Djibouti, Egypt, Pakistan (pilot phase in 2008 and expanded study in 2011–2012)Syrian Arab Republic, and Yemen (pilot phase in 2008 and expanded study in 2010). These surveys aim to determine the proportion of under-reporting of TB cases in a representative sample of public and private non-NTP facilities. This information is then used to estimate the total number of existing TB cases using capture-recapture analysis. A capture TB guide is under development by the task force.
