Proceeding from the CSPN mission and from the approach declared in the PRSP of the Republic of Armenia that civil society subjects should also be involved into the PRSP evaluation and monitoring activity, it was decided by the Network to implement monitoring and evaluation pilot project of social and economic changes taking place in rural communities of Armenia during 2003-2004s.

The objective of Monitoring and Evaluation Pilot Project is the following:

Evaluation and monitoring of the current situation and its change with physical and financial affordability of primary healthcare and irrigation water of rural communities

Selection of spheres was stipulated by the fact that according to the CSPN member-organizations, particularly proceeding from the experience of “Against Legal Arbitrariness” NGO, one of the most important factors conditioning social and economic development of rural communities is the affordability of irrigation water and the affordability of primary healthcare is the social service of utmost importance about which the population of rural communities are concerned. From the other hand the issues of primary healthcare have always been among the priorities of CSPN member-organizations and particularly Armenian Office of OXFAM GB.

Auxiliary objectives of the Project are as follows:

1. To promote effectiveness of PRSP implementation submitting the results of monitoring and evaluation project to all the organizations and persons involved in PRSP implementation.

2. Acquisition of monitoring and evaluation project implementation experience by CSPN member-organizations.

3. Application of monitoring and evaluation results for enhancing the efficiency of community supporting projects implemented by CSPN.

Project implementing organizations and project implementation territories

Under general supervision of the Armenian Office of OXFAM GB, monitoring and evaluation joint project is being implemented by CSPN members. CSPN selected four villages from each RoA marzes of Shirak, Vaiots Dzor and Sjunik to carry out qualitative evaluation and monitoring. In each marz the project is implemented by one monitor organization (Table 2).

Table 2: Territories of qualitative monitoring and evaluation and monitor organizations.

In addition to the above noted marzes, quantitative evaluation shall be carried out in Tavoush Marz too.

Project implementation methodology

Methodology forming the principles of project implementation are derived from the basic and auxiliary objectives of the project as well as the following subprovisions:

1. Quality of monitoring and evaluation activities directly depends on the level of knowledge of the object to be monitored and evaluated by the implementing subject.

2. Monitoring and evaluation shall be implemented by scientific methods.:

3. Monitoring and evaluation results shall be open for all the organizations and persons concerned regardless their legal status.

4. Monitoring and evaluation activities should have participation nature.

It is not difficult to note that paras 2 and 4 of the subprovisions have some inner contradiction the solution of which has also conditioned certain peculiarities of project methodology

Choice of monitoring and evaluation project methodology is based also on the following basic provisions relating to the objects to be monitored.

The spheres of irrigation water use and primary healthcare under consideration are sophisticated hierarchic systems and consequently are not subject to complete evaluation and monitoring by non of the pilot projects. Therefore in the systems under consideration, as monitoring and evaluation objects, have been chosen those pivotal elements and subsystems the evaluation and monitoring of activities of which are rather impressive to build up, under limited resources, general (but not complete) picture of the activities of these systems

In each system three elements (subsystems) are selected.

1. Legal-normative acts regulating the sphere
This subsystem defines the state ideal of sphere activities.Consequently, the study of legal-normative subsystems of the system gives the majority of those marginal values through which operational parameters of the sphere shall be evaluated.

The next two subsystems are chosen to evaluate and monitor the efficiency of the systems. Therefore it is essential to consider the contact points where the acts of providing services take place. These are:

2. Final Beneficiaries of the sphere. This is the quantity of those elements of the systems for certain requirements of which those systems are built up, i.e. the population of communities. The level of satisfaction of beneficiaries requirements significantly defines the efficiency of system activities.

3. In the hierarchy of systems under consideration, the subsystem through which sphere governing system directly contacts its final beneficiaries. In this case community health post staff, members of local governing bodies, employees of water use organizations.

It is necessary to conduct double study of contact points to reduce information subjectivism received from the subsystems forming it based on sphere regulating legal-normative acts.

The Chart of implementing pilot project based on above noted sub and basic provisions is given in Table 1.

Monitoring and evaluation project consists of three interrelated processes:

1. Legal and expert evaluation and monitoring. In this process it is carried out the expert evaluation of legal-normative environment of the activities of the systems under consideration and the monitoring of their changes.

2. Qualitative evaluation and monitoring. In this process it is considered the contact point of the system, from one side water consuming community members or those receiving primary healthcare and from the other side the organizations providing the consumers with irrigation water, i.e.companies, unions of water users, local governing bodies or other subjects (if there are such), health post staff.

3. Quantitative evaluation. This process is used for generalization of the results of qualitative evaluation and monitoring with sufficiently high statistical significance. While creating the instrument of this process the pivotal provisions of legal and expert evaluation are take into consideration.

These process and their separate components shall be considered further.

Legal and expert evaluation and monitoring

Is carried out tree times during the project implementation.
The goals of irrigation water and primary healthcare legal environment study are as follows:
1. Stating the initial situation of sphere changes monitoring.
2. Elaboration of adequate instruments for realizing other components of the project.
3. Upgrading the knowledge of project implementing persons in regard to the sphere and legal mechanisms of its activities.
4. Raising of the awareness of civil society, including the beneficiaries of the sphere, in regard to the issues of legal regulation of the spheres.

The second legal and expert evaluation shall be made at the beginning of the second half of the project duration (in around Spring of 2004). The necessity of second evaluation is caused by the adoption of the RoA PRSP. The implementation of PRSP shall start in 2004. These circumstances suppose intensification of solution of legal regulation matters relating to social services as well as different social and economic spheres. Consequently during implementation of monitoring and evaluation project it is not excluded rapid and significant modification of legal regulation environment of the spheres under consideration. Thus in order to ensure the quality of monitoring and evaluation project it is necessary to have monitoring point of legal environment modification for evaluation of the process of project and its final clarification.

The final legal and expert evaluation is mandatory as necessary component of evaluation procedure which supposes comparison of the object situations at the beginning and end of evaluation period. In addition, in case of pilot project continuity it will be converted into the estimate of initial situation evaluation of new project.

Qualitative monitoring and evaluation

It consists of three main phases: preliminary qualitative evaluation, monitoring, final qualitative evaluation.

Preliminary familiarization

It consists of subphases of preliminary familiarization, focus group studies, community generalization of results, marz generalization of results

The goals of preliminary familiarization phase are:

1. preliminary general familiarization of project implementing organizations with general social and economic conditions and problems of target communities through which internal environment of spheres subject to monitoring and evaluation is built. This phase is mainly realized by participation of CSPN members.

2. Formation of positive cooperation between the community population and local governing bodies which is mandatory condition for successful implementation of both the projects under consideration and for any other projects (see subprovision 4).

3 Collection of preliminary information for exercising different activities in given communities (not only monitoring and evaluation but other different support projects are implemented in CSPN target communities).

The goal of focus group studies is the receipt of necessary information precisely for preliminary qualitative evaluation. Focus groups will be studied in different groups of community population under consideration which represents the study of beneficiaries of the sphere as well as in the groups of community health post staff, local governing body members, representatives of irrigation water providing organizations which are the representatives of those subsystems through which spheres governing systems immediately contact the final beneficiaries. The studies are carried out by invited relevant specialists.

Community generalization and Marz generalization represent the analytical phases of preliminary evaluation phase. It is realized by invited relevant specialists (see subprovision 2) whose assistants can be CSPN organizations members.

Monitoring

It consists of four similar phases separated from each other with equal time periods.

Monitoring is the panel study of 12 community population which is carried out proceeding from the situation and most important issues revealed during the preliminary qualitative evaluation of the spheres under consideration as well as the pivotal information revealed in the result of legal and expert evaluation.
The goals of monitoring are:

1. Consideration of affordability parameter changes of irrigation water and primary healthcare within the time in regard to different strata of population.
2. Consideration of legal and normative acts and the impact ot their changes on affordability and quality of these spheres in the communities;
3. Preliminary identification of factors affecting affordability and quality of spheres under consideration

Monitoring shall be made by interviews applying structured questionnaires and by conducting profound interviews in regard to certain issues. Through this approach two problems are solved, during monitoring both qualitative and quantitative data are obtained and besides this the application of structured questionnaire ensures the minimum necessary quality of collected information under the conditions of different qualifications of the persons collecting information. Monitoring is carried out by the specialists and CSPN organization members (contribute to the solution of contradiction between 2 and 4 subprovisions).

Final qualitative evaluation

Final qualitative evaluation is the necessary component of evaluation procedure. From methodological viewpoint it is identical with preliminary qualitative evaluation where preliminary familiarization component or its analogue is naturally lacking.:

The goals of this phase are as follows:

1. Evaluation of changes between the initial and final conditions of the spheres under consideration.
2. Discovery and evaluation of the factors affected the changes.
Quantitative evaluation

The goals of this process are the following:

1. Statistically reliable generalization of results of qualitative evaluation and monitoring phases in the marzes of Shirak, Sjunik, Vaiots Dzor and Tavoush;
2. Identification of the factors affecting affordability of the spheres under consideration and comparative evaluation of their impacts;
3. Community typification according to the affordability factors of spheres under consideration;
4. Evaluation of anticipated impact due to the changes of factors revealed during the governing of spheres under consideration.

Proceeding from the provision of PRSP stating that

“45. Under the conditions of Poverty Evaluation Indicators System (PEIS) absence, the elaboration and evaluation of PRSP is forcedly based on poverty evaluation methodology applied by the RoA National Statistic Department where the material side of poverty is accentuated”
and ¨ in reply to the proposal of PRSP implementation responsible bodies to participate in elaboration of poverty evaluation indicators system, within the framework of SCPN it has been worked out the method of financial affordability evaluation of social spheres.
The goals of quantitative study are as follows:
Testing of new methodology in the spheres of irrigation water and primary healthcare financial affordability.
Quantitative study is composed of two phases: a) pilot phase during which the survey tool shall be improved and b) main study shall be carried out.
Project generalization

At the end of the project it will be carried out legal and expert, qualitative and contitative monitoring and project generalization of evaluation.
Internet promulgation of results

The results arising along with project implementation shall be promulgated in this Internet page.
For further information you are free to contact Armenian Office of OXFAM GB.

Republic of Armenia
Yerevan 375002
Gh. Parpetsi str. 20, apt 4
Tel.: 53 47 60, 53 84 18
E-mail: anavasardyan@oxfam.org.uk