CS198 RESEARCH PROPOSAL

TECS: Test Essay Checking Software

I. I NTRODUCTION Essay-type examination is a useful evaluation tool to fully capture a students knowledge on a certain topic. Studentwritten essays are usually scored by a well-dened rubric as basis. However, evaluating these essays can consume a huge amount of time and effort. Teachers are challenged to be consistent on grading essays to ensure no bias and same standards are applied in all essays especially when there is a pile of essays to check. Moreover, scores of human raters can be affected by certain factors like fatigue, mood, and distractions. Large-scale assessments are usually limited to multiplechoice question type. In some cases, response items like essays are already included. Considering the large number of essays, multiple human-raters are usually needed to score all responses. This raises another issue on the homogeneity of the scores given by different raters. Essays are platforms for students to demonstrate their critical thinking and deep understanding of a material. To highly reduce the cost by delivering a fast and accurate evaluation of multiple essays, automated essay scoring is introduced. The teacher trains the machine on grading essays then the machine learns to produce scores similar of that teacher. This study has the following objectives: Provide a fast and more accurate essay scoring by eliminating human rater fatigue and inter-rater reliability issue Highly reduce cost of effort and time consumption for grading multiple essays Able to give an instant feedback for an essay Provide a fast and memory-efcient algorithm for essay grading Introduce a breakthrough in education II. R EVIEW OF R ELATED L ITERATURE Extensive research and development efforts to produce applications with the potential to dramatically reduce the difculties associated with scoring of essays. If this will become successful, it will assist teachers and testing companies from low to high stakes assessment and from small to large scale assessment. Over the years, a lot of approaches to the automated assessment of free text answers are developed. The following systems are such: Project Essay Grade (PEG), Intelligent Essay Assessor (IEA), Educational Testing service I, Electronic Essay Rater (E-Rater), C-Rater, BETSY, Intelligent

Essay Marking System, SEAR, Paperless School free text Marking Engine, WriteToLearn and Automark. Several of these, used standard text-categorization techniques like Bayesian independence classiers and k -nearestneighbor classiers. These were trained to assign scores to manually-graded essays. Other method is Latent Semantic Indexing (LSI). It is an information retrieval technique using a low-rank singular value decomposition (SVD) of the term-document matrix. In some ndings, LSA measures word order as closely related to human judgments, and it predicted external measures of the same knowledge as well or better than did the human judgments. Although the LSI method has empirical success, it suffers from the lack of interpretation for the low-rank approximation and, consequently, the lack of controls for accomplishing specic tasks in information retrieval. From past studies and deveoplment, researchers used centroids of clusters or so called concept decomposition for lowering the rank of the term-document matrix that improves the LSI method.[?] From 2010, a new approach has been developed using the dimensionality reduction algorithm called Concept Indexing (CI). In experiments that were conducted, both versions of CI outperform LSI in Exact Agreement Accuracy and Pearsons Product-Moment Correlation Coefcient. In particular, CI with Fuzzy C-means achieves better results than CI with Kmeans in essay content evaluation, and CI with Fuzzy Cmeans is less sensitive to the preprocessing techniques such as stemming and stopwords removal and eventually, to noise. [1]

CS198 RESEARCH PROPOSAL

Fig. 1: Entity Relationship Diagram

III. M ETHODOLOGY A. Procedures and Algorithms TECS essay evaluation will consist of a spell check, grammar check and content analysis. The proponents are planning to use available open source spell checker and grammar checker on the web such as GNU Aspell and LanguageTool. For content analysis, the algorithm to be used in this study is Concept Indexing (CI). This is discussed in the paper Automated Essay Content Analysis based on Concept Indexing with Fuzzy C-means Clustering which is elaborated in the previous section. This new method is more effective than Latent Semantic Indexing (LSI), a well-known algorithm for content analysis, since it yielded better results on the experimentation. It is also more efcient since it does not need to perform Singular Value Decomposition (SVD) anymore which is memory-consuming. This study will follow an Agile software development process that is very exible to changes which is perfect for new algorithm implementation and system development projects such as this. B. Equipment Specications The proponents will be needing a subscription to a web hosting service (approx. P200.00 per month) or a dedicated server with 2-4GB RAM (approx. P6,000.00 per month) for the website. The server shall run Linux and use PostgreSQL. The languages that TECS will use will be PHP for the back end, HTML, CSS, and JavaScript for the frontend of the

program. Additionally, we will use YII(Yeah It Is) for the framework and apache as the server. C. Entity Relationship Diagram The above ERD consists the following entities:

USERS: The users has an attribute of rst name, last name, middle name, andusername as seen in the picture above. The role has for types: 0 for admin, 1 for school administrator, 2 teacher, and 3 for the students. The slug eld is composed of characters that will be sent to his/her mail for verication of accounts, and veried eld is to determine if the users already veried his/her account. BASIC INFO: This consists of users mobile number and his/her address: the street, municipality and province. CLASS INFO: What we did for the class name and class key, we separate it from the class list of a teacher. By this way, we can get rid of redundant data that the teacher will do. The user id eld is for the id of the teacher who created the class. CLASS: This table is for the students who will enroll for the class. It stores the user id of the students, and the class id of the class that was made by the teacher. EXAMS INFO: This is where the essays of the students will be saved, and the date when the student passed his/her essays. RESULT: It consists of the nale score given to the essay that has been passed by the students. It also has the feedback of the teacher if he/she wants to put comments

CS198 RESEARCH PROPOSAL

in the essay of a particular student. And the exam id is from exam info where the body of essay resides. EXAMS: This stores the user id of the student who passed an essay, an exam id where the body of essay resides, and the class id where the student is enrolled.

D. Data Collection and Analysis For data collection, the proponents will ask at least four teachers in elementary or in high school for their evaluation on at least 50 student essays each. We shall have two sets of essays: one will consist of technical or objective essay and the other set will consist of creative or subjective essays. Their evaluations results will then be compared to TECS and will be subjected to Pearsons r for the analysis of their correlation. For additional training and testing data set, the proponents will also use essays from Kaggle. R EFERENCES
[1] A. Razon, M. Vargas, R. Guevara and P. Naval, Automated Essay Content Analysis based on Concept Indexing with Fuzzy C-means Clustering, Circuits and Systems (APCCAS), 2010 IEEE Asia Pacic Conference on,2010, pp. 1167-1170. [2] S. Dikli. An Overview of Automated Scoring of Essays, Journal of Technology, Learning, and Assessment, 5(1), 2006. Retrieved June 2013 from http://www.jtla.org.