EMATM0067

Text Analytics Coursework

Spring 2025

Deadline: 13.00 on Monday 28th April

Overview

This coursework is worth 50% of the unit. It wiII take you through severaI text anaIytics tasks to give you experience with appIying and anaIysing the techniques taught during the Iabs and Iectures. The work wiII be assessed through both your code and your written report, in which you shouId aim to demonstrate your understanding of text anaIytics methods, evaIuate the methods criticaIIy and incorporate ideas from the Iectures.

We recommend that you first get a basic impIementation for aII parts of the required assignment, then start writing your report with some resuIts for aII tasks. You can then graduaIIy improve your impIementation and resuIts.

TotaI time required: 40 hours.

Support

The Iecturers and teaching assistants are avaiIabIe to provide cIarifications about what you are required to do for any part of the coursework. You can ask questions during our Iab sessions, post questions on MS Teams, or to the BIackboard discussion forum. If you don’t want to share your

question with the cIass, pIease contact Edwin by emaiI ([email protected]).

Part 1: Climate Sentiment – Jupyter Notebook (max 32%)

Many companies are required to pubIish (corporate discIosures’一 documents containing usefuI information about the business and its finances. The information in these discIosures is very usefuI for investors, reguIators and other stakehoIders. For exampIe, discIosures may present cIimate-reIated deveIopments as risks or opportunities for the business. In this part of the assignment, you wiII compare cIassifiers that cIassify the sentiment of a text as a cIimate risk, an opportunity, or neutraI. We wiII be working with the CIimateBERT dataset:Webersinke et aI., 2022.

Part 1 contains tasks 1.1 (13 marks), 1.2 (8 marks), and 1.3 (11 marks). Please see the accompanying Jupyter notebook text_analytics_part1.ipynb for detaiIs, which contains a series of tasks for you to compIete. Your answers to tasks 1.1, 1.2 and 1.3 shouId be saved in the notebook   itseIf, which you wiII need to submit. Submission detaiIs are in the notebook.

Task 2: Climate Sentiment – Report (max. 38%)

2.1. Present an evaIuation of the three methods impIemented in Part 1. Your evaIuation shouId be presented as the first page of your report. Your evaIuation shouId incIude the foIIowing points:

•     ExpIain the modifications you made to the naïve Bayes cIassifier in task 1.1c: what did you change, how does it heIp the cIassifier, and was there anything you tried that didn’t work? (3%, max. 150 words)

•     Present a comparison of results in a table or plot, along with your interpretation of how well each method worked. Your discussion should mention concepts from the lectures (e.g., transfer learning) and what could be improved in future work. To inform. this discussion, you may want to analyse some examples of misclassified texts. (10%).

2.2. Using the dataset, can you identify topics that are associated with climate-related risks or opportunities?

•     Explain the method you use to identify themes or topics. Make sure to motivate why you chose this approach and discuss its limitations.

•     It is important to test and compare different approaches to find out what works best for this dataset. Compare two variations of your method, e.g., by changing an important step or parameter.

•     Show your results (e.g., by listing or visualising topics associated with risks or opportunities).

• Interpret the results and summarise the limitations of your approach.    (25%)

Suggested length of report for task 2: 2.5 – 3 pages.

Task 2: Named Entity Recognition on Twitter (max. 30%)

Social media contains a wealth of information about public opinion and events, but this is often contained in unstructured text data. Your task is to build a tool for named entity recognition from Twitter posts that can help extract information about particular people, organisations and locations. To train and test the NER tagger, we will use the Broad Twitter Corpus (BTC) dataset, published by Derczynski et al., 2016.You can also find useful information on theHuggingFace dataset page.

2.1. Design and run a sequence tagger for the BTC dataset. Refer to the labs, lecture materials and textbook to identify a suitable method. You may choose any sequence tagging method you think is suitable, and you may wish to experiment with some variations in the choice of features or model  architecture to help justify your design. In your report:

• Briefly explain your chosen method and its main strengths and limitations.

•     If your model uses its own tokenizer, explain how you align the tokens with tags (this step is only needed if you use a neural sequence tagger that requires a particular tokenizer).

•     Show an example entity span from the dataset, that illustrates how entity spans are encoded as tags in this dataset.

•     Detail the features you have chosen, why you chose them, and hypothesise how your choice will affect your results.

• Higher marks are given for good, well-justified model design.    (17 marks)

2.2. Evaluate your method, then interpret and discuss your results. Include the following points:

• Explain your choice of performance metrics and their limitations.

• Describe the testing procedure (e.g., how you used each split of the dataset).

•     Show your results using suitable plots and/or tables.

•     Do your methods make any particular kinds of error? Show some examples of mislabelled sentences and suggest how the methods could be improved in future.   (13 marks)

Suggested length of report for task 2: 2 pages.

Implementation

The lab notebooks provide useful example Python code, which you may reuse. You may libraries introduced in the labs, or others of your choice. For tasks 2 and 3, you may write your code in either Jupyter notebooks or standard Python files.

Report Formatting

• Absolute maximum 5 pages

o   References do not count toward the page limit.

o  Aim for quality rather than quantity: you will receive higher marks if you write concisely.

•     To set the page layout, fonts, margins, etc., we recommend using the template from an academic conference, such as LREC-COLING 2024 if writing the report in Latex

o  You can use this template directly to write in Latex or follow the formatting style. in Word, Libreoffice, etc.

o   You don’t need to include an abstract or introduction or conclusion.

o   Please number your answers to each task clearly so that we can find them.

o   No less than 11pt font

o  Single line spacing

o  A4 page format

• The text in your figures must be big enough to read without zooming in.

Citations and References

Make sure to cite a relevant source when you introduce a method or discuss results from previous work. You can use the citation style. given in the LREC-COLING 2024 style. guide above. The details of  the cited papers must be given at the end in the references section (no page limits on the references list). Please only include papers that you discuss in the main body of the report.

Google Scholar and similar tools are useful for finding relevant papers. The‘cite’link provides bibtex code for use with latex and references that you can copy, but beware that this often contains errors.

Submission

•     Deadline for report + code: please see first page.

•     On Blackboard under the“assessment, submission and feedback”link.

Please upload the following three files:

1.   Your submission for task 1: please see the details in the Jupyter notebook. It should be submitted to the submission point“Text Analytics Part 1 Notebook”.

2.   Your report for tasks 2 and 3 as a PDF with filename .pdf, where

“”is repIaced by your student number from eVision (starting with a (2’, not your username).

o   UpIoad this to the submission point marked“Turnitin submission point - Text AnaIytics Coursework”.

o Please don’t include your name in the report itself: to ensure fairness, we mark the reports anonymousIy.

3.   Your code for tasks 2 and 3 a single zip file with filename .zip.

o   Inside the zip fiIe there shouId be a singIe foIder containing your code, with your student number as the foIder name.

o   PIease remove datasets and other Iarge fiIes to minimise the upIoad size.

o   UpIoad this fiIe to the submission point“Text AnaIytics Parts 2 & 3 Code”.

o   For tasks 2 and 3, your marks wiII be based on the contents of your report, rather than for good code structure or styIe. Assessment Criteria

To gain high marks, your report wiII need to demonstrate a thorough understanding of the tasks and the methods used, backed up by a cIear expIanation of your resuIts and anaIysis or errors. Marks wiII be awarded for appropriateIy incIuding concepts and techniques from the Iectures.