QBUS6840 Group Assignment

Key information

1.   Required submissions:

a. ONE written  report  (word  or  pdf  format,  through  Canvas-  Assignments- Report submission (group assignment)).

b. ONE code file (Jupyter Notebook “ .ipynb” or Python “.py”, through Canvas- Assignments- Code submission (group assignment)).

2. For the submission, each group should pick up a group representative who needs to

submit both files. Each group should only submit one report and one code file.

3.   Due date/time: Thursday 23 May 2024, 23:59 pm (Report and Code submission).

4. The late penalty for the assignment is 5% of the maximum mark per day. The closing date Sunday 2 June 2024, 23:59 pm is the last date on which an assignment will be accepted for marking.

5. Weight: 25% of the total mark of the unit.

6. The full marks of this group assignment are 65 marks, excluding the bonus marks.

a.   The maximum bonus marks based on the class forecasting competition are 3 marks.

b.   The maximum bonus marks for using Transformer for the forecasting task are 3 marks.

c.   Therefore, even if you receive a zero-bonus mark, your maximum possible    mark % for this assignment is still 100%. This makes the group assignment fair to all groups, then groups with good forecasting results or have successfully used Transformer can receive additional bonus marks as recognition of your quality work.

7. Groups: you should complete this group project in a group of four  students. You must follow the allocated groups on Canvas-People-QBUS6840 group page.

8. Presentation: please refer to the Presentation Instructions section of this  file for more detailed instructions, including the length requirement of the report, font size, etc. To facilitate your report writing process, a Report_Instructions.pdf file is also provided on Canvas.

9. Numbers with decimals should be reported to the four-decimal point.

10. Marking Criteria: please refer to the Marking Criteria section of this file for more detailed instructions.

11. Please include the name and student ID of all group members and group ID in the submitted report and code file. You do NOT need to include the cover page and table of content. The names of your report and code should follow the following formats   respectively,    by   replacing    "123"    with    your   group    ID.    Example: Group_123_Report, Group_123_Code.

Key rules

.   Carefully read requirements of the assignment.

.  Please follow any further instructions announced on Canvas.

.  You must use Python for the assignment.

.  If the training of your model involves generating random numbers, your Python code random seed must be fixed, by using np.random.seed(0).

.  Reproducibility is fundamental in data analysis, so that you will be required to submit a code file that generates your results. Not submitting your code will lead to a loss of 50% of the assignment marks.

.  Failure to read information and follow instructions may lead to a loss of marks. Furthermore, note that it is your responsibility to be informed of the University of Sydney and Business School rules and guidelines, and follow them.

.  Referencing:    Harvard    Referencing     System.    (You    may    find     the    details    at: http://libguides.library.usyd.edu.au/c.php?g=508212&p=3476130).

Background

The UnderEmployment rate is the number of underemployed people expressed as a proportion of the labour force. The underemployment refers to the condition in which people in a labor force are employed at less than full-time or regular jobs or at jobs inadequate with  respect to their training or economic needs. The underemployment rate is reported by the relevant government department in most countries. The underemployment rate can be used as an important indicator by the central bank of the country to determine the health of the economy when setting monetary policy.

Tasks and Datasets

For this group project, we have obtained the monthly historical underemployment rate data in a country (name omitted on purpose) from February  1978  to  December  2017,  as in dataset UnderemploymentRate_InSample.csv, which can be downloaded from the Canvas. The   dataset    contains    information   of   Date    (1/month/year,    so    monthly   data)    and underemployment Rate.

Your       task       is       to       develop        a       predictive       model,       trained       with UnderemploymentRate_InSample.csv, to forecast the monthly underemployment rate from January 2018 to December 2019. Note this is a 24-step-ahead forecasting task.

An out-of-sample test dataset which contains the true 2018 and 2019 underemployment rates, named UnderemploymentRate_OutofSample.csv in the same format as the in-sample data, is provided on Canvas. They will be used to assess the forecast accuracy of your produced models. Since you should assume the out-of-sample data is completely hidden from your model training/selection process, you must NOT use the out-of-sample test dataset in your model training/selection process. Otherwise, your model training process will be treated as having critical issues and you will receive significant mark deduction on the methodology and forecasting results, no matter how good your forecasting results are.

In  other words, the out-of-sample  test  dataset  should  be  only  used  to evaluate your forecast accuracy (details to be shown later).

Please note the assignment tasks are designed to be open-ended questions. This gives more freedom for you to explore a good solution and is similar to the situations that you might encounter in the real world.

You need to prepare a report for this assignment. The purpose of the report is to describe, explain, and justify your solutions with polished presentation. Be concise and objective. Find ways to say more with less.

You MUST submit your Python code which can be used to replicate the results in your report.   Please   note    even   if   you    fix   your   Python    code   random    seed   by   using np.random.seed(0),  changing  the  computer/CPU  could  have  impact  on  random  number generation  and  produce  slightly  different  results.  Please  note  the  key  target  of  having replicable results is to make sure that every group has genuine results reported. Therefore, if you have slightly different results for different runs/computers, it is fine. As long as the marker can re-run your code and have results that are close to yours, then it is fine.

Suggested Report Outline:

1. (2 marks) At the beginning (the first line) of your report, you should report your best out-of-sample forecasting result, by stating: “The best out-of-sample forecasting Root Mean Squared Error of our group is: ……”. Please note the markers will run your code and check whether your reported results can be produced/replicated.

Reporting false results deliberately can result in an up to 30% mark duction of the assignment marks.

2. (5  marks) Introduction.  Write a few paragraphs  stating the business problem  and summarising your works, etc. Use plain English and avoid technical language as much as possible in this section (it should be for the general audience).

3. (10 marks) Data pre-processing and exploratory data analysis (EDA). Write python program  to  clean  the  data,  e.g.,  checking/deleting  incomplete  information if any, making sure data is complete, or transforming the data if needed, etc. It is up to you on whether/how  to  transform.  the  data  so  that  the  resulting  dataset  can  be  well incorporated in training your chosen models.

Conduct initial analysis of the time series by plotting them or do what you can to reveal any patterns. Summarise what you have revealed or observed. In your report, carefully present your EDA procedure and findings, and discuss how the EDA results inform. you on the methodology section.

4. (40 marks) Methodology and forecasting results. In your report, you should present the details of your three best ranked models. The three models should be different types of models. For example, ARIMA(1,1,0) and ARIMA(2,0,1) are the same type of models. ARIMA and Seasonal ARIMA models will be counted as different types of models. Simple Exponential Smoothing, Trend Corrected Exponential Smoothing, Additive Seasonal Holt-Winters Smoothing and Multiplicative Seasonal Holt-Winters Smoothing models will be counted as different types of models. Neural Networks Autoregression and Recurrent Neural Networks (RNN) models will be counted as different types of models.

The details of the methodology/model should include: your rationale, how you train your  models,  model  selection  process,  some  interpretations,  your  findings  and justifications of your choices. You can try models that are not covered in our unit. However, for the three models presented, at least two models should be the models that we have covered in the lecture. The types of models could be the Moving Average, Decomposition method, Exponential Smoothing, ARIMA, Neural Networks Autoregression Model, RNN, Forecasting Simple Average, Forecasting Combination, etc. This choice is yours.

In particular, if you choose to use Transformer model and it ranks as one of your top three  performing  models  and  you  present  the  details  of  your  transformer  model implementation, then you will be potentially eligible for a maximum of 3 bonus marks. The validity of your modelling process of transformer will be still checked by the marker. Transformer architecture is a type of deep learning model introduced in the paper "Attention is All You Need" by Vaswani et al. in 2017 (paper attached on Canvas). ChatGPT refers to a Generative Pre-Trained model which is built on the transformer architecture. Transformer can be also used for the time series forecasting tasks. You may search online resources on using transformer for time series forecasting and see how to organize the given time series and fit it into the transformer  architecture to generate the required  forecasts.  Please note that using transformer is OPTIONAL for the assignment.

Below list contains some further clarifications on the methodology/modelling.

.    As mentioned above  "In your report, you should present the details of your three  best  ranked  models.",  while  in  the  assignment  working  process  in general you should try more than three models. This is because you need to provide rational and justifications on your choice, i.e., why do you initially choose to test these 5 or 10 models (rational)? Why do you finally decide to present these 3 models (justifications of your choices)? If I were to work on the assignment, I would try 5 or 10 or even more models and use the model selection  technique  (train/validation  split)  and/or  out-of-sample  forecasting results to decide my final three models.

.    Then in your report, you can present the details of the final three models and explain your whole assignment working process. Potentially you could also briefly include the working process and test RMSE values of other models that you have tried. By following this strategy, you provide strong rational and justifications on your final choice.

.    Rational  here means why this model is initially used/chosen. For example, suppose you have discovered some seasonality in the data with the EDA, then the rational here means you wanted to try  some models that  can  consider seasonality, i.e., rational means you have a decision in accordance with reason or  logic.  Then  in  the  rational  part,  you  could  mention  some  theoretical definition  with  mathematical  formula  of  this   seasonal  model,  i.e.,  how seasonality is modelled in the framework. You could also provide reason/logic on why you think this model could be a good candidate. Later, with the model training/selection and evaluation process, you can have further justifications on your choice.

.    If your  selected  model  does  not  require  a  model  selection  process, clear justifications on why this model is selected should be well documented. For example, based on the EDA, you can  argue that  additive HW exponential smoothing  model  is suitable for modelling the given time series.   Since additive HW exponential smoothing model has fixed model complexity, then you do not need to have the model selection process with train and validation split. However, if you choose additive HW exponential smoothing model and decided to do a train and validation split to evaluate   its   forecasting performance before the final out-of-sample testing, this is also fine.

.    If the selected model requires a model selection process, such as ARIMA or NN models or your other selected models, a formal model selection process must be implemented and well documented.

o For example, if your selected model is ARIMA or NN models, then you must have a model selection process with train and validation split.

o This means you need to select one ARIMA model from many potential ARIMA models with different lags (including seasonal ones), via using the optimal validation data performance (or criteria such as AIC/BIC). Do the same to select one NN model with different number of hidden layers and hidden neurons, and so on so forth.

o With  the  selected  ARIMA  model  specification/complexity  etc, re- train the selected model with the whole in-sample data and report its final out-of-sample forecasting RMSE.

o Always remember: "Since you should assume the out-of-sample data is completely hidden  from  your  model  training/selection  process, you must NOT use the out-of-sample test dataset in your model training/selection process.

Then you report the out-of-sample forecasting Root Mean Squared Error (RMSE) results of your three presented models. In particular, your best model’s out-of- sample RMSE forecasting  result  should be presented  at the beginning  of the report, as mentioned in the above point (1). This best model’s result will decide the forecast competition bonus marks for your group, refer to the Marking Criteria section later for more details.

Calculation of the out-of-sample forecasting results. You need to use your trained models to predict the 24 underemployment rates of 2018 and 2019. Please note that this is a 24–step-ahead forecasting task, since we assume you are in December 2017 (time stamp T) and have no knowledge about 2018 and 2019. Therefore, as mentioned you should assume the out-of-sample data is completely hidden from your model training/selection process, and you must NOT use the out-of-sample test dataset in your model training/selection process.

The 24 predicted values of the underemployment rate should be used to calculate the out-of-sample forecasting error. More specially, you need to use the Root Mean Squared Error (RMSE) to evaluate the forecast accuracy. The RMSE, computed on the out-of-sample data, is defined as follows. Let YT+ℎ|1:T  be the ℎ -step-ahead point forecast, based on the in-sample data Y1:T   = {Y1, Y2, Y3, … , YT − 1, YT }. The true ℎ -th underemployment rate value YT+ℎ  is included in the out-of-sample data UnderemploymentRate_OutofSample.csv. The out-of-sample RMSE is computed as follows:

here 24 is the number of observations in the out-of-sample data.

5. (3 marks) Final analysis, conclusion, limitations and future steps (non-technical).

6.   Appendix. In the appendix section, you MUST include three meeting minutes using the provided Minutes Template on Canvas. More detailed instructions are also given below. You can also put any other materials that you see appropriate into the Appendix section. The Appendix will NOT be counted into the length of the main report and there is no page limit for the Appendix.

Meeting Minutes

.    Your group is required to submit three meeting minutes which are to be attached to the report as the Appendix. Your group  should use the Minutes Template on Canvas for preparing agendas and meetings minutes.

.    Each minute should at least record the following information:

o Meeting dates/time/duration;

o Key points of the process of discussion, such as who said/did what;

o Action list, responsible member(s), task due time, etc. It is crucial that you clearly  document  the  actions  and  works  for  each  member  during  each meeting;

o Review/group      judgement       on      the       quality       of      individually completed/responsible  tasks.  The  purpose  of  this  is  to  infer  whether  a member is doing his/her share of work;

o The minute template contains some example input.

In case of a problem raised within a group, we will request minutes of all group meetings.   We will make an individual adjustment to the group mark, if there is sufficient evidence shows that a student has done significantly less works than other members. If a student has truly done very little work, a mark of 0 will be awarded for the student.

Marking Criteria

The full marks of this group project are 65 marks, including 60 marks for the report and 5 marks for the presentation. In addition, the maximum bonus marks based on the class forecast competition  are 3 marks. The maximum bonus marks for successfully using transformer (rank as one of the top 3 performing models) for the forecasting task are 3 marks. More details are shown below:

.    The content in your report Group_123_Report is worth 60  marks (with suggested report  structure  and  mark  break  down  as  above  in  the Suggest Report Outline section):

o Focus on the appropriateness of the chosen forecasting methods and provide full explanation and interpretation of any results you obtain in your report. Output without explanation will receive 0 marks.

o Describe your data analysis procedure in detail: how the data pre-processing is completed, how the EDA is done, what and why these models are used, how the models are trained, the model selection process, some interpretations, your findings and justifications of your choices. The descriptions should be detailed enough so that other data scientists, who are supposed to have background in your field, understand and are able to implement your work.

o Clearly and appropriately present any relevant graphs and tables.

o You  may  insert small section of  your  code  into  the  report  for  better interpretation when necessary.

.    The   Python    implementation.   The   main   program    file   should   be    named   as Group_ 123_code.ipynb  (or  Group_ 123_code.py).  Your  program  must be runnable and your out-of-sample forecasting RMSE results must be replicable. Reporting false results deliberately can result in an up to 30% mark deduction of the assignment marks.

The   idea   is    that,   when    the   marker   runs    your   Group_ 123_Code.ipynb    (or Group_ 123_Code.py),            with            the             in-sample            train            data UnderemploymentRate_InSample.csv and        out-of-sample         test        data UnderemploymentRate_OutofSample.csv in the same folder as the Python file, the marker expects to see the same (or at least close) out-of-sample RMSE value as you reported.  The  code  file  should  contain  sufficient  explanations  so  that  the  marker knows how to run your code.

.    Presentation  is  part   of  the   assessment.  The  marker  will  assign 5 marks for presentation.  The  detailed  instructions  are  shown  in  the   following  Presentation Instructions section.

.    We will allocate a maximum of 3 bonus marks (for each student in the group) for the forecast competition among the groups. Groups will receive marks according to the rank of your best  out-of-sample  forecast  RMSE value (the value that you reported at the beginning (the first line) of your report; the smaller the better), according to the following rules:

o If the out-of-sample forecast RMSE of your forecast is within top 5 percent in the class, then the full 3 bonus marks (for each student in the group) will be awarded;

o If the out-of-sample forecast RMSE of your forecast is between 5.1 percent (using one decimal rounding) and 20 percent in the class, then 2 bonus marks (for each student in the group) will be awarded;

o If the  out-of-sample  forecast  RMSE  of your  forecast  is between  20.1 percent (using one decimal rounding) and 40 percent in the class, then 1 bonus mark (for each student in the group) will be awarded;

o Otherwise, 0 bonus marks will be awarded.

.    We will allocate a maximum of 3 bonus marks (for each student in the group) for groups that successfully use transformer (rank as one of the top 3 performing models) for the forecasting task. The markers will check your implementation detail and decide the marks to be awarded.

Presentation Instructions

.    Your report should be provided as a word or pdf document.

.    Each group should submit one report and one code file by the group representative.

.    To facilitate your report writing process, a Report_Instructions.pdf file is provided.

.    The  report   should  be NOT more than 15 pages (excluding Appendix and

Reference list), with font size not smaller than 11pt. The page limit applies to all the content in your report, such as text, figures, tables, small sections of inserted codes, etc, but excluding the Appendix and Reference list. A violation of this rule will incur mark deduction on the presentation marks.

.    You do NOT need to include the cover page and table of content.

.    Numbers with decimals should be reported to the four-decimal point.

.    You report should:

o Include sections as suggested in Suggested Report Outline section.

o Include all the methodology details and steps as mentioned above.

o Demonstrate an understanding of the relevant principles of predictive analytics approaches used.

o Clearly and appropriately present any relevant figures and tables.

.    Your group is required to submit three meetings minutes. Your group should use the Minutes Template provided on Canvas to prepare agendas and meetings minutes. Not providing the meeting minutes will incur mark deduction on the presentation marks.

.    Later, the unit  coordinator will collect peer feedback on the performance of each group  member.  Therefore,  it  is  crucial  that  each  group  member  is contributing genuinely to the group assignment.