11 Chapter 11: Introduction to Modeling

Learning outcomes

At the end of this chapter, you should be able to

• Explain modeling principles
• List modeling techniques and tools
• Describe data modeling pitfalls
• Describe strengths and weaknesses of modeling techniques for decision making

Chapter content

Note: include practice and knowledge checks throughout chapter

From earlier material:

$$
PredictedFineAmount = 0.75 \times 600 + 1 \times 500 + 0.01 \times 55,000 = $1,500
$$

We only know if our prediction was good after we observe the actual fine amount. Over time as we get more new data we could evaluate how well our model worked.

– Model structure

All modelling approaches follow the same general approach. We propose a way to measure model fit, we propose a model that can capture patterns in the data, and we estimate parameters for the model to find the best fit based on the data. There are many modeling possibilities. Not all of these modeling approaches have a nice mathematical format. These modeling approaches might be called algorithms or methods for finding patterns in the data. However, there are still choices that must be made in how the algorithm identifies these patterns and the data informs these choices (i.e. parameters and hyper parameters). These algoritms without a particular mathematical form are often referred to as machine learning models. We will return to other models and algorithms later and discuss the structure they impose to find patterns.

– The risk from applying patterns to new data is significant and we use methods to try to reduce that risk

After using the training data to develop our models, we move to predicting outcomes. We take the same structure to new data where we do not observe the outcome to predict what we think it would be based on the model we estimate on the training data.

How well the training data model predicts outcomes on new data depends on many factors. Here I outline some of the biggest concerns.

– Under fitting

Some models may provide very poor measures of fit. In many cases model fit depends on the outcome being predicted. Because of differences across outcomes, what constitutes a good or a bad measure of fit depends on the outcome variable. Bad measures of fit can occur because the model parameters are not sufficiently tuned to best fit the data. Bad measures of fit can also occur because important predictor variables are missing from the model or the model structure cannot capture important variation in the outcome variables.

– Over training

Some models may provide good measures of fit on training data but then perform poorly on new data. This problem is typically referred to as overtraining although there may be various reasons why this may happen. Over training occurs when a model’s structure or parameters are fit so closely to the training data that they capture patterns in the data that are unlikely to occur in other data. Stated differently, over trained models capture random sample information that is unlikely to recur in other samples.

– Data shifting

Training data may differ from new data in important ways that limits how well a model fits new data. For example, a model fit on training data for large public companies may not apply to new data for small private companies. This problem may occur because the new data are not drawn from the same population as the training data.

– Data leakage

Models from training data may also fail when moving to new data for different reasons. For example, training data may use data that is not available for new data or the model may have included variables that are only available after observing the outcome variable.

## Return to Fraud Prediction Model

To complete the fraud prediction model example, here I will present a simple linear model based on the data we explored in the fraud example. This model tries to predict the probability that a company commits fraud in a given year. The details of how this model works are not important right now but will later become easier to understand as we model other outcomes. For now, I want you to see this piece of the process and to try to understand the intuition behind training and applying a model.

Here I will estimate a linear probability model. This model imposes a linear structure and uses a binary outcome variable.

“`{r,cache=TRUE,echo=FALSE,message=FALSE,warning=FALSE}
df2<-df %>%
select(misstate,at,bm,EBIT,ch_roa,dch_wc,ch_fcf)%>%
na.omit()
estlm<-lm(misstate~at+bm+EBIT+ch_roa+dch_wc+ch_fcf,data=df)
#library(modelsummary)
#modelsummary(estlm)
# would like to use this or something similar at some point…need updates
summary(estlm)
“`

There is a lot to potentially unpack here. Let’s focus only on a few things. First, the coefficient column gives the parallel to the $\beta$s mentioned earlier. These are the parameters that have been fit on the data to try to capture the patterns that occur in the training data. I’ll describe two pieces that are similar to what we saw before. The parameter for bm is positive 0.0089. This means that when bm increases by 1, the probability that the company-year is a fraud year increases by a rounded 1% (if we round to 0.01). The parameter (coefficient estimate in regression terms) on ch_fcf is -0.088. This means that when ch_fcf increases by 1, the probability that a company-year is a fraud year is approximately 9% lower.

We don’t need to go into the details at this point, but this model is not a particularly strong model for predicting fraud. In part, this is because fraud is an infrequent occurrence that is hard to predict, in part because we don’t have the strongest predictors of fraud, and perhaps in part because we have chosen the wrong structure for the model. We can take this model to new data to make a prediction for the probability that the observation has fraud.

The following observation comes from the data that has never been identified as having had a fraud.

“`{r, cache=TRUE,echo=FALSE,warning=FALSE,message=FALSE}
df2<-read.csv(“C:\\Users\\jgreen\\Documents\\Teaching\\NotesText\\Data Analytics with Accounting Data\\data_FraudDetection_JAR2020.csv”)
df2<-df2 %>%
group_by(gvkey) %>%
mutate(hasfraud = max(misstate)) %>%
ungroup() %>%
filter(hasfraud==0) %>%
select(fyear,misstate,at,bm,EBIT,ch_roa,dch_wc,ch_fcf)
df2<-df2[1,]

df2%>% kable(“html”,caption=”New data”)%>%kable_styling()
“`

“`{r, cache = TRUE, echo=FALSE, warning=FALSE, message=FALSE}
pred<-round(100*predict(estlm,df2),digits=0)
“`

We can use the same model (use the results from the regression above) to predict what the probability of this firm-year being a fraud. In this case, the probability would be `r pred`%.

There may be reasons to doubt this prediction, but based on the model we estimated, this is the prediction for the new data.

Note: one reason this is so high is because our model used only companies that at some point during the sample had been found to commit fraud (approximately 19% of the training sample was a fraud year). This new observation has never been identified as having committed fraud. Should the probability of a fraud year having been fraud be the same for this observation? Some research suggests that 10% of companies each year commit fraud [see here](https://link.springer.com/article/10.1007/s11142-022-09738-5?campaign_id=4&emc=edit_dk_20230114&instance_id=82723&nl=dealbook&regi_id=86591575&segment_id=122544&te=1&user_id=3a31c185ca43c0f889045eace6a8812b). As a practical reality, sometimes we can take estimates seriously and other times we have to use a different interpretation. For example, below is another new data point.

“`{r, cache=TRUE,echo=FALSE,warning=FALSE,message=FALSE}

df2b<-read.csv(“C:\\Users\\jgreen\\Documents\\Teaching\\NotesText\\Data Analytics with Accounting Data\\data_FraudDetection_JAR2020.csv”)
df2b<-df2b %>%
group_by(gvkey) %>%
mutate(hasfraud = max(misstate)) %>%
ungroup() %>%
filter(hasfraud==0) %>%
select(fyear,misstate,at,bm,EBIT,ch_roa,dch_wc,ch_fcf)
df2b<-df2b[25,]
df2b%>% kable(“html”,caption=”New data”)%>%kable_styling()
“`

“`{r, cache = TRUE, echo=FALSE, warning=FALSE, message=FALSE}
predb<-round(100*predict(estlm,df2b),digits=0)
“`

The prediction for this observation is `r predb`%. Notice the difference between the two is only 2% but the predictor variables are pretty different. Perhaps the only thing we can say is that the second observation might have a higher probability of being fraud.

<!–# Worthwhile to do a different model e.g. random forest like the paper for the data? –>

## Different application

### Generative AI text model

In this section we will discuss the modeling process for a different task, the task of creating and using a generative AI model. Generative AI refers to a model that generates new data. Most commonly this now refers to models that generate text. The same modelling process that we have discussed applies to generative AI text models like ChatGPT and CoPilot.

The first step for generative AI models is to train the model. This is done by providing the model with a large amount of text data. Some of these models presumably have been trained on terabytes of text data from every imaginable source. There are several challenges when working with text data that we will not go into here. However, to understand how the models work, we will use the simplest possible example. Let’s imagine that we start with the following sentences that we will use to create our training data. I will deliberately keep the example as simple as possible.

| Sentence |
|—————————————————————————–|
| Company ABC overstated its inventory and revenue from sales by \$3 million. |
| Company XYZ increased revenue from sales by 10%. |
| Sales revenue decreased compared with last year. |
| Every company in the industry paid the CEO a bonus. |

To model patterns in the training data, we would need to structure the data in a way that can be used an algorithm. Rather than work through setting up the data, let’s walk through the logic that an algorithm might use when trying to predict the next word in a sentence.

Suppose you give the model a word and ask it to predict the next word. If you provide “revenue” as the word, you can see in the training data that three of the four sentences contain the word revenue. In two of the three sentences the word after “revenue” is “from”. This means that the highest probability word to follow revenue is “from” based on the training data. The model would then predict “from” as the next word. Now the model has two words, “revenue from”. The model would then look at the training data to see what word follows “revenue from”. In this case, the word “sales” follows “revenue from” in every instance (2) where these two words appear together. The same process applies to the next word “by”. Now we have “revenue from sales by”. Now we have a problem because there is no higher probability next word. The model can choose either \$3 million or 10%. Now the model creator has to decide how to predict the next word. One option is to randomly choose between the two words. Another option is to force the model to consider additional words. This process continues until the model reaches a predetermined stopping criteria.

Let’s walk through a few things that we can learn from this example. First, a generative AI model depends crucially on the training data text it has used to create the model. Second, the model can only successfully predict on words or combinations of words for which it has sufficient training data to make a prediction. Third, the prediction is path dependent meaning that as it predicts the next word, the last prediction becomes part of the input for the next prediction. Fourth, the model creator has to make choices about how to predict the next word when the model has multiple options. Fifth, the model creator has to decide when to stop the prediction process. Note: we could also talk about the problem of predicting words when we think the words are from new data but they come from the training data ([see here](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4754678)).

### Fraud prediction model

Let’s use the example above to examine another application. Suppose we have a different type of model where the model is used to predict whether text has hints about whether a company commits fraud. We need training data again. This time the training data has labels for each sentence.

| Sentence | Label |
|—————————————————————————–|———–|
| Company ABC overstated its inventory and revenue from sales by \$3 million. | Fraud |
| Company XYZ increased revenue from sales by 10%. | Not Fraud |
| Sales revenue decreased compared with last year. | Not Fraud |
| Every company in the industry paid the CEO a bonus. | Fraud |

Models like one we might use here searches for individual words, combinations of words, grammatical structures, or other patterns that are best associated with the labeled observations. We can look at the observations labeled as fraud and compare those with the observations labeled as not fraud to get an idea for how an algorithm might work. We might see that “overstated” or “bonus” show up only in the “Fraud” observations. On the other hand “revenue” is in “Fraud” and “Non Fraud” observations. Perhaps then the model will predict that fraud is more likely when “overstated” or “bonus” appear in the text. However, “bonus” could appear in many other observations that are not fraud. The important challenge here is that the model crucially depends on the training data.

## Summary

In this chapter, we have introduced the modeling process. Creating models based on training data can help us identify patterns that can improve our predictions and decision making. Modeling data can have a wide range of applications. We have discussed fraud prediction and generative AI text models. The following points highlight the key concepts from this chapter.

– Modeling requires training data. The nature of the training data is important for the success of the model.
– Identifying patterns in the training data requires a model structure. The model structure is used to estimate parameters that best fit the data.
– Fitting a model requires a measure of fit. The measure of fit is used to determine how well the model captures patterns in the data.
– The model structure is used to predict outcomes on new data. The success of the model on new data depends on the similarity between the training data and the new data.

In the following chapters we will discuss how to apply these modeling concepts to accounting data, will also discuss some of the challenges when working with accounting data, and we will create models for various accounting related tasks.

## Review

### Conceptual questions

1. Explain the modeling process.
2. Write a summary in your own words for how modeling is used to find patterns in training data and to predict on new data.
3. What is the purpose of defining a measure of fit for modeling?
4. Why is a model structure necessary for finding patterns in training data?
5. What is the intuition for how a model is mathematically estimated?
6. Consider observable an attribute about a company that might be used to predict whether they have committed fraud. Why do you expect this attribute to be associated with fraud? How do you expect this attribute to be associated with fraud?
7. Why might a model for predicting fraud fail when making predictions on new data?
8. Describe how a generative AI model might be said to “understand” text.
9. Describe how a generative AI model trained on an accounting textbook might differ from a generative AI model trained on news articles if asked to determine using a company’s annual report whether the company is likely to have committed fraud.

### Practice questions

1. Statistical, econometric, and machine learning models are methods for \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_.
2. What is the primary assumption in the modeling process that is required for using a model to predict using new data?
3. In the fraud example, in order to create a model with the training data, we need two types of observations. What are these two types of observations?
4. In the fraud example, the mean for the column “misstate” is 0.2. What does this mean?
5. What is training data?
6. In the fraud example, fraud years have a ch_fcf of -0.1 and non-fraud years have a ch_fcf of -0.01. What can we infer from this information?
7. Why must predictor variables be available before the outcome variable or available when the outcome variable is not available?
8. Why is model fit necessary for training a model?
9. What is overtraining?
10. Name three reasons the fraud model in our example may perform poorly.
11. In the fraud example, we predicted a probability of being a fraud year to be 19% for a new observation. If that probability is too high, why might the model have predicted it to be so high?
12. Describe how a generative AI model predicts the next word.

## Solutions to practice questions

1. turning patterns in the data into specific insights.
2. That the patterns in the training data will apply to new data.
3. Observations with fraud and non-fraud labels.
4. That in the sample, 20% of the company-years were fraud years.
5. Training data is data that we use to find patterns that we can then use on new data to make predictions.
6. We might infer that companies are more likely to commit fraud when their cash flows are lower.
7. We make predictions on new data to try to predict what the outcome variable will be without knowing the outcome.
8. Model fit is used to find parameter values that best fit the training data.
9. Overtraining occurs when a model fits the training data so closely that it captures random sample information that is unlikely to recur in other samples.
10.
(1) Fraud is an infrequent occurrence that is hard to predict, (2) we don’t have the strongest predictors of fraud in the model, (3) we may have chosen the wrong (linear) structure for the model.
11. The model was trained on a sample in which the frequency of fraud was high.
12. A generative AI model predicts the next word by finding the highest probability word given the prompt word(s) based on the probabilities in the training data.

Tutorial video

Note: include practice and knowledge checks

Mini-case video

Note: include practice and knowledge checks