Conditional random fields

Conditional Random Fields
Andrea Passerini [e-mail]
Complex Systems

Conditional Random Fields

Generative vs discriminative models joint distributions Traditional graphical models (both BN and MN) model joint probability distributions p(x, y) In many situations we know in advance which variables will be observed, and which will need to be predicted (i.e. x vs y) Hidden Markov Models (as a special case of BN) also model joint probabilities of states and observations, even if they are often used to estimate the most probable sequence of states y given the observations x A problem with joint distributions is that they need to explicitly model the probability of x, which can be quite complex (e.g. a textual document)

Conditional Random Fields

Generative vs discriminative models y y1 y2 y n−1 yn yn+1

x1

xi

xn

x1

x2

xn−1

xn

xn+1

Naive Bayes

Hidden Markov Model

generative models Directed graphical models are called generative when the joint probability decouples as p(x, y) = p(x|y)p(y) The dependencies between input and output are only from the latter to the former: the output generates the input Naive Bayes classifiers and Hidden Markov Models are both generative models

Conditional Random Fields

Generative vs discriminative models

Discriminative models If the purpose is choosing the most probable configuration for the output variables, we can directly model the conditional probability of the output given the input: p(y|x) The parameters of such distribution have higher freedom wrt those of the full p(x, y), as p(x) is not modelled This allows to effectively exploit the structure of x without modelling the interactions between its parts, but only those with the output Such models are called discriminative as they aim at modeling the discrimination between different outputs

Conditional Random Fields

Conditional Random Fields (CRF, Lafferty et al. 2001)

Definition Conditional random fields are