The computational part of the Garzoni project aims at extracting and automatically process information from records of apprenticeship contracts. As introduced in a previous post, our computational workflow consists in 4 main steps:
- Document acquisition: this step corresponds to the digitization of the documents;
- Data extraction: data model design and document annotation;
- Data processing: normalization, disambiguation and inference;
- Data exploitation: search and visualization.
This post focuses on step (3), with the description of an approach for entity record linkage, also known as entity disambiguation. The objective is to resolve person name ambiguity, more specifically to:
- identify and link entity mentions which refer to the same entity (i.e. identification of person name variants, case on the left in the figure below)
- distinguish identical entity mentions which refer to different entities (i.e. identification of homographic person names, case on the right)
This corresponds basically to a matching problem, where we want an algorithm to be able to decide if 2 entity mentions (in the text) refer to the same person (in the world) or not.
If we are able to solve this ambiguity, it will open up the possibilities to search for genuine individuals, to aggregate multi-source information on single entities, and to reconstruct (part of) the social network of Venice guilds at a certain point in time.
To this end, given a pair of person mentions on our documents, we need to compute their similarity in order to estimate the probability of their co-reference. This principle of mention comparison and similarity computation is illustrated in the following figure:
Compared to « traditional » Natural Language Processing approaches for entity linking, record linkage on historical data faces the following challenges:
- overcoming data sparsity: entity mentions have just a few features and those features are not evenly sparse;
- overcoming the absence of knowledge base: the linkage process cannot benefit from unique entities and their related information stored in a knowledge base but rely solely on what could be known from the data.
Our approach intends to overcome these difficulties. Given a manually annotated data-set of person mentions and entities, we consider and implement three principles:
- Working with dedicated similarity measures for each feature type
This is to find the correct match between 2 mentions.
For example, if we compare 2 person mentions having as features age, surname and year of contract, the age and year of contract features will be compared via a distance based on diminishing returns, whereas the surname feature will be compare via a string distance metrics.
2. Exploiting feature combinations
This resembles to creating rules and allows us to handle feature dependencies.
Additionally to feature similarity, the approach takes into account the information of feature combinations for some pairs of mentions are more likely to match when they shows certain feature combinations.
3. Filtering mentions according to activated features
Finally, we filter the pairs to compare based on the presence/absence of core features. This allows us to handle sparse features and to (drastically) reduce the search space.
Eventually, these 3 information are merged (see our publication for more details) and a final similarity score is computed.
Because the approach takes into account these 3 perspectives, it is possible to adjust those parameters and see which setting performs best.
We evaluated our approach in terms of precision and coverage. The main findings are the followings:
- performances are better when an equal importance is given to feature scores (point 1 above) and feature combinations (point 2);
- performances are better when filtering over sparse features is activated (point 3).
For more details please refer to the following publication:
Giovanni Colavizza, Maud Ehrmann, Yannick Rochat. Record Linkage with Sparse Historical Data. In: Proceedings of the Digital Humanities Conference, Kraków, Poland, 11-14 July 2016.
Author: Maud Ehrmann & Giovanni Colavizza.