Precision
Screen6 uses probabilistic algorithms to match User IDs that belong to the same person. These matches should be as accurate as possible. When deterministic verification data, such as login IDs or email addresses, are available then the client and Screen6 are able to compute the accuracy of the verifiable part of the graph. This figure is called: precision.
How we compute precision
We compute precision and recall using deterministic data that clients provide to us. As the device graphs that we produce are private to the client, we are not able to use deterministic data collected elsewhere for verification.
Pairs and cluster based verification
We use two methods for computing precision: pairs based and cluster based.
The pairs based method looks at individual pairs of user IDs. A pair is labelled correct when both user IDs have the same deterministic ID; a pair is incorrect when the two user IDs have a different deterministic ID.
Precision = correctPairs / (correctPairs + incorrectPairs)
The cluster based method looks at all the user IDs belonging to a single cluster or Match ID. A cluster has been matched correctly when there’s only one Deterministic ID found for the cluster, and if that Deterministic ID is linked to at least two of its User IDs.
Precision = correctClusters / (correctClusters + incorrectClusters)
Cleaning verification data
Before calculating precision, the verification data is cleaned:
- User IDs, and their corresponding Deterministic IDs, that are not present in the source event data that was sent to Screen6 are discarded during verification
- User IDs should not get linked to more than one Deterministic ID. In such case the related Deterministic IDs are removed from the verification data
- A Deterministic ID should link at least two User IDs. Deterministic IDs that link to only one User ID may only point to an incorrect match but will never be able to confirm a correct match. As such they put an aggressive bias in the precision calculation
Inaccuracies in deterministic data
When computing precision, the provided deterministic data is considered to be a truth set. However it is important to point out that most of the time these data sets are not flawless themselves. For example someone may login on site 1 as johndoe@gmail,com and on site 2 as [email protected]. When Screen6 matches the corresponding User IDs the match is done correctly, but when computing precision the match will be deemed incorrect.
Recall and coverage
Recall indicates what percentage of matches in a deterministic verification data set has also been matched correctly in the device graph. Note that the same cleaning of the verification data should be applied as mentioned above.
Coverage doesn't relate to verification data. Coverage is the amount of User IDs or impressions that can be linked to a Match ID. Read more about Coverage here.