Predicting Age and Gender Online


Note: This was originally published as a marketing paper.

NextStage Evolution’s Evolution TechnologyTM (ET) is a highly disruptive technology powerful enough to fund a new industry – Jim Sterne

Third party independent analysts were hired in April 2009. The purpose of the test was to determine Evolution TechnologyTM‘s ability to determine a website visitor’s Age&Gender by analyzing mouse movements alone.

Background

Most people in the web analytics and marketing communities and certainly those who’ve attended eMetrics conferences for the past few years know of Joseph Carrabis, the company he founded in 2001, NextStage Evolution, and the claims made about his patented Evolution TechnologyTM[[we have several Evolution Technology or “ET” based patents now]]. Not only were the claims “otherworldly” — the ability to determine a visitor’s age, gender, their purchasing behaviors, what types of marketing will cause them to act favorably and nonfavorably, for example — they dealt with areas so foreign to people trained in classic web analytics and marketing that blind acceptance wasn’t possible.

For example, classic web analytics tools typically measure anonymous user behaviors via cookie-based systems. Every click is recorded and tied back to an anonymous cookie, which then represents the best approximation for a unique visitor. The challenges presented by such systems — and that NextStage claimed to overcome — included

  • The failure of cookie based visitor detection methods to guarantee that each session was unique to a given individual.
  • The failure to tie one set of unique cookies to a one unique individual.
  • The failure to recognize and respond to individuals as the sum of their experiences.
People bring the sum of all their past experiences to each new experience they have

That last bullet, the “sum of their experiences” part, is crucial because it is at the heart of true one-to-one relationships. Everyone claims to want to build relationships with their customers and visitors and few appreciate that doing so requires the ability to recognize that each individual responds based on their personal experiences. Not just their experiences at the moment they’re on the page, but at this moment based on all such moments. Visitors interaction with a website is based on their anxieties, their joys, their curiosities, their fatigues, their exasperations, their desires, their frustrations and more. Knowledge of and understanding these visitor elements is not possible through classical web analytics&survey tools.

Unlike classical web analytics tools, including declarative surveys, ET determines these factors by analyzing psychomotor behavioral cues that have been collectively recognized as indicative of a given user’s non-conscious, cognitive, behavioral/effective and motivational processes and methodologies – collectively called the “{C,B/e,M} matrix”.

Testing What You Can’t See

Peers in the industry have logically questioned the accuracy of this highly disruptive technology. NextStage chose to answer these questions by hiring 3rd party firms to conduct a series of tests of ET’s accuracy in both determining visitor metrics and predicting visitor actions – what action items visitors would choose from all screen options available. This paper focuses on tests performed to determine ET’s accuracy determining two visitor dominant visitor metrics – Age&Gender. Future papers will focus on ET’s predictive abilities.[[ there are lots of papers on this now. Check Joseph’s LinkedIn profile for a list]]

The following details the actual test and protocol that were followed (the complete research paper. [[NextStage Members can access the full research paper, Machine Detection of Visitor Age and Gender via Analysis of Psychomotor Behavioral Cues, on the Members’ Papers page.]]

The test itself was performed using the latest methods developed by social scientists. The internet provides social scientists with access to much larger populations for their studies. The bad side of this is that much of the baggage from traditional; non-internet survey methods have been carried onto this new medium.

The ideal would be to have a research tool that acts over the internet much like the anthropologist’s blind (gathering metrics non-invasively, totally passively, never interfering in the tests subject’s experience, never asking questions, never intruding on the subject in the act of being his/herself, monitoring and reporting how people interact with the information), within conformity of W3C standards.

Abstract

This paper describes a test that focuses on proving the accuracy of Age&Gender determination through the use of NextStage’s Evolution Technology, commonly known as “ET”. The determination of human Age and Gender are neurological and based entirely on human interaction with the machine.

Tests were performed to determine Age&Gender using ET. The results of ET’s Age&Gender determinations were then compared to the actual Age&Gender of the individuals involved in the test by an independent 3rd party. Accuracy of Age prediction was 98%, with an average age estimate error of roughly 3.18 years (well within the ±the 2s of 14.96 years). Accuracy of Gender prediction was 99.3%.

The Test&Protocol details

The initial conditions for this test were as follows:

  1. Four websites were used for the test: A, B, C and D
  2. All four websites were designed by NextStage Evolution (NSE)
  3. The first website was maintained&housed by NSE. The other three by an independent marketing consultant (MC)
  4. B was used to gather small world ControlGroup data (20 individuals). It’s population consisted of individuals both known to and selected by NSE, and was used to confirm and proof the test methodology
  5. C was used to gather large world ControlGroup data (300 individuals). It’s population consisted of individuals comprising a marketing research panel
  6. D was used to gather TestGroup data. It’s population also consisted of individuals (300) comprising a marketing research panel that was wholly separate and unique from the C group
  7. These marketing research panels were selected through a third-party marketing research firm (MRF) contracted by the independent marketing consultant (MC)
  8. Only the Marketing Research Firm (MRF) and the Marketing Consultant (MC) knew the D TestGroup and C large world ControlGroup’s exact demographics
  9. NSE had no contact with the MRF or any of the marketing research panels
  10. NSE had no knowledge of the D Test- or C large world ControlGroup’s exact demographics other than the selection criteria below:
    1. 17-75 years old
    2. Mixed male&female
    3. Diverse income groups
    4. Diverse ethnicity
    5. Continental USA geographic locations
  11. The MRF had knowledge of the C and D questionnaire elements but not its intentions
  12. The Web page navigation for all groups was through a four page solicitation form:
    1. An Introductory page followed by
    2. A 1st level Survey page followed by
    3. A 2nd level Survey page followed by
    4. A Thank You page
  13. The purpose of the individual pages was as follows:
    1. The Introductory page was designed to create a baseline for psychomotor behavioral measurement and determine a “neutral” {C, B/e, M} matrix for each user
    2. The 1st level Survey page was designed to force specific personality aspects to dominate non-conscious neural activity
    3. The 2nd level Survey page was designed to demonstrate those specific personality aspects of non-conscious neural activity
    4. The Thank You page was designed to return visitors to their neutral {C, B/e, M} matrix states.

The test went as follows:

  1. Groups B, C and D) interacted with identical Introductory pages
  2. 1st level survey page:
    1. The entirety of the large world ControlGroup (C) and select members of the small world ControlGroup (B) interacted with the 1st level Survey page. This page included two questions (age, gender) in order to determine self-identification error rates for the final determination
    2. The entirety of the TestGroup (D) and the other part of the select members of the small world ControlGroup (B) interacted with the 1st level Survey page without the two age&gender questions
  3. 2nd level survey page:
    1. The entirety of the large world ControlGroup (C) and select members of the small world ControlGroup (B) interacted with the 2nd level Survey page
    2. The entirety of the TestGroup (D) and the other part of the select members of the small world ControlGroup (B) interacted with the 2nd level Survey page
  4. Groups B, C and D interacted with identical Thank You pages

Collection and Analysis went as follows:

  1. All NSE access to C and D was removed before the study went live
  2. All C and D questionnaire results data, along with all marketing research panel demographic data, were maintained by the independent Marketing Consultant (MC)
  3. NSE delivered neurological age and gender predictions based on the D data collected, as defined by NSE’s Evolution Technology (ET) to an independent auditor for evaluation
  4. The MC delivered the D data to the auditor
  5. The auditor determined valid and invalid data points via a matched-pair algorithm between ET’s D gender and age predictions to the known D demographic data supplied by the MC
  6. The auditor determined the accuracy of ET’s D gender and age predictions of collected data to the known D demographic data supplied by the MC
  7. The auditor then delivered the accuracy determinations to NextStage Analytics (NSA)
  8. Internal Controls on the TestGroup (D):
    1. After delivering the D data to the auditor, the MC delivered the C and D data to NSE
    2. NSE selected random D data points and replaced them with ten randomly selected C data points to provide internal controls to the TestGroup D data

The results

The auditor compared the results as described in the point #20 and provided us with his conclusions. In his own words:

I’ve been contracted by NextStage Evolution to perform an audit on the results of a test to determine accuracy of its age and gender determination using Evolution Technology.

I used the prediction numbers NextStage Evolution sent to me originally, along with the project director’s survey results that came thereafter. There was something meaningful in my mind to receive the prediction
before the project director released the answers.

  • I found only 2 gender errors in 300 predictions.
  • I found only 6 age range errors in 300 predictions.
  • Twenty-two of the age predictions were absolutely correct. That data is faulty because it seemed about half of the predictions were made with fractions, but none of the data was gathered with fractions, so half of the predictions were incapable of being absolutely correct.
  • 36 age predictions were within 0.5 years of the age given.
  • 45 age predictions were less than 1 year from the age given.
  • 74 age predictions were within one year of the age given.
  • Overall, the age predictions were within 3.18 years of the age given.

With only 8 errors in 600 entries when combining the two totals, the average age/gender estimate confidence accuracy varied only slightly from averaged age/gender estimate confidence.

Final Conclusions

ET’s accuracy determining chronological age range of survey participant – 98% accurate
ET’s accuracy determining physical gender of survey participant – 99.33% accurate


Posted in , , , , , , ,