The test took 20 lawyers from a range of legal backgrounds in the USA. There were newly qualifieds, in-house lawyers with large corporations, sole practitioners and some with global law firms. It also took a computer loaded with LawGeex software. It gave them all five Non-Disclosure Agreements for review. The results published in a summary are impressive: the lawyers took an average of 96 minutes to review all five documents. The software took just 26 seconds. Neither the lawyers nor the computer had seen the specific NDAs before the test.

That's not all: the software and the lawyers showed that the best lawyer and the software reached 94% accuracy (taking into account false positives and false negatives). But the worst performing lawyer managed a lowly 67%. That's horribly poor, especially given that, for the lawyers taking part in the test, an NDA is bread and butter, almost grunt work. Equally startling is the spread of time taken: the quickest lawyer took 51 minutes: the slowest took 158 minutes or more than two and a half hours. The available results do not say if there was a correlation between speed and accuracy.

The company says that there is less than a 0.7% chance that the results are random and the methodology of the test was reviewed by a panel of academics and practitioners.

We tried to download the full report but the link didn't work and simply took us back to the top of the page we were already on, which was frustrating especially given that we were trying to get information offered by a software company.

Lawgeex' promotional material says

LawGeex Artificial Intelligence engine reads and analyses incoming contracts, suggesting edits based on a company’s pre-defined legal policies. Contracts that meet these policies can be automatically approved within an hour. Contracts that don’t align with your policies are escalated for guided editing and approval.

 

Legal can maintain full control and mitigate risk while giving other departments the freedom they need to get business moving faster.

- lawgeex.com

The process is simple: customers upload to the LawGeex server a soft-copy of the contract. The software compares it to a set of criteria defined by the customer and produces an exception report whereupon the customer takes action, for example to amend, accept or reject suggested changes (from the bank of options provided by the customer).

This raises two issues: first, the contract must be a software format that LawGeex recognises. LawGeex says "Support for a wide range of file types, including: doc, docx, pdf and images." Given the inadequacies of Optical Character Recognition, this raises risk. Basically, to be safe, it means that the uploaded document must have been delivered to LawGeek's customer in soft-copy or a hard copy must have been scanned, OCRd and proof read and corrected by the customer. That, of course, requires time and therefore cost.

The second is the thorny question of data protection. While data protection law in the USA is mostly notable by its absence, and where there is law it is weak and toothless, that is not the case in many other parts of the world. The impending General Data Protection Regulation, which is EU-wide, is already causing huge questions over the keeping, transfer and processing of data. Whether uploading sensitive data to a third party for processing is easy or hard and what compliance / risk management information must be adopted remains an open question.

The target turnaround time of one hour looks to be out of sync with the test results but that is not a sensible view: far more sensible is to consider that, if a partner gives the document to an assistant, it goes into that assistant's to-do-pile and he gets to it when he can. To compare the time from instructions to results on the partner's desk is far more instructive and, given that the actual contact time with the document is so much shorter, it is extremely unlikely that the assistant will be able to match the one-hour turnaround time.

Is it clever? From what we were able to glean, the answer is that it's clever software but it's not a clever lawyer. A comment on the LegalGeex website says"Bring your legal policies online. Easily define which clauses your business requires, what your acceptance criteria are, and what should be rejected. LawGeex AI will apply these policies to every incoming contract to ensure your business standards are followed every time."

There's the rub: you set up the system and tell it what to look for. Whether the system can identify nuances is open to question: for example, the extract above includes the Oxford Comma but in traditional legal drafting commas are usually left out of sentences.

In the early days of legal software, there were "expert systems" which were designed, in part, to facilitate exactly this process. But many firms found that the time and cost involved in training them was a sunk-cost that the firm did not recover, especially as that training had to be done by the firm's best (and therefore highest fee-earning) lawyers. Those firms that tried to introduce automated dictation found that it was quicker and easier to teach fee earners to type than to try to get voice recognition to work reliably, with the attendant cost of proof reading the results. The supposed world leader in so-called artificial intelligence, Google, can't understand the difference between English and American (try asking for results relating to grammes: Google search changes it to grams and provides those results as the default).

The test, therefore, cannot be regarded as proof that computers will replace lawyers across the board but it does show the direction that the profession must think about. First, clients are not going to want to pay USD1,000 an hour for a document review that someone else (truthfully or not) says can be done at least as well in under a minute. The one place that the system will prove ultra-useful and will help lawyers increase margins is in relation to standard-form conveyancing and, particularly, leases.

Will the technology make lawyers redundant? No. The simple fact is that what the system does, in essence, is compare an incoming document against a template (or to be more precise a checklist) for documents of that type created by the law firm itself, with all the skill required to so do. It's a tool that, if properly trained, will dramatically increase productivity of junior staff. Whether it's worth the investment of time by senior fee-earners remains to be seen: its output must still be reviewed, even though it allows editing documents on the web before they are downloaded.

And, of course, for less ethical firms, it will allow for fee-padding where, instead of time spent, a flat rate is charged for contract review so that fee earners can disassociate the revenue they generate from the time they spend feeding documents into a third party's system.