Better work with AI
After 10 days of AI assistance, we sent all participating lawyers a packet of inventor materials for a hypothetical invention and asked them to draft a patent. We did the same after 90 days, with a different set of simulated inventor materials. We saw the expected AI boost in performance on the drafting tasks at both time periods. At 10 days, AI tool access raised drafting scores (graded on a rubric with Likert scales for five different aspects of quality by independent legal experts) by 0.34 standard deviations, equivalent to a 10-point climb in percentile ranking among control group scores; by 90 days, this improvement increased to 0.38 SD, implying an 11-percentile point increase. These improvements occurred with remarkable consistency across all quality dimensions. Tellingly, better performance came from lower frequency of poor scores and higher frequency of good scores, and no change in the frequency of excellent scores. Formally, scores among lawyers allocated AI access moved from the bottom to the lower-middle and middle quintiles, but there were no noticeable changes in the number of exceptional scores. This pattern was particularly clear for junior lawyers, whose improvements in quality were also matched by significant time savings (18 minutes faster than the control group’s average speed of 124 minutes on the 10-day task, for example).
But patent lawyers don’t just draft patents; they also review each other’s work for critical mistakes (sometimes called “patent profanity”) that can make patents unenforceable in court — for instance, by overclaiming the novelty or scope of an invention. We therefore added another task at 90 days, which required subjects to redline (to manually mark up and correct) an existing hypothetical patent that contained many errors, both substantive and stylistic. This test task mirrors the kind of professional judgment that more senior lawyers routinely use and for which they most often do not have the luxury of relying on AI. Critically, while treatment group lawyers were encouraged to use the unreleased AI tool or any other AI tool on the drafting tasks, no one was allowed to use AI for the redlining task, making their scores on this task a measure of how well their skills had developed. For all tasks, drafting as well as redlining, we worked with third party patent professionals to score submissions on five dimensions of quality: (1) enforceability, (2) accuracy, (3) strategic ambiguity (the tactical scoping of claims), (4) completeness, and (5) clarity.