In today’s “e”-dense world, attorneys often look to leverage technology to facilitate production of electronically stored information (“ESI”) during the discovery process.  We do so in an effort to streamline the collection, review and production process whereby containing costs.  However, as recent decisions demonstrate, parties often disagree on what methodology to use and which analytic tools are best.  Livingston v City of Chicago, 16 CV 10156, 2020 WL 5253848 (N.D. Ill. Sept. 3, 2020) provides a good example of this precise issue.

The Livingston litigation involves allegations of discrimination by the City of Chicago (“City”) against female candidates seeking paramedic positions with the City Fire Department.  After two years of settlement negotiations, with only minimal success, the parties proceeded to discovery.  The process, however, resulted in a number of disputes that required judicial intervention, including a motion to compel the adoption of Plaintiffs’ discovery protocol, and a subsequent motion by Plaintiffs’ counsel to compel the City’s compliance with that protocol. The details of these applications, and the parties’ disagreement over analytics and processes, are set forth below.

Part I – The November 2019 Order

In 2019, when discovery began, the parties were unable to reach agreement on either the method for searching and collecting ESI, or the content of what was to be produced. Specifically, regarding production, the parties were at an impasse on the issue of whether, once potentially relevant emails were identified by search terms, those emails should be produced without further review.

The Plaintiffs’ proposed protocol required an outside vendor to export the City’s emails, perform keyword searches on that email, and produce the “hits” of those search terms without further review for responsiveness or privilege.  The City, on the other hand, proposed using its own Microsoft Tool to perform a simple search prior to exporting any data.  The Plaintiffs rejected this proposal out of purported concern that the City’s basic search would not comprehensively identify ESI for collection.  The City conceded that its collection tool was not fool-proof but countered it would miss only a “negligible quantity” of documents.  The court split the difference: requiring the City to retain an outside vendor to export emails and use Plaintiffs’ keywords to identify potentially responsive content.  The Court, however, denied Plaintiffs’ request that the “hits” be produced without further review for privilege or responsiveness (“November 2019 Order”).

The City subsequently reported that the Court-Ordered process resulted in 192,000 unique emails.  Given this volume, the City informed the Court and plaintiffs that it intended to use technology-assisted review (“TAR”) to identify responsive documents for production.* Plaintiffs objected to this process, claiming use of TAR would exclude responsive documents from the review process, and further claiming the City’s use of TAR was inconsistent with the November 2019 Order.  And so, Plaintiffs’ filed a motion seeking the Court order the City to comply with the November 2019 Order or, in the alternative, for entry of a proposed TAR protocol submitted by Plaintiffs that would require the use of TAR on the entire collection.

Part II – The September 2020 Order

In response to Plaintiffs’ second motion, the court found the November 2019 Order addressed the methodology for collecting emails, not the methodology for determining responsiveness. Thus, the November 2019 Order did not preclude TAR, and the issue of the City performing a responsiveness review was outside the scope of the November 2019 Order.

Plaintiffs further argued that since TAR is more effective at identifying responsive documents than traditional manual review, pre-TAR culling would eliminate large amounts of potentially relevant ESI. While the court did not deny that using TAR from the outset may reveal more responsive documents overall, based on the number of documents that were discarded using Plaintiffs’ proposed search terms (only 15% of 9 million pages of documents hit on Plaintiffs’ search terms), pre-TAR culling would achieve the best possible review in this case. The court found this method to satisfy the reasonable inquiry standard under Fed. R. Civ. P. 26.

Plaintiffs also argued that that attorney reviewers would improperly train TAR by making incorrect responsiveness decisions. The court recognized, however, that this concern is present no matter which methodology is employed, and that “uncertainty in determining responsiveness is not unique to TAR.” The court was also satisfied with the variety of quality controls offered by AL.

Ultimately, the court denied Plaintiffs’ motion, finding City, as the responding party, was best situated to decide how to search for and produce responsive documents to Plaintiffs’ requests.** 


This case reminds us of a number of things including the importance of understanding your clients’ ESI, and engaging meaningfully with opposing counsel on issues involving ESI collections and production.  However, when a responding party is fully familiar with its ESI, courts believe it is the responding party who is best situated to decide how to search for and produce responsive documents, and may be permitted to devise custom TAR methodologies to best fit the needs of the case.


*Specifically, the City sought to use Relativity’s Active Learning (“AL”), a type of software that uses learning algorithms to prioritize documents for its attorneys to review manually. As described in the decision, AL “merely shuffles the order of the documents being reviewed based on the coding decisions [i.e., responsive or nonresponsive] made by the attorney review team. All documents marked responsive and ultimately produced are done so by human reviewers.” However, as a result of using AL, there is a portion of the ESI collection deemed nonresponsive and not reviewed by attorney reviewers.

**The court cites to The Sedona Principles, Principle 6, which states, “[r]esponding parties are best situated to evaluate the procedures, methodologies, and technologies appropriate for preserving and producing their own [ESI].”