Excelの業務フロー、AIには「見出し5個」しか見えていなかった — 図形に隠れた180個が消える罠When feeding Excel workflow diagrams to AI, text embedded in Shape objects is…

Spreadsheets remain one of the most common formats for documenting business processes, and teams increasingly hand those files to large language models to summarize, restructure, or automate the workflows they describe. A recent post on Zenn highlights a subtle but consequential failure in that pipeline: when an Excel workflow diagram assembled from Shape objects was fed to an AI tool, the model appeared to recognize only about five of the roughly 180 headings present. The remaining text was effectively invisible to the model, and nothing in the output signaled that anything was missing.

The cause is structural rather than a limitation of the model's reasoning. A modern Excel file (.xlsx) is a ZIP archive containing multiple XML parts. Cell values live in the worksheet XML, but text typed into drawing objects — text boxes, autoshapes, connectors, and the boxes that make up a flowchart — is stored separately in DrawingML parts under a drawings folder. To a person viewing the sheet, both look like text on a grid. To a program that reads only cell values, the shape text does not exist.

This distinction matters because many tools that prepare Excel data for an LLM extract cell contents and ignore the drawing layer. Popular Python libraries such as pandas and openpyxl are oriented around cells and tables; openpyxl exposes some drawing information but does not surface shape text in a convenient, first-class way. When such a converter turns a spreadsheet into CSV, Markdown, or plain text for a model, any content that lived inside shapes is dropped before the model ever sees it. The model then answers confidently based on the fraction of the document it actually received.

The problem is especially acute in workflows common in Japanese enterprises, where Excel is frequently used as a general-purpose diagramming and layout tool. Process flows, organization charts, and approval routes are often drawn with connected shapes rather than entered into cells, a style sometimes associated with the "Excel graph paper" convention. A file that looks information-rich on screen can therefore carry most of its meaning in a layer that text-based extraction skips entirely.

The practical remedy is preprocessing: extract and convert the shape text into a form the model can read before passing the file along. Because the drawing XML is accessible inside the archive, one approach is to parse those parts directly and pull the text runs from each shape, optionally preserving position or connection data so the flow's structure survives. On Windows, COM automation through libraries like pywin32, or tools built on the Office object model, can enumerate shapes and read their text more reliably than pure-parsing approaches. Commercial libraries also offer shape-aware extraction. The common thread is that the transformation step, not the model, is where fidelity is won or lost.

An alternative worth considering is treating the diagram as an image. Rendering the sheet or the shape region and passing it to a vision-capable model lets optical character recognition and layout understanding recover text that a structural parser might miss, though this trades one set of errors for another and can struggle with dense or overlapping elements. In some cases, combining both signals — extracted shape text plus a rendered image — is likely to produce the most complete result.

The broader lesson extends beyond Excel. Document ingestion for AI systems, including retrieval-augmented generation, depends heavily on how faithfully a file is converted to text. PDFs, slide decks, and scanned forms all have layers or encodings that naive extractors miss, and the failures tend to be silent: the model produces fluent output regardless of how much context it actually received. Validating that extraction captured what a human sees — for instance, by checking that the number of detected headings matches expectations — is a cheap safeguard against confidently wrong answers.

The Zenn case is a useful reminder that when AI results seem thin or incomplete, the problem may lie upstream in the data handed to the model rather than in the model itself. As organizations move toward automating document-heavy workflows, the reliability of these systems will depend less on raw model capability and more on the unglamorous work of parsing, extraction, and verification that happens before a prompt is ever sent.