When initially experimenting with LLMs and agentic AI, software program engineers at Notion AI utilized superior code technology, advanced schemas, and heavy instructioning.
Rapidly, although, trial and error taught the workforce that it may get rid of all of that complicated data modeling. Notion’s AI engineering lead Ryan Nystrom and his workforce pivoted to easy prompts, human-readable representations, minimal abstraction, and acquainted markdown codecs. The end result was dramatically improved mannequin efficiency.
Making use of this re-wired method, the AI-native firm launched V3 of its productiveness software program in September. Its notable characteristic: Cutomizable AI brokers — which have shortly grow to be Notion’s most profitable AI software to date. Based mostly on utilization patterns in contrast to earlier variations, Nystrom calls it a “step operate enchancment.”
“It is that feeling of when the product is being pulled out of you relatively than you making an attempt to push,” Nystrom explains in a VB Beyond the Pilot podcast. “We knew from that second, actually early on, that we had one thing. Now it is, ‘How may I ever use Notion with out this characteristic?’”
‘Rewiring’ for the AI agent period
As a standard software program engineer, Nystrom was used to “extraordinarily deterministic” experiences. However a lightweight bulb second got here when a colleague advised him to merely describe his AI immediate as he would to a human, relatively than codify guidelines of how brokers ought to behave in numerous situations. The rationale: LLMs are designed to perceive, “see” and motive about content material the identical manner people can.
“Now, at any time when I am working with AI, I’ll reread the prompts and power descriptions and [ask myself] is this one thing I may give to an individual with no context and so they may perceive what is going on on?” Nystrom mentioned on the podcast. “If not, it is going to do a foul job.”
Stepping again from “fairly sophisticated rendering” of knowledge inside Notion (corresponding to JSON or XML) Nystrom and his workforce represented Notion pages as markdown, the common device-agnostic markup language that defines construction and that means utilizing plain textual content with out the want for HTML tags or formal editors. This permits the mannequin to work together with, learn, search and make modifications to textual content recordsdata.
In the end, this required Notion to rewire its programs, with Nystrom’s workforce focusing largely on the middleware transition layer.
In addition they recognized early on the significance of exercising restraint when it comes to context. It’s tempting to load as a lot information right into a mannequin as doable, however that may sluggish issues down and confuse the mannequin. For Notion, Nystrom described a 100,000 to 150,000 token restrict as the “candy spot.”
“There are circumstances the place you possibly can load tons and tons of content material into your context window and the mannequin will battle,” he mentioned. “The extra you place into the context window, you do see a degradation in efficiency, latency, and likewise accuracy.”
A spartan method is additionally necessary in the case of tooling; this may also help groups keep away from the “slippery slope” of countless options, Nystrom advised. Notion focuses on a “curated menu” of instruments relatively than a voluminous Cheesecake Manufacturing facility-like menu that creates a paradox of alternative for customers.
“When folks ask for brand new options, we may simply add a software to the mannequin or the agent,” he mentioned. However, “the extra instruments we add, the extra selections the mannequin has to make.”
The underside line: Channel the mannequin. Use APIs the manner they had been meant to be used. Do not attempt to be fancy, do not attempt to overcomplicate it. Use plain English.
Hear to the full podcast to hear about:
-
Why AI is nonetheless in the pre-Blackberry, pre-iPhone period;
-
The significance of “dogfooding” in product growth;
-
Why you shouldn’t fear about how value efficient your AI characteristic is in the early phases — that may be optimized later;
-
How engineering groups can maintain instruments minimal in the age of MCP;
-
Notion’s evolution from wikis to full-blown AI assistants.
Subscribe to Past the Pilot on Apple Podcasts, Spotify, and YouTube.
Disclaimer: This article is sourced from external platforms. OverBeta has not independently verified the information. Readers are advised to verify details before relying on them.
