Looking for good patterns of managing LLM/AI jsonschema metadata/description, possibly through registries #5685
Yuripetusko
started this conversation in
General
Replies: 1 comment
-
|
What would be really cool if there was a way to define an object schema and register all fields with metadata registry 1, and then be able to somehow provide metadata registry 2 and if it has any overlapping metadata ids in registry 2, they would be replaced with the values of registry 2 (all nested schema fields would also be replaced if matched), when doing .meta() or toJsonSchema() |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
I am looking at Registries and Metadata as a possible solution to my needs, but so far I probably lack a little bit of imagination on how to structure it the best.
For those who work with structured data and LLMs know that you need to provide additional description in .meta which will be passed as description of jsonschema to underlying models when using ai-sdk or when doing .toJSONSchema
The problem is that we have many such calls that often operate on the same schema with some modifications, so the schema often gets extended or description of base schema's fields need to be updated to provide different instructions for this new call.
For example the initial call might be
"Find me all Suppliers in London that would fit this event I am planning"and the supplier schema might look like thisThen the next call might be to a tool that enriches the supplier
"Enrich supplier with provided tools"Now for example if we have LLM call that does repeated search and then merging/deduplication like
Compare previously discovered supplier provided in the input below with this newly discovered supplier. If this is the same supplier, identify updated fields and return merged supplier objectNow I have to provide a schema again, but extending
enrichedSupplierSchemacan be confusing to LLM because the description of some fields mentions an actionFind...which can lead to hallucinations easily.This problem multiplies of course when the schema is much bigger and includes some other nested schemas. It's very easy to lose track of what extends what and what metadata descriptions it carries.
My idea is that I can define a base schema with no meta descriptions. Then have a
sharedSupplierSchemaRegistrywhere I can add the base descriptions that don't usually change per call and just describe type of information and type of data expected in field.Then a
searchSupplierSchemaRegistrycan have some search specific only fields that sometimes require some more actionable descriptions. And thensearchSupplierSchemacan potentially use a mix of base registry and search registry schemas etc.What's not clear is what are the good patterns of creating this, so far the docs only show a basic registries with a single schemas. Also not clear if I can use the value returned by myRegistry.get when adding a schema to a different registry?
Like would this work for example? (I know I know I should just try, but for now I'm just toying around with the idea and gathering some info)
Additionally can a registry schema be z.object({}) that itself references metas either from other registries or with inline meta?
i.e. can I do this?
And can I even add an object schema to registry and provide meta for nested fields? I am guessing not and description will only be added to the object definition itself right?
Would appreciate any thoughts or maybe there are some established patterns already
Beta Was this translation helpful? Give feedback.
All reactions