AI Needs a New Storage Engine
A typical DBMS consists of multiple components. In no particular order, this includes Query Processing, Replication, Concurrency, Transactions, Buffer Management, and the Storage Engine.
The storage engine stands out as the lowest level in the DBMS architecture. Because of this, separating the storage engine from the typical DBMS design offers several advantages, which can greatly enhance flexibility, scalability, and performance. The storage engine is often the least interesting part of database innovation. However, it consumes much of the processing power and memory and is an important element of the solution's overall performance. The storage engine is also the least well-understood part of the DBMS for most developers, as the storage engine must deal with a lot of the messy stuff underneath.
Benefits of separating out the storage layer
Flexibility: Developers can choose storage engines based on specific needs, avoiding lock-in to a single architecture.
Optimization: Different engines excel at different tasks, allowing tailored performance for various workloads. Also, dedicated storage engines are likely better optimized for the various devices supported
Scalability: Custom storage engines can be optimized better for the platform
Maintenance: Upgrades and maintenance are easier with modular architecture, minimizing downtime and risk.
Interoperability: Integration with external systems is simplified, facilitating hybrid cloud deployments.
Customization: Developers can extend functionality through custom storage plugins, tailoring the DBMS to specific needs.
Conclusion
Many databases already support storage engine plug-ins using a key-value based API. We believe adoption of this approach will increase over time. Typically, this is using the rocksdb API, but more work is needed to flesh out this layer. As technology advances we envision a more powerful and flexible storage engine emerging, empowering developers to create efficient, scalable, and customizable data storage solutions.
At QiStor, we are obsessed with building a better AI storage engine. We think this will be a game-changer for the database stack.
About QiStor
QiStor is a tech company specializing in Key Value Store acceleration technology. Key-Value enables every modern app in social media, mobile, web, AI and gaming spaces to store data at scale. QiStor’s Key Value Store solution will reduce compute power by 10x for web-scale applications, providing businesses with the most efficient and cost-effective database solutions. QiStor’s services can be easily deployed as a Platform-as-a-Service (PaaS), empowering businesses to achieve their goals faster, more efficiently, and more sustainably