So how does the Notes database work?
Basically the Notes database contains a header, and then just one type of data object, called a Note. There are different types of Notes, one for ACL (Access Control Lists), one for Design elements, and one for Data notes.

View indexes, built when a view is indexed, are also kept in Notes. View indexes don't replicate. As it says in Inside Notes (see http://www-10.lotus.com/ldd/today.nsf/lookup/inside_notes for more):

One pillar of Notes/Domino design is the use of a single and simple data structure, called a note, to store all information -- including design elements such as forms and views, as well as standard user documents -- within a Notes database. This simple design feature leads to an incredibly powerful benefit: NOS (Notes Object Services) implements a single set of note-oriented services to create and manage all the different types of information that can be in a database.

Competitive products use distinct programs to create, manage, and disseminate different types of important data in the system. Developing a program to run in a multiplatform, multinetwork environment is hard enough without having to worry about multiple data structures, too. Using one simple note-oriented data model allows Notes developers to concentrate on making one small set of efficient, bug-free programs.

Inside Notes also says,

Fundamental support for security and groupware features can be found in all Notes software -- in client and server programs, as well as in the NOS functions that support them. Regarding security, for example, the Notes Client program and the Domino Server program work together to authenticate a user prior to letting the user access a database, and the NOS function used to update a note in a database first checks the user's rights to do so against information in the database ACL note.

A large portion of support for groupware features is implemented in NOS because many Notes groupware capabilities are tightly linked to the note-oriented data model, and all of the low-level note-oriented functions are implemented in the NOS. For example, NOS contains the logic that updates a newly edited note in a database. If NOS detects that two users are trying to update the same note at the same time, NOS accepts the first update and rejects the second.

How does this differ from an RDBMS?
There are several areas of difference between Notes and a traditional relational database management system.

1. Notes keeps application code and data in the one database.
For sure, many RDBMS keep code in the database, as stored procedures, but this code is mainly database management code that deals with properly managing database operations, rather than implementing business and application logic.

Notes does not have any concept of code that is attached to data storage events. QuerySave and other events are attached to UI objects, and their operations. Notes's application code is all kept in the database, and replicates with it.