How molecules are stored
Optional reading: this explains what happens out of sight. See The data centre for what this section is, and Data for how to do things.
A column of structures can be in one of two states, and you move it from the first to the second yourself.
State 1: text, in one table
When an SDF file is imported, each molecule’s structure goes into the dataset’s table as text (a molfile) in a column called molecule, beside the file’s other
properties. Nothing about chemistry is computed. That makes the import fast (a few seconds for tens of thousands of molecules) and the table an ordinary one.
Everything that only needs the structure’s text works straight away:
| Needs | How it is done |
|---|---|
| Drawing 2D structures (the card grid, tooltips, the R-group table) | The browser fetches the text of only the molecules on screen (a page of 50 cards) and RDKit in the browser draws them |
| The 3D viewer | The browser fetches the text of the molecules to show (at most Max ligands, 200 by default) and NGL draws them |
| The interaction map and diagram | The browser fetches the ligands’ text and NGL detects the contacts, headlessly |
| The R-group table and the mutation-shells view | The browser fetches the rows (up to 5,000 and 2,000 respectively) and RDKit decomposes or compares them there |
| Calculate molecular properties | The database parses each row’s text with the cartridge, once, and writes the chosen properties as ordinary columns |
| Export as SDF | The database streams the text back out |
What does not work on text is a search: a substructure query or a similarity query. Answering one means looking at every molecule, and that has to be done where the data is. So until the structures are indexed, the card grid shows its search controls greyed out, with a note saying why. It does not fall back to fetching every molecule into the browser, which would be slow and heavy (see Performance).
State 2: indexed, in two tables
Index structures… builds a second table, the structure index, from the first. For every row it holds:
- the molecule as the cartridge has read it (RDKit’s own representation, which is what a substructure test runs against), and
- one fingerprint for each kind you chose - a fixed-length string of bits that summarises which small pieces of structure a molecule contains. The default is a Morgan fingerprint, radius 2, 2048 bits; Feature Morgan, RDKit, atom-pair and topological-torsion fingerprints are also available.
Each of those columns carries a GiST index, the chemistry cartridge’s own kind. The dataset table itself is not rewritten: the two are linked by a molecule id, and the dataset’s text stays where it was.
A substructure search uses the molecule index to rule out almost every molecule cheaply (a molecule without the pattern’s small pieces cannot contain it) and then tests only the survivors exactly. A similarity search builds the query’s fingerprint with the same settings the index was built with, and the fingerprint index finds the molecules whose Tanimoto similarity is above your threshold. The first fingerprint in the index is the one similarity uses.
Why a second table, rather than adding the columns to the first?
- It is optional, and the cost is yours to choose. Import stays fast, and you decide which fingerprints to build, when. Building the index of a docking run of 35,090 poses takes about 37 seconds; for 1,000 molecules, about a second.
- It can be replaced without touching the data. Indexing again with other fingerprints builds a new index and swaps it in; the dataset is not rewritten.
- It can be shared. Many datasets can use one structure index (see below), so the cost is paid once.
A molecule that cannot be read
Some structures do not parse - an empty molfile, a valence RDKit rejects. The index records them with no molecule rather than failing the whole job, and counts them. Such a
molecule matches no search, and it is in neither a search nor its negation, just as a row with a missing number is in neither x > 3 nor not (x > 3).
What happens to the index when you change the dataset
| You | The index |
|---|---|
| Calculate fields, calculate molecular properties, change a type, add or remove another column’s index | stays as it is (the structure text is unchanged) |
| Apply a filter | stays; molecules hidden by the filter are simply not searched |
| Make a filter permanent | the entries for the deleted molecules are removed, except any that another dataset sharing the index still uses (see below) |
| Join with another dataset | the new dataset uses the same index; its rows carry the molecule id, and a row with no structure (the unmatched side of an outer join) matches nothing |
| Rename the structure field | stays; the dataset’s link follows the new name |
| Delete the structure field | the dataset lets go of the index; the index table is deleted unless another dataset still uses it |
| Delete the dataset | the same |
An index is kept for as long as any dataset uses it and removed when the last one lets go, so sharing never leaves an index behind or removes one from under a dataset.
Making a filter permanent deletes rows, so the index is tidied to match: an entry is removed when no dataset uses its molecule any more. “Uses” is judged across every dataset that shares the index, and counts a row another dataset has hidden with its own filter, since clearing that filter brings the row back. So an entry stays if a join made from this dataset still holds its molecule, or if another dataset sharing the index has it hidden. If nothing else shares the index, every entry for a deleted row goes. The space the removed entries held is reused by the index rather than handed back to the system, so the index does not get smaller on disk by itself. To give the disk space back, choose Compact for the index’s table in About database, or index the dataset again.
If a join is being made from the dataset at the very moment its filter is made permanent, one of the two is stopped with a message asking you to run it again, because the join could otherwise have been given molecules that were just removed.
How big is it, and how long does it take?
The Performance page has the measurements. In short: the index is about the same size as the table of molecule text it was built from (a little under 2 KB per molecule each), building it takes about 0.6 to 1 millisecond per molecule, and a search then costs milliseconds to a few seconds depending on how many molecules there are and how many match.