Performance

Optional reading: this explains what happens out of sight. See The data centre for what this section is, and Data for how to do things.

These are measurements, not promises. They say what the data centre does as a dataset grows, so you can judge what to expect of yours, and they say where it stops.

How they were measured

• Machine: one development workstation (12 cores, 24 threads, 62 GB of memory) with PostgreSQL 15.8 and the RDKit cartridge running in a container on the same machine, with PostgreSQL’s default memory settings. The browser side was a script calling the same HTTP interface the application uses, so each time includes the database, the server and the JSON, but no network. A shared production database, a network, and a drawing browser add to these times; how the times grow with size is what carries over.
• Plain tables: five columns (an integer id, three decimals, a short label), 1,000 to 10,000,000 rows. Each time is the median of five runs after one warm-up.
• Structures: a docking run of 35,090 poses (45 fields; each structure is a molfile of about 3.1 KB), and larger sets made by repeating it three and nine times, so every pattern keeps the same share of matches at each size. Fingerprint: Morgan, radius 2, 2048 bits. Each time is the median of five runs.
• “In the browser” figures come from loading the same files with the library’s own parser, as a browser tab would, and measuring the memory kept afterwards.

Plain tables: memory

Rows	Imported in	On disk (data centre)	In a browser tab, if loaded there
1,000	0.6 s	128 KB	1.3 MB
10,000	0.6 s	1.0 MB	5 MB
100,000	1.1 s	9.7 MB	43 MB
1,000,000	4.3 s	95 MB	440 MB
10,000,000	43 s	944 MB	about 4.4 GB (not attempted: more than a tab will hold)

The database holds about 100 bytes a row for this table, on disk, where the browser holds about 440. More importantly, the browser does not hold it at all in the data centre: what it keeps is what it draws, which is the next table.

Plain tables: how long a chart waits

Median milliseconds, from asking to having the answer.

Rows	Scatter, 10,000-point sample	Histogram	Hexbin	Box plot	Select half the rows	Every point, no sample
1,000	5	7	15	11	8	5
10,000	25	18	48	20	163	23
100,000	34	87	426	193	341	153
1,000,000	68	845	1,652	1,713	2,008	1,527
10,000,000	329	1,675	14,522	17,563	19,840	14,135

What the browser receives, for the same questions:

Rows	Scatter sample	Histogram, hexbin or box plot	Every point
1,000	22 KB	a few hundred numbers	22 KB
100,000	242 KB	a few hundred numbers	2.4 MB
1,000,000	252 KB	a few hundred numbers	25 MB
10,000,000	262 KB	a few hundred numbers	about 250 MB

How to read them:

• The default scatter plot is almost flat. It asks for at most 10,000 points, so it is about a quarter of a megabyte and well under a second at every size, including ten million rows. This is the sampling described in How charts get their data.
• The grouped charts grow with the rows, but gently, because the database does the counting in one pass over the columns it needs: a histogram is under a second to a million rows and about 1.7 seconds at ten million. Hexbin and box plot do more arithmetic per row and cost more: about twice a histogram at a million rows and about nine times at ten million.
• Drawing every point is the thing to avoid. At a million rows it is 25 MB and 1.5 seconds before the browser tries to make a million SVG marks - which is why the point charts cap what they draw.
• Selecting half of a big dataset is the slowest ordinary thing, because the answer is a list of five million row ids. A selection that large is rarely what you want to wire to another chart; a smaller one is proportionally quicker.

Structures: memory and build time

Molecules	Imported in	Index built in	Table (structures as text)	Structure index	In a browser tab, if loaded there
1,000	1.1 s	1.1 s	1.6 MB	1.7 MB	5 MB
35,090	4.2 s	37 s	61 MB	59 MB	376 MB
105,270	9.4 s	76 s	182 MB	177 MB	about 1.1 GB
315,810	25 s	187 s	546 MB	517 MB	about 3.4 GB

(The browser figure for 35,090 molecules was measured; the larger two are scaled from it, because the repeated files share their text and would understate the memory.)

The index is about the same size as the table: a little under 2 KB a molecule for each. Building it takes 0.6 to 1 millisecond per molecule. Importing does not build it, which is why importing is quick and indexing is a separate step you choose.

Structures: how long a search takes

Median milliseconds, in the data centre, for a search on the indexed structures.

Molecules	Substructure, common pattern (a benzene ring)	Substructure, [Cl,Br,I]	Substructure, rare (a carboxylic acid)	Similarity above 0.4	Similarity above 0.8	A page of 50 cards
1,000	34	32	30	21	21	3
35,090	372	286	117	52	44	3
105,270	935	696	348	136	114	3
315,810	2,736	1,802	1,358	368	299	3

For comparison, searching the same molecules without the data centre, in the browser with RDKit, costs about 0.4 ms for each molecule (measured on the 35,090 poses, where it took about 15 seconds): roughly 0.4 s for 1,000 molecules, 14 s for 35,090, and two minutes for 315,810, every time, plus having every structure in the browser to test. The indexed searches above are between 10 and 300 times quicker at those sizes, and the gap widens with size.

How to read them:

• A page of cards costs the same at every size. Only the 50 structures on screen are fetched.
• Similarity is the quickest search and grows roughly in step with the number of molecules: the index finds the molecules above the threshold without comparing every pair.
• Substructure depends on how many molecules match. The index discards the molecules that cannot match, and the survivors are tested exactly, so a common pattern (two thirds of the molecules here contain a benzene ring) costs several times what a rare one does, and part of its time is sending back the ids of 207,000 hits. A selective pattern is fast at any size.

Limits today

• Every database statement has a time limit, and a job’s is longer than a chart’s. A question a chart asks is cancelled after 60 seconds (nothing on this page comes close: the slowest, a box plot of ten million rows, is about 18). A statement inside a job - an import, an index build, an edit, a join, a filter - is cancelled after 30 minutes, because jobs have long steps: the first step of building a structure index, which reads every molecule and builds its fingerprint, took 94 seconds for 315,810 molecules. An index of that size therefore builds (187 seconds in all, as the table shows), and by that step’s rate the limit would be reached only at several million molecules. An administrator can change either limit, or remove it, with two settings of the application’s pod (DATA_CENTRE_STATEMENT_TIMEOUT and DATA_CENTRE_JOB_STATEMENT_TIMEOUT, in seconds).
• A question that runs past the limit is stopped, and says so. You see “The query took longer than the 60 second limit. Try a narrower search or filter.” in place of the chart or grid it was meant for (the number is whatever limit the application is set to), and nothing is retried. The usual cause is a search that matches most of a very large dataset: at three million molecules, a substructure search for a pattern nearly every molecule has (a benzene ring) ran past 60 seconds, where a selective one took three or four. Make the pattern more specific, filter the dataset first, or ask the administrator to raise the limit.
• The very largest grouped charts take seconds, not milliseconds. At ten million rows a hexbin or box plot is about 15 to 18 seconds. They still work, and they leave the browser idle, but you will wait for them; at a million rows or fewer they are around a second or less.
• Applying a filter reads every row to work out which to hide, so it is slower than the charts that follow it: about 12 seconds for a million rows. Clearing one takes about a second.
• The tables hold what you imported, plus what you add. Calculated columns, properties and indexes add to the sizes shown above. For a column of a million numbers, a B-tree index is about 22 MB and builds in under a second, a hash index about 34 MB and about a second, and a BRIN index 24 KB in a third of a second (BRIN is tiny because it only records a range for each block of rows; it helps when the numbers are stored roughly in order, and not for a column of random values like this one).