The data centre
You do not need to read this section to use the Visualisation Designer. It describes what goes on out of sight, behind the Data tab and the charts. Read it if you want to understand why things behave as they do - why a million points draw quickly, why searching structures needs an index, why a histogram costs almost nothing however big the dataset is. For how to do something, go to Data and the other component pages.
The idea in one page
Everything you draw comes from a dataset, and every dataset lives in the data centre: a PostgreSQL database, with the RDKit chemistry cartridge added, that belongs to your Data Manager project. When you import a file, the file’s contents go into a table in that database and the file is not needed again.
A chart never receives the whole table. Instead it asks the database a question and receives only the answer:
| A chart wants to draw | It asks | What comes back |
|---|---|---|
| A scatter plot | for the two columns it plots, for a sample of the rows | at most 10,000 points, whatever the dataset’s size |
| A histogram | for the number of rows in each of its bars | about 30 numbers |
| A hexbin plot | for the number of rows in each hexagon | a few hundred numbers |
| A box plot | for the quartiles, whiskers and outliers of each group | a few dozen numbers, and the outlying rows |
| A page of molecule cards | for the structures of the 50 molecules on the page | 50 molecules |
| The result of a structure search | for the rows whose structure matches | the matching row ids |
So the browser’s memory use and the time a chart takes to draw depend on what is drawn, not on how many rows there are. The work that does depend on the number of rows is done by the database, next to the data, with the tools a database has for it: indexes, parallel scans, and a query planner.
The pages in this section
- Why a database - what goes wrong when a dataset is held in the browser, and what changes when it is not.
- Why PostgreSQL and RDKit - why that engine, and what the chemistry cartridge adds.
- How molecules are stored - the simple one-table form, the indexed two-table form, and what each can do.
- How a join works - a join is a copy made inside the database, with new row ids, and what it shares with its source.
- How charts get their data - grouping for histograms and the other binned charts, sampling for point charts, and how a selection finds its rows.
- Performance - measurements: memory and speed against the number of rows and molecules, and where the limits are today.
The pages are theory. Where a page touches something you can do, it links to the practical page for it.