There are several ways to load data into LUSID's numerous storage options. Some are more suited to bulk loading data than others.
Note: When you load data into LUSID there may be a small propagation delay before it can be retrieved unless you specify the recorded asAt datetime of the upsert operation when you make a List* API call (or equivalent).
LUSID REST API
The LUSID REST API has an upsert model for entities. For example, the UpsertTransactions API:
- Updates an existing transaction if it exists, or
- Creates a new one if it does not.
LUSID does not generally have separate update and insert APIs.
Note that the REST API has restrictions on:
- The number of entities you can upsert per API call. You must batch calls if you need to upsert more than an endpoint's maximum:
Upsert API Maximum # entities per call BatchUpsertTransactions, UpsertTransactions, BatchUpsertCorporateActions, UpsertInstrumentEvents 10,000 UpsertInstruments, UpsertQuotes, UpsertComplexMarketData, UpsertLegalEntities, UpsertCustodianAccounts, UpsertCustomEntities 2,000 All other upsert APIs 1 - The quality and quantity of data you can supply for entity fields.
- The size of the payload. By default, this is 50MB per API call.
- The number of API calls you can make in total per minute. By default, you are rate-limited to 1000 API calls per 60 second window.
The alternatives to the REST API below might be better suited to bulk loading data into LUSID.
LUSID SDK
The LUSID SDK is available in 5 programming languages and frameworks (Python, C#, Java, JavaScript and Angular) and provides a convenient wrapper around the REST API.
You may prefer to use an SDK over Luminesce (see below) if your environment is suited to manipulating data using one of the languages/frameworks listed above rather than SQL.
LUSID Python Tools
LUSID Python Tools (LPT) is a wrapper around the LUSID Python SDK (above), with tooling for loading of Pandas DataFrames, CSV files and Excel files.
Reasons to prefer using LPT over the LUSID Python SDK to bulk load data include:
- Your preferred language for manipulating data is Python.
- LPT handles batching automatically.
- LPT makes asynchronous calls to improve performance.
- LPT streamlines the process of creating properties for entities if they do not already exist.
Luminesce
Luminesce is LUSID's data virtualisation engine and is optimised for bulk loading data in the following formats and storage locations into LUSID:
- CSV, Excel, SQLite, XML, Parquet and plain text files stored in Drive, LUSID's file repository and manager. More information.
- Relational data stored in MySql, Odbc, Oracle, PostgreSql, SqLite, SqlServer, GoogleBigQuery, Db2 or AwsAthena SQL-compatible databases. More information.
- CSV, Excel, SQLite, XML, Parquet and plain text files stored locally, in a NAS or similar device. More information.
- CSV, Excel, SQLite, XML, Parquet and plain text files hosted on a SFTP server. More information.
- CSV, Excel, SQLite, XML, Parquet and plain text files stored in AWS S3. More information coming soon.
- Snowflake. More information coming soon.
Reasons to use Luminesce include:
- Your data exists in the storage locations and data formats listed above.
- Your preferred language for manipulating data is SQL.
- Your payloads exceed 50MB.
- Luminesce handles batching automatically.
For a demonstration of Luminesce please reach out to your FINBOURNE representative.
Horizon
Horizon is FINBOURNE's partnership program for industry-leading data vendors and service providers. It consists of utilities optimised for the bulk loading of subscription data into LUSID.
Utilities for Copp Clark market holiday data and SIX Flex instrument and quote data are available now, with many more coming soon.
For more information please reach out to your FINBOURNE representative.