When implementing the Good Enough Recommendations (GER) engine, a core requirement was to let users insert large amounts of data quickly in order to bootstrap the recommendations engine. Additionally, this bootstrapping should be available over HTTP, as this will become the primary channel for interaction with GER.
PostGres (which GER uses) has the COPY command that is "optimised for loading large numbers of rows" in various formats, and npm has the package pg-copy-streams that pass Node.js streams to COPY. This would work well with the Hapi.js web application framework which can turn an uploaded file into a Node.js stream without having to hold the entire file in memory or create a temporary file on disk.
In this post I will describe how to upload a file and directly insert its data into PostGres using Node.js streams with Hapi.js and pg-copy-streams.