# Extracting and Parsing Multi-Receipts Tutorial (Node.js) ## Overview ### When to use this? It may be a common occurrence for you or your users that taking one picture of a bunch of receipts at once might be way faster than taking a bunch of different pictures. In one such case, you may find that it may be tough to split the ensuing picture into sub-files before sending them to the Receipt API. No worries, the **Multi-Receipts Detector API** takes care of that. The underlying principle is simple, an AI scans the contents of a file and identifies the coordinates of the receipts it sees. Those coordinates then can be used to extract those receipts from the initial picture into their own files. {% hint style="info" %} **Note:** for PDFs the principle applies at a page-level: i.e. each page will be treated as a separate picture. The results can be retrieved either all at once or on a page-by-page basis. {% endhint %} ### Initial Setup The goal of this tutorial is to use the Multi-Receipts Detector API to isolate receipts and then parse them on the Receipts OCR. To do this, we'll be using the following dummy file:
Before uploading, be sure that you have taken the following into account: * Receipts are clear, unstained and properly unfolded * Receipts don't overlap, and they are not spilling out of the picture * Receipts are aligned in roughly the same direction * No other type of documents are mixed in with the receipts The Multi-Receipts Detector API is quite robust, but it will not distinguish between receipts and other similar pieces of paper if you mistakenly add some. ## Subscribing to the Multi-Receipts API For this tutorial, you'll need to be in possession of a [valid and up-to-date API key(](https://docs.mindee.com/v1/get-started/create-api-key) and a valid subscription to the [Multi-Receipts API](https://platform.mindee.com/mindee/multi_receipts_detector) as well as the [Receipt OCR](https://platform.mindee.com/mindee/expense_receipts/live-interface). If you aren't sure whether your subscription is enabled or not, go on the `API` page of the main interface, click on `Utilities`:
Then click on the `Multi Receipts Detector` button. That's it, your subscription to the API should be enabled. To do the same for the receipts OCR, simply head to the `Document Catalog`:
Then, like in the previous step, click on our product of interest (`Expense Receipt`). {% hint style="warning" %} Be aware that you need to enable **both subscriptions** to get this whole tutorial working. {% endhint %} ## Calling the API Time to get coding! For this tutorial, we'll be using the official Mindee client library for Node.js. Other programming languages are supported, check [the list](https://github.com/mindee/legacy-docs/blob/main/libraries/librairies.md). Let's create a directory to store our project and install the Mindee client: ```shell mkdir multi_receipts_tutorial cd multi_receipts_tutorial npm install -s mindee touch demo_multi_receipts.js ``` To get started with the code, we'll head on over to the `Documentation` page by using the link on the left of the interface. From there we'll click on your language (`NODEJS`), then `Select an API key`, and finally use the copy code button:
Using your IDE, open the `demo_multi_receipts` file we created, and paste this code. We'll use it as a base. Make sure the API key is correctly filled in, if not, simply grab one from the Mindee interface. Find the line: ```js const inputSource = mindeeClient.docFromPath("/path/to/the/file.ext"); ``` And put in a real file path on your drive. You should test this file in the [live interface](https://platform.mindee.com/mindee/multi_receipts_detector/live-interface) before checking. Now run the code: ```shell node demo_multi_receipts.js ``` The result should look something like this: ```rst ######## Document ######## :Mindee ID: d7c5b25f-e0d3-4491-af54-6183afa1aaab :Filename: default_sample.jpg Inference ######### :Product: mindee/multi_receipts_detector v1.0 :Rotation applied: Yes Prediction ========== :List of Receipts: Polygon with 4 points. Polygon with 4 points. Polygon with 4 points. Polygon with 4 points. Polygon with 4 points. Polygon with 4 points. Page Predictions ================ Page 0 ------ :List of Receipts: Polygon with 4 points. Polygon with 4 points. Polygon with 4 points. Polygon with 4 points. Polygon with 4 points. Polygon with 4 points. ``` You'll notice that receipts only have a description of a polygon here. That is because this is only a summary of what's going on, but fret not, our receipts coordinates are of course accessible! ## Image extraction code The method to call to extract the images is called in the following way: ``` const { imageOperations } = require("mindee"); //... const someResult = imageOperations.extractReceipts(myInputFile, myInferenceResult); ``` We'll add the method call to the response handling part of the script.\ You should have something like this: ```js const { Client, imageOperations } = require("mindee"); const mindeeClient = new Client(); //... rest of the basic implementation // Handle the response Promise apiResponse.then((resp) => { const someResult = imageOperations.extractReceipts(myInputFile, resp.document.inference); someResult.then((extractionResp) => { //some code... }); }); ``` This is not very pretty, though. We can make this a bit more legible and usable by switching to something like this. ```js const { Client, product, imageOperations } = require("./../dist/src"); const { setTimeout } = require("node:timers/promises"); async function parseReceipts() { // fill in your API key or add it as an environment variable const mindeeClient = new Client(); const multiReceiptsFile = mindeeClient.docFromPath("path/to/your/file.ext"); const resp = await mindeeClient.parse(product.MultiReceiptsDetectorV1, multiReceiptsFile); let receipts = await imageOperations.extractReceipts(multiReceiptsFile, resp.document.inference); } parseReceipts(); ``` ### Parsing the Extracted Files Now one last piece is missing: what to do with our `receipts`? Well, parse them, of course! The code for this part is mostly up to you, but for this tutorial we'll use a simple loop to parse the documents and then display the results: ``` for (const receipt of receipts) { const respReceipt = await mindeeClient.parse(product.ReceiptV5, receipt.asSource()); console.log(respReceipt.document.toString()); await setTimeout(1000); // wait some time between requests as to not overload the server } ``` This should print something like this: ``` ######## Document ######## :Mindee ID: 8506a82c-bc1f-4d5b-abd1-174f60459091 :Filename: receipt_p0_0.pdf Inference ######### :Product: mindee/expense_receipts v5.1 :Rotation applied: Yes Prediction ========== :Expense Locale: fr-FR; fr; FR; EUR; :Purchase Category: miscellaneous :Purchase Subcategory: :Document Type: CREDIT CARD RECEIPT :Purchase Date: 2022-08-24 :Purchase Time: 12:45 :Total Amount: 16.50 :Total Net: :Total Tax: :Tip and Gratuity: :Taxes: +---------------+--------+----------+---------------+ | Base | Code | Rate (%) | Amount | +===============+========+==========+===============+ :Supplier Name: :Supplier Company Registrations: 81187803200010 :Supplier Address: :Supplier Phone Number: :Line Items: Page Predictions ================ Page 0 ------ :Expense Locale: fr-FR; fr; FR; EUR; :Purchase Category: miscellaneous :Purchase Subcategory: :Document Type: CREDIT CARD RECEIPT :Purchase Date: 2022-08-24 :Purchase Time: 12:45 :Total Amount: 16.50 :Total Net: :Total Tax: :Tip and Gratuity: :Taxes: +---------------+--------+----------+---------------+ | Base | Code | Rate (%) | Amount | +===============+========+==========+===============+ :Supplier Name: :Supplier Company Registrations: 81187803200010 :Supplier Address: :Supplier Phone Number: :Line Items: ######## Document ######## :Mindee ID: f8e5b080-533e-43b6-b4bb-ef77208a74a3 :Filename: receipt_p0_1.pdf Inference ######### :Product: mindee/expense_receipts v5.1 :Rotation applied: Yes Prediction ========== :Expense Locale: fr-FR; fr; FR; EUR; :Purchase Category: food :Purchase Subcategory: restaurant :Document Type: EXPENSE RECEIPT :Purchase Date: 2022-08-25 :Purchase Time: 12:43 :Total Amount: 21.50 :Total Net: 19.55 :Total Tax: 1.95 :Tip and Gratuity: :Taxes: +---------------+--------+----------+---------------+ | Base | Code | Rate (%) | Amount | +===============+========+==========+===============+ | 19.55 | TVA | 10.00 | 1.95 | +---------------+--------+----------+---------------+ :Supplier Name: CHES :Supplier Company Registrations: 87880027500017 :Supplier Address: 35 rue charlot 75003 paris :Supplier Phone Number: 0100000000 :Line Items: +--------------------------------------+----------+--------------+------------+ | Description | Quantity | Total Amount | Unit Price | +======================================+==========+==============+============+ | Repas | 1.00 | 21.50 | | +--------------------------------------+----------+--------------+------------+ Page Predictions ================ Page 0 ------ :Expense Locale: fr-FR; fr; FR; EUR; :Purchase Category: food :Purchase Subcategory: restaurant :Document Type: EXPENSE RECEIPT :Purchase Date: 2022-08-25 :Purchase Time: 12:43 :Total Amount: 21.50 :Total Net: 19.55 :Total Tax: 1.95 :Tip and Gratuity: :Taxes: +---------------+--------+----------+---------------+ | Base | Code | Rate (%) | Amount | +===============+========+==========+===============+ | 19.55 | TVA | 10.00 | 1.95 | +---------------+--------+----------+---------------+ :Supplier Name: CHES :Supplier Company Registrations: 87880027500017 :Supplier Address: 35 rue charlot 75003 paris :Supplier Phone Number: 0100000000 :Line Items: +--------------------------------------+----------+--------------+------------+ | Description | Quantity | Total Amount | Unit Price | +======================================+==========+==============+============+ | Repas | 1.00 | 21.50 | | +--------------------------------------+----------+--------------+------------+ ######## Document ######## :Mindee ID: 6812060a-b9e4-4e45-8492-00aef14de36d :Filename: receipt_p0_2.pdf Inference ######### :Product: mindee/expense_receipts v5.1 :Rotation applied: Yes Prediction ========== :Expense Locale: fr-FR; fr; FR; EUR; :Purchase Category: food :Purchase Subcategory: restaurant :Document Type: EXPENSE RECEIPT :Purchase Date: 2022-08-22 :Purchase Time: 12:27 :Total Amount: 13.50 :Total Net: :Total Tax: :Tip and Gratuity: :Taxes: +---------------+--------+----------+---------------+ | Base | Code | Rate (%) | Amount | +===============+========+==========+===============+ :Supplier Name: LA CERISE SUR LA PIZZA :Supplier Company Registrations: 51092131500063 :Supplier Address: 14 rue froissart 75003 paris france :Supplier Phone Number: :Line Items: +--------------------------------------+----------+--------------+------------+ | Description | Quantity | Total Amount | Unit Price | +======================================+==========+==============+============+ | EMPORTER LA CASTE | 1.00 | 11.50 | 11.50 | +--------------------------------------+----------+--------------+------------+ | CANETTE SODA | 1.00 | 2.00 | 2.00 | +--------------------------------------+----------+--------------+------------+ Page Predictions ================ Page 0 ------ :Expense Locale: fr-FR; fr; FR; EUR; :Purchase Category: food :Purchase Subcategory: restaurant :Document Type: EXPENSE RECEIPT :Purchase Date: 2022-08-22 :Purchase Time: 12:27 :Total Amount: 13.50 :Total Net: :Total Tax: :Tip and Gratuity: :Taxes: +---------------+--------+----------+---------------+ | Base | Code | Rate (%) | Amount | +===============+========+==========+===============+ :Supplier Name: LA CERISE SUR LA PIZZA :Supplier Company Registrations: 51092131500063 :Supplier Address: 14 rue froissart 75003 paris france :Supplier Phone Number: :Line Items: +--------------------------------------+----------+--------------+------------+ | Description | Quantity | Total Amount | Unit Price | +======================================+==========+==============+============+ | EMPORTER LA CASTE | 1.00 | 11.50 | 11.50 | +--------------------------------------+----------+--------------+------------+ | CANETTE SODA | 1.00 | 2.00 | 2.00 | +--------------------------------------+----------+--------------+------------+ ######## Document ######## :Mindee ID: ac5a337b-7ca2-4acd-8a3f-a2cf54697ede :Filename: receipt_p0_3.pdf Inference ######### :Product: mindee/expense_receipts v5.1 :Rotation applied: Yes Prediction ========== :Expense Locale: fr-FR; fr; FR; EUR; :Purchase Category: food :Purchase Subcategory: restaurant :Document Type: EXPENSE RECEIPT :Purchase Date: 2022-08-24 :Purchase Time: 12:45 :Total Amount: 16.50 :Total Net: 15.00 :Total Tax: 1.50 :Tip and Gratuity: :Taxes: +---------------+--------+----------+---------------+ | Base | Code | Rate (%) | Amount | +===============+========+==========+===============+ | 15.00 | TVA | 10.00 | 1.50 | +---------------+--------+----------+---------------+ :Supplier Name: BUR :Supplier Company Registrations: FR95811878032 81187803200010 :Supplier Address: 22 rue rambuteau 75003 paris france :Supplier Phone Number: :Line Items: +--------------------------------------+----------+--------------+------------+ | Description | Quantity | Total Amount | Unit Price | +======================================+==========+==============+============+ | Repas | 1.00 | 16.50 | 16.50 | +--------------------------------------+----------+--------------+------------+ Page Predictions ================ Page 0 ------ :Expense Locale: fr-FR; fr; FR; EUR; :Purchase Category: food :Purchase Subcategory: restaurant :Document Type: EXPENSE RECEIPT :Purchase Date: 2022-08-24 :Purchase Time: 12:45 :Total Amount: 16.50 :Total Net: 15.00 :Total Tax: 1.50 :Tip and Gratuity: :Taxes: +---------------+--------+----------+---------------+ | Base | Code | Rate (%) | Amount | +===============+========+==========+===============+ | 15.00 | TVA | 10.00 | 1.50 | +---------------+--------+----------+---------------+ :Supplier Name: BUR :Supplier Company Registrations: FR95811878032 81187803200010 :Supplier Address: 22 rue rambuteau 75003 paris france :Supplier Phone Number: :Line Items: +--------------------------------------+----------+--------------+------------+ | Description | Quantity | Total Amount | Unit Price | +======================================+==========+==============+============+ | Repas | 1.00 | 16.50 | 16.50 | +--------------------------------------+----------+--------------+------------+ ######## Document ######## :Mindee ID: 2fc54836-61eb-4c0f-b3f3-35717505f7aa :Filename: receipt_p0_4.pdf Inference ######### :Product: mindee/expense_receipts v5.1 :Rotation applied: Yes Prediction ========== :Expense Locale: fr-FR; fr; FR; EUR; :Purchase Category: food :Purchase Subcategory: restaurant :Document Type: EXPENSE RECEIPT :Purchase Date: 2022-08-23 :Purchase Time: 12:52 :Total Amount: 14.50 :Total Net: :Total Tax: 0.21 :Tip and Gratuity: :Taxes: +---------------+--------+----------+---------------+ | Base | Code | Rate (%) | Amount | +===============+========+==========+===============+ | | TVA | 5.50 | 0.21 | +---------------+--------+----------+---------------+ :Supplier Name: CHEZ ALAIN MIAM MIAM :Supplier Company Registrations: 83833640200014 :Supplier Address: 26 rue char lot 75003 paris france :Supplier Phone Number: :Line Items: +--------------------------------------+----------+--------------+------------+ | Description | Quantity | Total Amount | Unit Price | +======================================+==========+==============+============+ | MIAM MIAM | 1.00 | 10.50 | 10.50 | +--------------------------------------+----------+--------------+------------+ | Limonade | 1.00 | 4.00 | 4.00 | +--------------------------------------+----------+--------------+------------+ Page Predictions ================ Page 0 ------ :Expense Locale: fr-FR; fr; FR; EUR; :Purchase Category: food :Purchase Subcategory: restaurant :Document Type: EXPENSE RECEIPT :Purchase Date: 2022-08-23 :Purchase Time: 12:52 :Total Amount: 14.50 :Total Net: :Total Tax: 0.21 :Tip and Gratuity: :Taxes: +---------------+--------+----------+---------------+ | Base | Code | Rate (%) | Amount | +===============+========+==========+===============+ | | TVA | 5.50 | 0.21 | +---------------+--------+----------+---------------+ :Supplier Name: CHEZ ALAIN MIAM MIAM :Supplier Company Registrations: 83833640200014 :Supplier Address: 26 rue char lot 75003 paris france :Supplier Phone Number: :Line Items: +--------------------------------------+----------+--------------+------------+ | Description | Quantity | Total Amount | Unit Price | +======================================+==========+==============+============+ | MIAM MIAM | 1.00 | 10.50 | 10.50 | +--------------------------------------+----------+--------------+------------+ | Limonade | 1.00 | 4.00 | 4.00 | +--------------------------------------+----------+--------------+------------+ ######## Document ######## :Mindee ID: dfb53ec3-7f05-4e6a-a26a-c727a52b8984 :Filename: receipt_p0_5.pdf Inference ######### :Product: mindee/expense_receipts v5.1 :Rotation applied: Yes Prediction ========== :Expense Locale: fr-FR; fr; FR; EUR; :Purchase Category: miscellaneous :Purchase Subcategory: :Document Type: CREDIT CARD RECEIPT :Purchase Date: 2022-08-23 :Purchase Time: 12:52 :Total Amount: 14.50 :Total Net: :Total Tax: :Tip and Gratuity: :Taxes: +---------------+--------+----------+---------------+ | Base | Code | Rate (%) | Amount | +===============+========+==========+===============+ :Supplier Name: :Supplier Company Registrations: 83833640200022 :Supplier Address: :Supplier Phone Number: :Line Items: +--------------------------------------+----------+--------------+------------+ | Description | Quantity | Total Amount | Unit Price | +======================================+==========+==============+============+ | NO AUTO: MONTANT | | 14.50 | | +--------------------------------------+----------+--------------+------------+ Page Predictions ================ Page 0 ------ :Expense Locale: fr-FR; fr; FR; EUR; :Purchase Category: miscellaneous :Purchase Subcategory: :Document Type: CREDIT CARD RECEIPT :Purchase Date: 2022-08-23 :Purchase Time: 12:52 :Total Amount: 14.50 :Total Net: :Total Tax: :Tip and Gratuity: :Taxes: +---------------+--------+----------+---------------+ | Base | Code | Rate (%) | Amount | +===============+========+==========+===============+ :Supplier Name: :Supplier Company Registrations: 83833640200022 :Supplier Address: :Supplier Phone Number: :Line Items: +--------------------------------------+----------+--------------+------------+ | Description | Quantity | Total Amount | Unit Price | +======================================+==========+==============+============+ | NO AUTO: MONTANT | | 14.50 | | +--------------------------------------+----------+--------------+------------+ ``` Despite showing a filename, the results shouldn't needlessly create temporary files, as everything is managed through file buffers. A working implementation of this code can be found in the `example/multiReceiptsTutorial.js` file on the [Node.js repo](https://github.com/mindee/mindee-api-nodejs/blob/main/example/multiReceiptsTutorial.js). ## Custom Implementation If you are using a non-supported language, or simply want to implement things your own way, here's a gist of how things work under the hood... But first, if you are not familiar with other products yet, know that a response from the API will contain a `document` field, which in turns contains the following attributes of interest: `inference` > `prediction` and `inference` > `pages`. `pages` being a list of objects, all containing their own `prediction` attributes. Think of the `inference`'s `prediction` as an aggregate of these. > **Note** implementation might change depending on the language you use, but the structure will always follow that of the raw json response, which you can find the scheme of in [this page](https://platform.mindee.com/mindee/multi_receipts_detector/documentation#section=%22response-scheme%22). Why are these `prediction`s relevant? Because they contain our `receipts`' coordinates. Which in turn contains a bounding box, which is simply a series of 4 coordinates. The coordinates have a very simple structure: a list of four points, which are each a list of two elements (x position and y position). Those coordinates represent an offset to the top-left origin, displayed in percentage (`float` or equivalent between 0.0 and 1.0). Which means that in order to get a receipt's coordinates, you need to multiply its x coordinate by the width of the file, and its y coordinate by the height of the file. {% hint style="warning" %} Be aware that some libraries flip the y side and consider the bottom-right point to be the origin, which will result in upside-down images if you aren't careful! {% endhint %} The next step is pretty straightforward: use the bounding boxes of each receipts to cut out a piece of the original file, and then either save those file or send them to the Receipt OCR for parsing!