Images End to End

Overview

Below is an outline of how to add the processing required to allow a user to add an image through a form and have that image saved to the database.

It refers throughout to our DINDR app - so please clone it to follow through.

We have opted to store images in their own collection (actually two collections but we'll come to that later) and associate them with users by adding the filename to a new field: profileUpload, in the user collection.

Resources

Excellent tutorial on: Uploading Files to MongoDB With GridFS

Further resources - see the Photo upload section of Appendix/Resources.

Front End - breakdown

Adding a profile pic when creating a new user

see src/components/UserNewForm.js

UserNewForm.js adds an input of type="file" and name="file" to the pre-existing new user form. In the most part the processing for this field is the same as the others.

Previously the addToUsers function, which is associated with the form, simply posted the data to /api/users/new causing the user to be created.

A simple addition in addToUsers passes the file object to the file_upload function:

  fileUpload(file) {
    const url = '/api/profile/new';
    const formData = new FormData();

    formData.append('file', file);
    formData.append('username', this.state.username);

    const config = {
      headers: {
        'content-type': 'multipart/ form-data'
      }
    };
    return axios.post(url, formData,config);
  }

file_upload creates a new form object formData containing the file object and posts this, together with some content info, to /api/profile/new

/api/profile/new, we can assume, has responsibility for adding the image file to the database and associating it with the user.

Retrieving a profile pic for the user profile

see src/components/UserPreview.js

There's not much new code here but you can see that the image has been added to returned structure: 

img src = {`http://localhost:4444/api/profile/image/${this.props.user.profileUpload}`}

Note: The url is currently hard-coded as we are in discovery mode.

Back End - breakdown

Overview

Before proceeding there are a few things that it's handy to know.

There is a certain amount of complication here - I have attempted to give an overview before diving in to the code but also refer you to the excellent tutorial in Resources above.

 see api/profile.js for all server side routing related to images
Chunking in MongoDB

MongoDB stores objects in a binary format called BSON. However, MongoDB objects are typically limited to 4MB in size. To deal with this, files are “chunked” into multiple objects that are less than 4MB each.

This means that we have to chunk our image files as they may be greater than 4MB.

Note: I have also seen 16MB specified so perhaps this is configureable in MongoDB.

GridFS

To avoid having to work out how to chunk files, we have opted to use GridFS.

GridFS uses two collections to store files:

  • a .files collection which stores meta-data like the filename and content-type
  • and a .chunks collection which hold the file itself as binary data.

We have opted to call our images collection uploads - therefore we have two collections in the database: uploads.files and uploads.chunks

Using gridfs-stream, multer and multer-gridfs-storage

These three packages work together to store and retrieve chunked files in MongoDB.

gridfs-stream easily streams files to and from MongoDB GridFS.

_Multer _is a node.js middleware for handling multipart / form-data, which is primarily used for uploading files.

multer-gridfs-storage is a storage engine for Multer.

Using crypto

We use crypto to create a random fixed length hexadecimal string which, being likely to be unique, we will use for the filename.

The actual filenames are far to likely to be duplicated which would cause problems.

Not Using methodOverride

methodOverride lets you use HTTP verbs such as PUT or DELETE in places where the client doesn't support it.

We haven't implemented delete functionality so don't be confused by its presence.

The Back-end Code

User model

The User model has changed only to add the profileUpload which will contain the unique filename of the user's profile pic.

See models/users.js
Routing for Images

The bulk of the work for storing and retrieving image files in our mongoDB instance occurs in profile.js.

see api/profile.js below
Open a connection

Initially a gridfs-stream connection is made to the DB and 'uploads' is identified to it as the collection we will be working with. 

conn.once('open', ()=> {
  gfs = Grid(conn.db, mongoose.mongo);
  gfs.collection('uploads');
});
router.post /new

router.post /new is the route used to save an image file when creating a new user.

It has two callbacks:

  • upload.single('file') which is Multer middleware, saves a file to the DB. It has been passed a GridFsStorage object called storage (below). Multer will add a 'file' property for request. The function storage.file will be executed once the image has been saved.
  • anonymous function which find the appropriate user via the username passed in and updates the profileUpload with the new filename.

It returns the file object.

router.post('/new', upload.single('file'), (req, res) => {
  User.find({ username: req.body.username }, function(err, user){
    user = user[0];
    user.profileUpload = req.file.filename;
    user.save(function (err){
      if(err) {
        console.log('filename not added');
      }
    });
    res.json({file: req.file});
  });
});
storage
  • url will be used by multer to address the correct DB although we don't get to see that.
  • file will pass back a resolved promise containg the filename - this will be used to update user.profileUpload.
    const storage = new GridFsStorage({
url: mongoURI,
file: (req, file) => {
  return new Promise((resolve, reject) => {
    crypto.randomBytes(16, (err, buf) => {
      if (err) {
        return reject(err);
      }
      const filename = buf.toString('hex') + path.extname(file.originalname);
      const fileInfo = {
        filename: filename,
        bucketName: 'uploads'
      };
      resolve(fileInfo);
    });
  });
}
});
    router.get('/image/:filename'...

This route is used to display images in each user preview.

It uses the gridfs-stream object gfs to find the required file.

gfs.files.findOne({filename: req.params.filename}

Asumming no errors, it then pipes the image back using

  const readstream = gfs.createReadStream(file.filename);
  readstream.pipe(res);

api/profile.js - the heavy work

import express from 'express';
import multer from 'multer';
import path from 'path';
import crypto from 'crypto';
import mongoose from 'mongoose';
import GridFsStorage from 'multer-gridfs-storage';
import Grid from 'gridfs-stream';
import methodOverride from 'method-override';
import bodyParser from 'body-parser';

let gfs;
let User = require('../models/users.js');
const mongoURI = process.env.MONGOLAB_URI;
const conn = mongoose.createConnection(mongoURI);
const router = express.Router();
const upload = multer({ storage });

router.use(bodyParser.json());
router.use(methodOverride('_method'));

conn.once('open', ()=> {
  gfs = Grid(conn.db, mongoose.mongo);
  gfs.collection('uploads');
});

const storage = new GridFsStorage({
  url: mongoURI,
  file: (req, file) => {
    return new Promise((resolve, reject) => {
      crypto.randomBytes(16, (err, buf) => {
        if (err) {
          return reject(err);
        }
        const filename = buf.toString('hex') + path.extname(file.originalname);
        const fileInfo = {
          filename: filename,
          bucketName: 'uploads'
        };
        resolve(fileInfo);
      });
    });
  }
});

router.post('/new', upload.single('file'), (req, res) => {
  User.find({ username: req.body.username }, function(err, user){
    user = user[0];
    user.profileUpload = req.file.filename;
    user.save(function (err){
      if(err) {
        console.log('filename not added');
      }
    });
    res.json({file: req.file});
  });
});

router.get('/:filename', (req, res) => {
  gfs.files.findOne({filename: req.params.filename}, (err, file) =>{ // gets filename from url
    if (!file || file.length === 0){
      return res.status(404).json({
        err: 'No file exists'
      });
    }
    return res.json(file);
  });
});

router.get('/image/:filename', (req, res) => {
  gfs.files.findOne({filename: req.params.filename}, (err, file) =>{ // gets filename from url
    if (!file || file.length === 0){
      return res.status(404).json({
        err: 'No file exists'
      });
    }
    if (file.contentType === 'image/jpeg' || file.contentType === 'image/png') {
      const readstream = gfs.createReadStream(file.filename);
      readstream.pipe(res);
    } else {
      res.status(404).json({
        err: 'Not an image'
      });
    }
  });
});

export default router;

results matching ""

    No results matching ""