Tutorial: Multi Plant Image Workflow

PlantCV is composed of modular functions that can be arranged (or rearranged) and adjusted quickly and easily. Workflows do not need to be linear (and often are not). Please see workflow example below for more details. A global variable "debug" allows the user to print out the resulting image. The debug has three modes: either None, 'plot', or print'. If set to 'print' then the function prints the image out, if using a Jupyter notebook you could set debug to 'plot' to have the images plot to the screen. Debug mode allows users to visualize and optimize each step on individual test images and small test sets before workflows are deployed over whole datasets.

For multi-plant workflows, images with multiple plants are processed and result in individual pictures for each plant, allowing a secondary workflow (see VIS tutorial for example) to be used. The challenge of multi-plant processing is that a single plant can be composed of several contours, therefore contours need to be sorted and clustered together in some way. There are several functions that help with multi-plant image processing. First, the current clustering functions work by asking the user to provide an approximation of the number of desired 'rows' and 'columns' that they would like to split the image into. There does not need to be a plant in each spot, but the grid is used as an approximate region to cluster contours within. The rotation and shift functions allow the image to be moved to optimize accurate clustering. Major assumptions that are made are that plants grow but that the imaging position does not change drastically. Also, the clustering functions will not work properly once plants start overlapping, since contours would also start overlapping.

Binder Check out our interactive multi-plant tutorial!

Also see here for the complete script.

Workflow

  1. Optimize workflow on individual image with debug set to 'print' (or 'plot' if using a Jupyter notebook).
  2. Run workflow on small test set (ideally that spans time and/or treatments).
  3. Re-optimize workflows on 'problem images' after manual inspection of test set.
  4. Deploy optimized workflow over test set using parallelization script.

Running A Workflow

To run a multi-plant workflow over a single VIS image there are two required inputs:

  1. Image: Images can be processed regardless of what type of VIS camera was used (high-throughput platform, digital camera, cell phone camera). Image processing will work with adjustments if images are well lit and free of background that is similar in color to plant material.
  2. Output directory: If debug mode is set to 'print' output images from each step are produced.

Optional inputs:

  • Names File: path to txt file with names of genotypes to split images into (order of names would be top to bottom, left to right).
  • Debug Flag: Prints an image at each step.

Sample command to run a workflow on a single image:

  • Always test workflows (preferably with -D 'print' option for debug mode) before running over a full image set
./workflowname.py -i multi-plant-img.png -o ./output-images -n names.txt -D 'print'

Walk Through A Sample Workflow

Workflows start by importing necessary packages, and by defining user inputs.

#!/usr/bin/python

import numpy as np
import argparse 
from plantcv import plantcv as pcv

### Parse command-line arguments
def options():
    parser = argparse.ArgumentParser(description="Imaging processing with opencv")
    parser.add_argument("-i", "--image", help="Input image file.", required=True)
    parser.add_argument("-o", "--outdir", help="Output directory for image files.", required=True)
    parser.add_argument("-n", "--names", help="path to txt file with names of genotypes to split images into", required =False)
    parser.add_argument("-D", "--debug", help="Turn on debug, prints intermediate images.", action=None)
    args = parser.parse_args()
    return args

Start of the Main/Customizable portion of the workflow.

The image input by the user is read in.

### Main workflow
def main():
    # Get options
    args = options()

    # Read image
    img, path, filename = pcv.readimage(filename=args.image)

    pcv.params.debug=args.debug #set debug mode

Figure 1. Original image. This particular image was captured by a raspberry pi camera, just to show that PlantCV works on images not captured on a high-throughput phenotyping system with idealized VIS image capture conditions. In this dataset images were captured over time of a flat (throughout the day and night).

Screenshot

Check if this is an image captured at night.


    # STEP 1: Check if this is a night image, for some of these dataset's images were captured
    # at night, even if nothing is visible. To make sure that images are not taken at
    # night we check that the image isn't mostly dark (0=black, 255=white).
    # if it is a night image it throws a fatal error and stops the workflow.

    if np.average(img) < 50:
        pcv.fatal_error("Night Image")
    else:
        pass

White balance the image so that color can be compared across images and so that image processing can be the same between images (ideally).


    # STEP 2: Normalize the white color so you can later
    # compare color between images.
    # Inputs:
    #   img = image object, RGB colorspace
    #   roi = region for white reference, if none uses the whole image,
    #         otherwise (x position, y position, box width, box height)

    # white balance image based on white toughspot

    img1 = pcv.white_balance(img=img,roi=(400,800,200,200))

Figure 2. White balance the image so that later image processing is easier.

Screenshot

Screenshot

Screenshot

Rotate image slightly so that plants line up with grid (later step)


    # STEP 3: Rotate the image
    # Inputs:
    #   img = image object, RGB color space
    #   rotation_deg = Rotation angle in degrees, can be negative, positive values 
    #                  will move counter-clockwise 
    #   crop = If True then image will be cropped to original image dimensions, if False
    #          the image size will be adjusted to accommodate new image dimensions 

    rotate_img = pcv.rotate(img=img1, rotation_deg=-1, crop=False)

Figure 3. Rotated image

Screenshot

Shift image slightly so that plants line up with grid (later step)


    # STEP 4: Shift image. This step is important for clustering later on.
    # For this image it also allows you to push the green raspberry pi camera
    # out of the image. This step might not be necessary for all images.
    # The resulting image is the same size as the original.
    # Inputs:
    #   img    = image object
    #   number = integer, number of pixels to move image
    #   side   = direction to move from "top", "bottom", "right","left"

    shift1 = pcv.shift_img(img=img1, number=40, side='top')
    img1 = shift1

Figure 4. Shifted image

Screenshot

Convert the image from RGB to LAB and select single color channel that has contrast between the target object and background.


    # STEP 5: Convert image from RGB colorspace to LAB colorspace
    # Keep only the green-magenta channel (grayscale)
    # Inputs:
    #    rgb_img = image object, RGB colorspace
    #    channel = color subchannel ('l' = lightness, 'a' = green-magenta , 'b' = blue-yellow)

    a = pcv.rgb2gray_lab(rgb_img=img1, channel='a')

Figure 5. Green-magenta channel from LAB color space from original image.

Screenshot

Use the binary threshold function to threshold green-magenta image.


    # STEP 6: Set a binary threshold on the saturation channel image
    # Inputs:
    #    gray_img    = img object, grayscale
    #    threshold   = threshold value (0-255)
    #    max_value   = value to apply above threshold (usually 255 = white)
    #    object_type = 'light' or 'dark'
    #       - If object is light then standard thresholding is done
    #       - If object is dark then inverse thresholding is done

    img_binary = pcv.threshold.binary(gray_img=a, threshold=120, max_value=255, object_type='dark')
    #                                                        ^
    #                                                        |
    #                                          adjust this value

Figure 6. Thresholded image.

Screenshot

Fill noise in the image.


    # STEP 7: Fill in small objects (speckles)
    # Inputs:
    #    gray_img = image object, grayscale. img will be returned after filling
    #    size     = minimum object area size in pixels (integer)

    fill_image = pcv.fill(gray_img=img_binary, size=10)
    #                                                ^
    #                                                |
    #                                 adjust this value

Figure 7. Fill noise.

Screenshot

Dilate binary image.


    # STEP 8: Dilate so that you don't lose leaves (just in case)
    # Inputs:
    #    gray_img = input image
    #    ksize    = kernel size, integer
    #    i        = iterations, i.e. number of consecutive filtering passes

    dilated = pcv.dilate(gray_img=fill_image, ksize=2, i=1)

Figure 8. Dilated image.

Screenshot

Find objects within the image.


    # STEP 9: Find objects (contours: black-white boundaries)
    # Inputs:
    #    img  = image that the objects will be overlayed
    #    mask = what is used for object detection

    id_objects, obj_hierarchy = pcv.find_objects(img=img1, mask=dilated)

Figure 9. Identified objects.

Screenshot

Define a rectangular region of interest in the image.


    # STEP 10: Define region of interest (ROI)
    # Inputs:
    #    img       = img to overlay roi
    #    x_adj     = adjust center along x axis
    #    y_adj     = adjust center along y axis
    #    h_adj     = adjust height
    #    w_adj     = adjust width
    # roi_contour, roi_hierarchy = pcv.roi.rectangle(img1, 10, 500, -10, -100)
    #                                                      ^                ^
    #                                                      |________________|
    #                                            adjust these four values

    roi_contour, roi_hierarchy = pcv.roi.rectangle(img=img1, x=6, y=90, h=200, w=390)

Figure 10. Define ROI.

Screenshot

The objects within and overlapping are kept with the region of interest objects function. Alternately the objects can be cut to the region of interest.


    # STEP 11: Keep objects that overlap with the ROI
    # Inputs:
    #    img            = img to display kept objects
    #    roi_contour    = contour of roi, output from any ROI function
    #    roi_hierarchy  = contour of roi, output from any ROI function
    #    object_contour = contours of objects, output from "Identifying Objects" function
    #    obj_hierarchy  = hierarchy of objects, output from "Identifying Objects" function
    #    roi_type       = 'partial' (default, for partially inside), 'cutto', or 'largest' (keep only largest contour)

    roi_objects, roi_obj_hierarchy, kept_mask, obj_area = pcv.roi_objects(img=img1, roi_contour=roi_contour, 
                                                                          roi_hierarchy=roi_hierarchy,
                                                                          object_contour=id_objects,
                                                                          obj_hierarchy=obj_hierarchy, 
                                                                          roi_type='partial')

Figure 11. Define ROI.

Screenshot

Screenshot

Screenshot

Cluster plants based on defined grid. Optionally, users can add a visualization step to more clearly see which contours get clustered together. See the documentation for pcv.visualze.clustered_contours for an example.


    # STEP 12: This function take a image with multiple contours and
    # clusters them based on user input of rows and columns

    # Inputs:
    #    img               = An RGB or grayscale image
    #    roi_objects       = object contours in an image that are needed to be clustered.
    #    roi_obj_hierarchy = object hierarchy
    #    nrow              = number of rows to cluster (this should be the approximate  number of desired rows in the entire image even if there isn't a literal row of plants)
    #    ncol              = number of columns to cluster (this should be the approximate number of desired columns in the entire image even if there isn't a literal row of plants)
    #    show_grid         = if True then a grid gets displayed in debug mode (default show_grid=False)


    clusters_i, contours, hierarchies = pcv.cluster_contours(img=img1, roi_objects=roi_objects, 
                                                             roi_obj_hierarchy=roi_obj_hierarchy, 
                                                             nrow=4, ncol=6)

Figure 12. Cluster contours

Screenshot

Split the images.


    # STEP 13: This function takes clustered contours and splits them into multiple images,
    # also does a check to make sure that the number of inputted filenames matches the number
    # of clustered contours. If no filenames are given then the objects are just numbered
    # Inputs:
    #    img                     = ideally a masked RGB image.
    #    grouped_contour_indexes = output of cluster_contours, indexes of clusters of contours
    #    contours                = contours to cluster, output of cluster_contours
    #    hierarchy               = object hierarchy
    #    outdir                  = directory for output images
    #    file                    = the name of the input image to use as a base name , output of filename from read_image function
    #    filenames               = input txt file with list of filenames in order from top to bottom left to right (likely list of genotypes)

    # Set global debug behavior to None (default), "print" (to file), or "plot" (Jupyter Notebooks or X11)
    pcv.params.debug = "print"

    out = args.outdir
    names = args.names

    output_path, imgs, masks = pcv.cluster_contour_splitimg(img=img1, grouped_contour_indexes=clusters_i, 
                                                            contours=contours, hierarchy=hierarchies, 
                                                            outdir=out, file=filename, filenames=names)

Figure 13. Split image based on clustering.

Screenshot

Screenshot

Screenshot

Screenshot

Screenshot

Screenshot

Screenshot

Screenshot

Screenshot

Screenshot

Screenshot

Screenshot

Screenshot

Screenshot

Screenshot

Screenshot

Screenshot

Screenshot

To deploy a workflow over a full image set please see tutorial on workflow parallelization.

Multi Plant Script

In the terminal:

./workflowname.py -i multi-plant-img.png -o ./output-images -n names.txt -D 'print'

  • Always test workflows (preferably with -D flag set to 'print') before running over a full image set

Python script:

#!/usr/bin/python

import sys, traceback
import cv2
import os
import re
import numpy as np
import argparse
import string
from plantcv import plantcv as pcv

### Parse command-line arguments
def options():
    parser = argparse.ArgumentParser(description="Imaging processing with opencv")
    parser.add_argument("-i", "--image", help="Input image file.", required=True)
    parser.add_argument("-o", "--outdir", help="Output directory for image files.", required=True)
    parser.add_argument("-n", "--names", help="path to txt file with names of genotypes to split images into", required =False)
    parser.add_argument("-D", "--debug", help="Turn on debug, prints intermediate images.", action=None)
    args = parser.parse_args()
    return args

### Main workflow
def main():
    # Get options
    args = options()

    # Read image
    img, path, filename = pcv.readimage(args.image)

    pcv.params.debug=args.debug #set debug mode

    # STEP 1: Check if this is a night image, for some of these dataset's images were captured
    # at night, even if nothing is visible. To make sure that images are not taken at
    # night we check that the image isn't mostly dark (0=black, 255=white).
    # if it is a night image it throws a fatal error and stops the workflow.

    if np.average(img) < 50:
        pcv.fatal_error("Night Image")
    else:
        pass

    # STEP 2: Normalize the white color so you can later
    # compare color between images.
    # Inputs:
    #   img = image object, RGB colorspace
    #   roi = region for white reference, if none uses the whole image,
    #         otherwise (x position, y position, box width, box height)

    # white balance image based on white toughspot

    img1 = pcv.white_balance(img=img,roi=(400,800,200,200))

    # STEP 3: Rotate the image
    # Inputs:
    #   img = image object, RGB color space
    #   rotation_deg = Rotation angle in degrees, can be negative, positive values 
    #                  will move counter-clockwise 
    #   crop = If True then image will be cropped to original image dimensions, if False
    #          the image size will be adjusted to accommodate new image dimensions 


    rotate_img = pcv.rotate(img=img1,rotation_deg=-1, crop=False)

    # STEP 4: Shift image. This step is important for clustering later on.
    # For this image it also allows you to push the green raspberry pi camera
    # out of the image. This step might not be necessary for all images.
    # The resulting image is the same size as the original.
    # Inputs:
    #   img    = image object
    #   number = integer, number of pixels to move image
    #   side   = direction to move from "top", "bottom", "right","left"

    shift1 = pcv.shift_img(img=img1, number=300, side='top')
    img1 = shift1

    # STEP 5: Convert image from RGB colorspace to LAB colorspace
    # Keep only the green-magenta channel (grayscale)
    # Inputs:
    #    img     = image object, RGB colorspace
    #    channel = color subchannel ('l' = lightness, 'a' = green-magenta , 'b' = blue-yellow)

    a = pcv.rgb2gray_lab(img=img1, channel='a')

    # STEP 6: Set a binary threshold on the saturation channel image
    # Inputs:
    #    img         = img object, grayscale
    #    threshold   = threshold value (0-255)
    #    max_value   = value to apply above threshold (usually 255 = white)
    #    object_type = light or dark
    #       - If object is light then standard thresholding is done
    #       - If object is dark then inverse thresholding is done

    img_binary = pcv.threshold.binary(img=a, threshold=120, max_value=255, object_type='dark')
    #                                                   ^
    #                                                   |
    #                                     adjust this value

    # STEP 7: Fill in small objects (speckles)
    # Inputs:
    #    img  = image object, grayscale. img will be returned after filling
    #    size = minimum object area size in pixels (integer)

    fill_image = pcv.fill(img=img_binary, size=100)
    #                                          ^
    #                                          |
    #                           adjust this value

    # STEP 8: Dilate so that you don't lose leaves (just in case)
    # Inputs:
    #    img    = input image
    #    ksize  = kernel size
    #    i      = iterations, i.e. number of consecutive filtering passes

    dilated = pcv.dilate(img=fill_image, ksize=1, i=1)

    # STEP 9: Find objects (contours: black-white boundaries)
    # Inputs:
    #    img  = image that the objects will be overlayed
    #    mask = what is used for object detection

    id_objects, obj_hierarchy = pcv.find_objects(img=img1, mask=dilated)

    # STEP 10: Define region of interest (ROI)
    # Inputs:
    #    img       = img to overlay roi
    #    x_adj     = adjust center along x axis
    #    y_adj     = adjust center along y axis
    #    h_adj     = adjust height
    #    w_adj     = adjust width
    # roi_contour, roi_hierarchy = pcv.roi.rectangle(img1, 10, 500, -10, -100)
    #                                                      ^                ^
    #                                                      |________________|
    #                                            adjust these four values

    roi_contour, roi_hierarchy = pcv.roi.rectangle(img=img1, x=6, y=90, h=200, w=390)

    # STEP 11: Keep objects that overlap with the ROI
    # Inputs:
    #    img            = img to display kept objects
    #    roi_contour    = contour of roi, output from any ROI function
    #    roi_hierarchy  = contour of roi, output from any ROI function
    #    object_contour = contours of objects, output from "Identifying Objects" function
    #    obj_hierarchy  = hierarchy of objects, output from "Identifying Objects" function
    #    roi_type       = 'partial' (default, for partially inside), 'cutto', or 'largest' (keep only largest contour)

    roi_objects, roi_obj_hierarchy, kept_mask, obj_area = pcv.roi_objects(img=img1, roi_contour=roi_contour, 
                                                                          roi_hierarchy=roi_hierarchy,
                                                                          object_contour=id_objects,
                                                                          obj_hierarchy=obj_hierarchy, 
                                                                          roi_type='partial')

    # STEP 12: This function take a image with multiple contours and
    # clusters them based on user input of rows and columns

    # Inputs:
    #    img               = An RGB image
    #    roi_objects       = object contours in an image that are needed to be clustered.
    #    roi_obj_hierarchy = object hierarchy
    #    nrow              = number of rows to cluster (this should be the approximate  number of desired rows in the entire image even if there isn't a literal row of plants)
    #    ncol              = number of columns to cluster (this should be the approximate number of desired columns in the entire image even if there isn't a literal row of plants)
    #    show_grid         = if True then a grid gets displayed in debug mode (default show_grid=False)

    clusters_i, contours, hierarchies = pcv.cluster_contours(img=img1, roi_objects=roi_objects, 
                                                             roi_obj_hierarchy=roi_obj_hierarchy, 
                                                             nrow=4, ncol=6)

    # STEP 13: This function takes clustered contours and splits them into multiple images,
    # also does a check to make sure that the number of inputted filenames matches the number
    # of clustered contours. If no filenames are given then the objects are just numbered
    # Inputs:
    #    img                     = ideally a masked RGB image.
    #    grouped_contour_indexes = output of cluster_contours, indexes of clusters of contours
    #    contours                = contours to cluster, output of cluster_contours
    #    hierarchy               = object hierarchy
    #    outdir                  = directory for output images
    #    file                    = the name of the input image to use as a base name , output of filename from read_image function
    #    filenames               = input txt file with list of filenames in order from top to bottom left to right (likely list of genotypes)

    # Set global debug behavior to None (default), "print" (to file), or "plot" (Jupyter Notebooks or X11)
    pcv.params.debug = "print"

    out = args.outdir
    names = args.names

    output_path, imgs, masks = pcv.cluster_contour_splitimg(img=img1, grouped_contour_indexes=clusters_i, 
                                                            contours=contours, hierarchy=hierarchies, 
                                                            outdir=out, file=filename, filenames=names)

# Call program
if __name__ == '__main__':
    main()