An Algorithm for Individual Intermediate Filament Tracking

Abstract

In this paper we propose an algorithm for automatic tracking of individual intermediate filaments (IF), which form a highly branched network inside living cells. The algorithm is based on Stretching Open Active Contours (SOAC) combined with a novel method for processing the snake’s open ends. The main element of our approach (Endpoint-Controlled Active Contours) is a newly introduced potential function, which is included into the snake’s energy term forcing its end-points to coincide with the filament’s end- or junction-points. The comparison of our method with similar tracking algorithms for the problem of IF tracking shows a significantly improved accuracy of the overall tracking.

1 Introduction

Intermediate filaments are major cytoskeletal components of vertebrate cells. Their molecular composition varies depending on cell type, cellular function and the microenvironment. Typically, intermediate filaments form a filamentous network within the cell and are mechanically connected to neighbor cells. This network is a main contributor to the biomechanical properties of cells and tissues supporting processes such as wound healing and tumor cell invasion [7, 9]. The topology of intermediate filament networks is highly different depending on the cell type, filament type and physiological conditions. The organization and dynamics of intermediate filaments can be best studied in living cells producing fluorescent intermediate filament protein reporters. However, an approach to classify and distinguish between different types of filament topologies is missing. A quantitative description of network’s characteristics using manual labeling of the filament’s trajectories would be extremely time-consuming. Automated filament tracking or segmentation algorithms have been successfully proposed for individual actin filaments which were well separated by background [8, 10]. The task of filament tracking within their highly branched network, in contrast, leads to additional complications due to a high spatio-temporal variability in the contrast of filaments and the risk of confusion with neighboring filaments. As an experimental evaluation shows (see Sect. 4), SOAC-based algorithms tend to overgrow or switch to the neighboring filaments.

In this paper, we introduce a new potential function to the snake’s energy term. This potential function enables better control over the snake’s endpoints preventing the issues with SOAC-based methods. After providing a scheme of the overall tracking procedure in the following subsection and describing the image processing steps in Sect. 2 we present our novel snake-based approach in Sect. 3. An evaluation on microscopic image data and a comparison to other tracking approaches is presented in Sect. 4 before concluding in Sect. 5.

Fig. 1.

Block-diagram of the overall tracking algorithm

Overall Tracking Algorithm. Our procedure for tracking of individual cytoskeletal filaments consists of two main steps: (1) fitting the filament’s position to the image of the current frame and (2) transferring the filament’s curve to the next frame in the sequence. For the second step, we use pyramidal Lucas-Kanade optical flow [2]. It allows to roughly estimate the filament’s position in case of large deformations. Mapping errors are eliminated further on by the repetition of the snakes-based fitting step. In detail, the tracking algorithm consists of the following single steps (see Fig. 1):

• (A) Initialization: The filament’s position is initialized on the first frame (manually by the user or using an additional segmentation procedure [12]). The endpoint control term is initialized as described in Sect. 3.2;

• (B) Image processing of the current frame according to the workflow described in Sect. 2;

• (C) Gradient vector flow (GVF): It is computed from the enhanced image (step (B), for details see step 2 in Sect. 2);

• (D) Endpoint Controlled Snake Evolution: The snake is fitted to the current image based on the GVF obtained in (C) using a stretching [8] and an endpoint control term as described in Sect. 3.2;

• (E) Conditional Statement: If the current image is the last in the sequence, terminate the algorithm, else go to the next step;

• (F) Pyramidal optical flow: It is calculated for the current image with respect to the next one in the time-sequence as described in [2];

• (G) Select the next image of the sequence, transfer the snake to it based on the calculated optical flow field. Update the endpoint control term as described in Sect. 3.2 and repeat the whole procedure starting from step (B).

Fig. 2.

The main image processing steps: (a) original image; (b) Frangi filter and gamma correction; (c) Thresholding using Otsu; (d) Skeletonization and junction- or endpoints detection (Color figure online)

2 Image Processing

Image processing of confocal microscopy videos showing intermediate filament networks should suppress noise, homogenize the highly variable intensity distribution along the filament’s length and enhance the contrast of a filament. Intermediate filaments have a tubular network structure similar to a blood vessel tree but with a higher branchness and a lower variability in radius. In [5] it was shown that image processing algorithms used for vessels’ contrast enhancement and segmentation can be adapted to confocal images of intermediate filaments. For every image of the sequence of N confocal images, we apply the following image processing steps based on the workflow proposed by [1]:

• (1) Firstly, the original grayscale image (see Fig. 2(a)) is inverted and smoothed with a Gaussian kernel ($\sigma \approx 0.5$); the contrast of the image is enhanced with the CLAHE algorithm [1];

• (2) the Hessian-based ridge enhancement filter by Frangi et al. [3] is applied in order to enhance tubular structures and suppress blobs and image noise. The contrast of the filtered image is further enhanced using gamma correction (with $\gamma \approx 0.5$) (cf., Fig. 2(b) for an example).

• (3) Otsu’s threshold $\mathrm {T_{Otsu}}$ is computed [1] and the image I(x, y) is transformed using Otsu as an upper threshold: $I_{new}(x,y) = \mathrm {T_{Otsu}}$, for $I(x,y) \ge \mathrm {T_{Otsu}}$. The intensity range of the image is rescaled to [0, 1] (see Fig. 2(c));

• (4) A threshold value (${\approx }0.25$) is fixed in order to obtain a binary image;

• (5) Noisy components of the binary image (with small area) are removed and a skeleton of the binary image is computed (Fig. 2(d)). Finally, the hit-and-miss morphological transform is used to detect branching/endpoints of the binary skeleton (see red points in Fig. 2(d)).

Fig. 3.

Penalty function $C_{total}$ and it’s terms $C_1$, $C_2$ and $C_3$ computed for every point of the skeleton and encoded in color from blue for small values to red for large values. The green triangle represents $p^{t-1}$ and the magenta one the bifurcation point $p_b$ corresponding to $p^t$. Note that without $C_2(p)$ $p^t$ might be wrongly selected as the next bifurcation below the green triangle. (Color figure online)

3 Filament Tracking Algorithm

3.1 Active Contour Models: Snakes

Let’s denote by $\mathbf {x}^t(s)=[x(s), y(s)], s \in [0,1]$ an open ended curve representing the filament’s position on the image $I_t, t=\overline{0,N-1}$. We are fitting this curve to the current image by minimizing the following so-called “energy” functional [6]:

$$\begin{aligned} E = \int _{0}^{1} \frac{1}{2} \big ( \alpha |\mathbf {x}^t_{s}(s)|^2 + \beta |\mathbf {x}^t_{ss}(s)|^2 \big ) + E_{ext} \big (\mathbf {x}^t(s)\big ) ds \end{aligned}$$

(1)

where $\alpha$ and $\beta$ are parameters controlling the stretching and bending resistance of the curve, correspondingly. According to [6] we solve the following differential equation with an artificial time variable $\tau$ in order to obtain the minimum of the functional (1) above:

$$\begin{aligned} \mathbf {x}^t_{\tau }(s, \tau ) = \alpha \mathbf {x}^t_{ss}(s, \tau ) + \beta \mathbf {x}^t_{ssss}(s, \tau ) - \nabla E_{ext}(\mathbf {x}^t(s, \tau )) \end{aligned}$$

(2)

In this case the “external energy” $E_{ext}$ or the gradient of “external energy” $\nabla E_{ext}$ are computed based on the image data. $\nabla E_{ext}$ defines a structure within the image towards which the snake is converging.

Gradient Vector Flow (GVF). Xu et al. [11] are proposing to replace $\nabla E_{ext}$ by the vector field $\mathbf {v}(x,y)$, called gradient vector flow (GVF). The authors state that GVF is more robust and has a larger capture range in comparison with $\nabla E_{ext}$ defined in [6]. In our procedure we compute GVF based on the enhanced image obtained from the image processing step (3).

Stretching Open Active Contours (SOACs). Applying open-ended snakes as defined in [6] for individual filament tracking has the drawback that the snake tends to shrink over time. In [12] this issue is approached by using a stretching term for open ends $\nabla E_{str}(\mathbf {x}^t(s))$ in the $\nabla E_{ext}$ of Eq. (2).

3.2 Endpoint-Contolled Active Contours (ECAC)

In case of tracking intermediate filaments within their network SOACs tend to increase their length over time, grow beyond their actual position or even move into neighboring filaments. In this section we propose a method, which allows to avoid such undesired behavior by tracking the snake’s endpoints independently.

Endpoint Control Term. Therefore, we define the following term $E_{ends}$, which allows to control the snake’s endpoints and to prevent it from overgrowth. In the first analyzed image of the sequence we initialize a distance-based potential P as a sum of all RBF functions centered at the respective branching points $p_b^0$. These branching points are detected as described in Sect. 2, step (5). For every image $I^t$ of the sequence we define two distance potential functions $E_0^t$ and $E_1^t$ corresponding to the two ends of any snake $\mathbf {x}^t(s)$. Initially, for $I^0$ we set $E_0^0 = E_1^0 = P$. For the next images $I_t : t > 0$ we define $E_0^t$ and $E_1^t$ separately. $E_0^t$ (and equally $E_1^t$) is initialized by placing an RBF centered at a point $p_*^t$, which is the solution of the minimization problem described in the subsection “Endpoints tracking” below. Thus, we define $E_{ends}$ based on $E_0^t$ and $E_1^t$ as follows:

$$\begin{aligned} E_{ends}(\mathbf {x}^t(s)) := {\left\{ \begin{array}{ll} E_0^t(\mathbf {x}^t(s)), &{} \text {if } s=0 \\ E_1^t(\mathbf {x}^t(s)), &{} \text {if } s=1 \\ 0, &{} \text {if } 0< s < 1 \end{array}\right. } \end{aligned}$$

(3)

The position of the snake on the current frame $\mathbf {x}^t(s)$ is optimized by solving Eq. (2), where $\nabla E_{ext}$ incorporates also $E_{ends}$:

$$\begin{aligned} \nabla E_{ext}(\mathbf {x}^t(s)) := \mathbf {v}(\mathbf {x}^t(s)) + \nabla E_{str}(\mathbf {x}^t(s)) + \nabla E_{ends}(\mathbf {x}^t(s)) \end{aligned}$$

(4)

As a result, snake endpoints are captured by a force field and propelled towards their respective nearest detected junction- or endpoint on the skeleton.

Endpoints Tracking. Let’s denote by $p^{t-1}$ the position of the snake’s endpoint on the image $t-1$. In order to determine the position of the RBF center point $p_*^t$ on the current frame we minimize the following cost function: $p_*^t = \arg \min _{p} C_{total} (p; p^{t-1})$, where p is a point sampled from the skeleton of the current image (image processing step (5)) in the neighborhood of point $p^{t-1}$ in the previous frame. The total cost function is defined as a linear combination of three terms:

$$\begin{aligned} C_{total}(p; p^{t-1}) = C_1(p; p^{t-1}) + \gamma C_2(p; p^{t-1}) + \delta C_3(p) \end{aligned}$$

(5)

where $\gamma \ge 0$ and $\delta \ge 0$ are weights balancing the contribution of each term. These three terms were designed to guide the tracking as follows (c.f., Fig. 3):

• (1) $C_1(p; p^{t-1}) := \Vert p - p^{t-1} \Vert$ is a penalty for a large shift with respect to the position of $p^{t-1}$;

• (2) $C_2(p; p^{t-1}) := 1 - \langle \hat{I}^t_{p,s}, \hat{I}^{t-1}_{p^{t-1}, s} \rangle$ is a penalty for local image pattern disparity based on the normalized cross correlation, where $\langle \cdot , \cdot \rangle$ is a binary operator defining zero-normalized cross-correlation of two images and $\hat{I}^t_{p,s}$ and $\hat{I}^{t-1}_{p^{t-1}, s}$ denote image windows of size s centered at point p of the current image and at point $p^{t-1}$ in the previous image, respectively.

• (3) $C_3(p) := \min _{p^t_b} \Vert p - p^t_b \Vert$ is a penalty for a large distance to the detected branching points $p^t_b$ on the current image (see step (5) of Sect. 2);

Finally, the distance-based potentials $E^t_0$ or $E^t_1$ are generated around the corresponding optimal solutions $p_*^t$.

4 Evaluation and Results

The tracker is initialized by interactively labelling the filament on the first frame and propagates it through the whole image sequence. On every frame of the image sequence except the first one we evaluate four distance measures between tracked filament and the ground truth filament.

Fig. 4.

Evaluation of the contribution of each penalty term to the tracking accuracy. The subfigure on the left shows the error distribution for the Frechet distance and the Endpoint distance for image sequences with IDs from 1 to 5. The subfigure on the right depicts the error accumulation over time. Thereby, the mean Frechet distance over all image sequences has been computed.

Table 1. Ground truth dataset

Ground Truth Annotation. In order to evaluate the intermediate filament tracking error we have produced a ground truth dataset comprising 5 sequences of confocal images.

The filaments selected for ground truth annotations were chosen randomly based on the criterion that a filament can be tracked visually by eye on every frame of the image sequence. In total we obtained 350 ground truth filament trajectories. Short description of the dataset is shown in Table 1. More details: https://github.com/if-tracking/if-dataset

Distance Measures. We use four metrics for calculating the distance between two curves: Frechet distance, Endpoint (tip distance), Body distance, Length distance. The three latter measures have been defined in [4].

Fig. 5.

Comparison of the tracking results for algorithms: “Li2009” [8], its modification “Li2009+” and our method (with endpoints control). The ground truth position of the filament is highlighted with a dark green thick line. (Color figure online)

Evaluation Results. We selected the parameters $\alpha$ and $\beta$ in Eq. (5) using a grid search on a training ground truth set. In order to illustrate the importance of each term of our penalty function $C_{total}$ (Sect. 3.2) based on our distance measures, we evaluated our algorithm omitting one of these terms while applying the two others (cf., Fig. 4). Especially, when considering the accumulation of the error over time we observe that omitting only one term increases the error at least by the factor of 1.5.

We also compared our method (ECAC) to the filament tracking algorithm in [8] denoted here by “Li2009” and our modification of [8] denoted by “Li2009+” and described below. Both methods are based on SOAC combined with a Particle Filter (PF) for endpoint tracking. The main difference between “Li2009” and “Li2009+” is in the type of likelihood model $P_l(Z_t|X_t)$ used in PF. $P_l(Z_t|X_t)$ describes the probability of an image patch $Z_t$ for a given position of an endpoint $X_t$. “Li2009” defines $P_l(Z_t|X_t)$ based on an appearance model (which according to [8] works well for tracking individual actin filaments). For the problem of IF tracking we modify (referred as “Li2009+”) the likelihood model as follows: $P_l(Z_t|X_t=p) \propto exp\{ - \mu \min _{p_b} \Vert p - p_b \Vert \},$ where the expression under the exponential is designed to take into account the branching points $p_b^t$ in the current image and is equal to $-\mu C_3(p)$ (see Sect. 3.2). For our data we set $\mu =0.5$.

As it’s shown in Figs. 5 and 6, the combination of three penalty terms for endpoint tracking in ECAC significantly reduces the overall tracking error in terms of all defined measures and outperforms “Li2009” and “Li2009+”. The described above modification of the likelihood model in “Li2009+” also boosts the performance of this algorithm comparing to the original model in “Li2009”’.

Fig. 6.

Comparison of average errors of ECAC with the filament tracking algorithm “Li2009” and its modification “Li2009+” for each of the 5 image sequences.

5 Conclusion

We presented a novel robust snake-based approach called Endpoint-controlled active contours (ECAC) for tracking single intermediate filaments within their cytoskeletal network. Our approach outperforms state-of-the-art approaches for tracking cytoskeletal filaments by at least a factor of 1.5 in terms of mean Frechet distance reduction for each evaluated video. This allows to automate the analysis of geometric and dynamic properties of intermediate filaments over time and to characterize different filament populations within the same cell. In conclusion, the provided algorithm will help to understand how different architectures of intermediate filament network contribute to fundamental cell behavior.