Abstract
Time-lapse fluorescent microscopy (TLFM) combined with predictive mathematical modelling is a powerful tool to study the inherently dynamic processes of life on the single-cell level. Such experiments are costly, complex and labour intensive. A complimentary approach and a step towards in silico experimentation, is to synthesise the imagery itself. Here, we propose Multi-StyleGAN as a descriptive approach to simulate time-lapse fluorescence microscopy imagery of living cells, based on a past experiment. This novel generative adversarial network synthesises a multi-domain sequence of consecutive timesteps. We showcase Multi-StyleGAN on imagery of multiple live yeast cells in microstructured environments and train on a dataset recorded in our laboratory. The simulation captures underlying biophysical factors and time dependencies, such as cell morphology, growth, physical interactions, as well as the intensity of a fluorescent reporter protein. An immediate application is to generate additional training and validation data for feature extraction algorithms or to aid and expedite development of advanced experimental techniques such as online monitoring or control of cells.
Code and dataset is available at https://git.rwth-aachen.de/bcs/projects/tp/multi-stylegan.
C. Reich and T. Prangemeier—Both authors contributed equally.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Abdal, R., Qin, Y., Wonka, P.: Image2StyleGAN++: how to edit the embedded images? In: CVPR, pp. 8296–8305 (2020)
Bailo, O., Ham, D., Shin, Y.M.: Red blood cell image generation for data augmentation using conditional generative adversarial networks. In: CVPRW (2019)
Bandiera, L., Hou, Z., Kothamachu, V.B., Balsa-Canto, E., Swain, P.S., Menolascina, F.: On-line optimal input design increases the efficiency and accuracy of the modelling of an inducible synthetic promoter. Processes 6(9) (2018)
Barratt, S., Sharma, R.: A note on the inception score. In: ICML Workshop (2018)
Carreira, J., Zisserman, A.: Quo vadis, action recognition? A new model and the kinetics dataset. In: CVPR, pp. 6299–6308 (2017)
Chessel, A., Carazo Salas, R.E.: From observing to predicting single-cell structure and function with high-throughput/high-content microscopy. Essays Biochem. 63(2), 197–208 (2019)
Comes, M.C., et al.: Multi-scale generative adversarial network for improved evaluation of cell-cell interactions observed in organ-on-chip experiments. Neural. Comput. Appl. 33, 3671–3689 (2020)
Goldsborough, P., Pawlowski, N., Caicedo, J.C., Singh, S., Carpenter, A.E.: CytoGAN: generative modeling of cell images. BioRxiv, p. 227645 (2017)
Goodfellow, I., et al.: Generative adversarial nets. In: NeurIPS, vol. 27, pp. 2672–2680 (2014)
Hall, M.S., Decker, J.T., Shea, L.D.: Towards systems tissue engineering: elucidating the dynamics, spatial coordination, and individual cells driving emergent behaviors. Biomaterials 255, 120189 (2020)
Han, L., Murphy, R.F., Ramanan, D.: Learning generative models of tissue organization with supervised GANs. In: WACV, pp. 682–690 (2018)
Han, L., Yin, Z.: Transferring microscopy image modalities with conditional generative adversarial networks. In: CVPRW, pp. 851–859 (2017)
Henningsen, J., Schwarz-Schilling, M., Leibl, A., Gutiérrez, J., Sagredo, S., Simmel, F.C.: Single cell characterization of a synthetic bacterial clock with a hybrid feedback loop containing dCas9-sgRNA. ACS Synth. Biol. 9(12), 3377–3387 (2020)
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs trained by a two time-scale update rule converge to a local nash equilibrium. In: NeurIPS, vol. 30, pp. 6626–6637 (2017)
Hofmann, A., et al.: A tightly regulated and adjustable CRISPR-dCas9 based AND gate in yeast. Nucleic Acids Res. 47(1), 509–520 (2019)
Johnson, G.R., Donovan-Maiye, R.M., Maleckar, M.M.: Generative modeling with conditional autoencoders: building an integrated cell. arXiv:1705.00092 (2017)
Karras, T., Aittala, M., Hellsten, J., Laine, S., Lehtinen, J., Aila, T.: Training generative adversarial networks with limited data. In: NeurIPS, vol. 33, pp. 12104–12114 (2020)
Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: CVPR, pp. 4401–4410 (2019)
Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of StyleGAN. In: CVPR, pp. 8110–8119 (2020)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: ICLR (2015)
Lee, G., Oh, J.W., Her, N.G., Jeong, W.K.: DeepHCS++: bright-field to fluorescence microscopy image conversion using multi-task learning with adversarial losses for label-free high-content screening. Med. Image Anal. 70, 101995 (2021)
Lee, G., Oh, J.-W., Kang, M.-S., Her, N.-G., Kim, M.-H., Jeong, W.-K.: DeepHCS: bright-field to fluorescence microscopy image conversion using deep learning for label-free high-content screening. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11071, pp. 335–343. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00934-2_38
Leygeber, M., et al.: Analyzing microbial population heterogeneity - expanding the toolbox of microfluidic single-cell cultivations. J. Mol. Biol. 431, 4569–4588 (2019)
Lugagne, J., Lin, H., Dunlop, M.: DeLTA: automated cell segmentation, tracking, and lineage reconstruction using deep learning. PLOS Comput. Biol. 16(4) (2020)
Mescheder, L., Geiger, A., Nowozin, S.: Which training methods for GANs do actually converge? In: ICML, pp. 3481–3490 (2018)
Osokin, A., Chessel, A., Carazo Salas, R.E., Vaggi, F.: GANs for biological image synthesis. In: ICCV (2017)
Pepperkok, R., Ellenberg, J.: High-throughput fluorescence microscopy for systems biology. Nat. Rev. Mol. Cell Biol. 7(9), 690–696 (2006)
Prangemeier, T., Wildner, C., Françani, A.O., Reich, C., Koeppl, H.: Multiclass yeast segmentation in microstructured environments with deep learning. In: IEEE CIBCB, pp. 1–8 (2020)
Prangemeier, T., Lehr, F.X., Schoeman, R.M., Koeppl, H.: Microfluidic platforms for the dynamic characterisation of synthetic circuitry. Curr. Opin. Biotechnol. 63, 167–176 (2020)
Prangemeier, T., Reich, C., Koeppl, H.: Attention-based transformers for instance segmentation of cells in microstructures. In: IEEE BIBM, pp. 700–707 (2020)
Prangemeier, T., Wildner, C., Hanst, M., Koeppl, H.: Maximizing information gain for the characterization of biomolecular circuits. In: Proceedings of the 5th ACM/IEEE NanoCom, pp. 1–6 (2018)
Salimans, T., et al.: Improved techniques for training GANs. In: NeurIPS, vol. 29, pp. 2234–2242 (2016)
Schonfeld, E., Schiele, B., Khoreva, A.: A U-Net based discriminator for generative adversarial networks. In: CVPR, pp. 8207–8216 (2020)
Sinha, S., Zhao, Z., Goyal, A., Raffel, C.A., Odena, A.: Top-k training of GANs: improving GAN performance by throwing away bad samples. In: NeurIPS, vol. 33, pp. 14638–14649 (2020)
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: CVPR, pp. 2818–2826 (2016)
Ulman, V., Svoboda, D., Nykter, M., Kozubek, M., Ruusuvuori, P.: Virtual cell imaging: a review on simulation methods employed in image cytometry. Cytometry A 89(12), 1057–1072 (2016)
Unterthiner, T., van Steenkiste, S., Kurach, K., Marinier, R., Michalski, M., Gelly, S.: FVD: a new metric for video generation. In: ICLR Workshop (2019)
Wang, N.B., Beitz, A.M., Galloway, K.: Engineering cell fate: applying synthetic biology to cellular reprogramming. Curr. Opin. Syst. Biol. 24, 18–31 (2020)
Wang, X., Girshick, R., Gupta, A., He, K.: Non-local neural networks. In: CVPR, pp. 7794–7803 (2018)
Wieslander, H., Gupta, A., Bergman, E., Hallström, E., Harrison, P.J.: Learning to see colours: generating biologically relevant fluorescent labels from bright-field images. bioRxiv (2021)
Zhang, H., et al.: High-throughput, high-resolution deep learning microscopy based on registration-free generative adversarial network. Biomed. Opt. Express 10(3), 1044–1063 (2019)
Acknowledgements
We thank Markus Baier for aid with the computational setup, Klaus-Dieter Voss for aid with the microfluidics fabrication, and Tim Kircher, Tizian Dege, and Florian Schwald for aid with the data preparation.
This work was supported by the Landesoffensive für wissenschaftliche Exzellenz as part of the LOEWE Schwerpunkt CompuGene. H.K. acknowledges support from the European Research Council (ERC) with the consolidator grant CONSYN (nr. 773196).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Supplementary material 2 (mp4 3260 KB)
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Reich, C., Prangemeier, T., Wildner, C., Koeppl, H. (2021). Multi-StyleGAN: Towards Image-Based Simulation of Time-Lapse Live-Cell Microscopy. In: de Bruijne, M., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2021. MICCAI 2021. Lecture Notes in Computer Science(), vol 12908. Springer, Cham. https://doi.org/10.1007/978-3-030-87237-3_46
Download citation
DOI: https://doi.org/10.1007/978-3-030-87237-3_46
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-87236-6
Online ISBN: 978-3-030-87237-3
eBook Packages: Computer ScienceComputer Science (R0)
